MRI images are understandably complex and data-heavy.

Because of this, the developers training large language models (LLMs) for MRI analysis must slice the acquired 2D images. But this results in only an approximation of the original image, thus limiting the model’s ability to analyze intricate anatomical structures. This creates challenges in the complex cases involved brain tumorsskeletal diseases or cardiovascular diseases.

but GE Healthcare appears to have overcome this major hurdle, introducing the industry’s first full-body 3D MRI research foundation model (FM) this year AWS re: Invention. For the first time, models can use full 3D images of the entire body.

GE Healthcare’s FM was built on AWS from the ground up – there are few models specifically designed for medical imaging such as MRIs – and is based on more than 173,000 images from more than 19,000 studies. The developers say that they were able to train the model with five times less computing than previously required.

GE Healthcare has yet to commercialize the foundation model; it is still in the evolutionary stage of research. An early evaluator, Mass. General Brighamis set to start experimenting with it soon.

“Our vision is to put these models in the hands of technical teams working in healthcare systems, giving them powerful tools for improving research and clinical applications that’s faster, and also more cost-effective,” GE HealthCare chief AI officer Parry Bhatia told VentureBeat.

Enable real-time analysis of complex 3D MRI data

While this is a groundbreaking development, generative AI and LLMs are not new territory for the company. The team has been working with advanced technologies for more than 10 years, Bhatia explained.

One of its flagship products is AIR Recon DLa deep learning-based reconstruction algorithm that allows radiologists to more quickly achieve crisp images. The algorithm removes noise from raw images and improves the signal-to-noise ratio, cutting scan times by up to 50%. As of 2020, 34 million patients have been scanned using the AIR Recon DL.

GE Healthcare started working on its MRI FM at the beginning of 2024. Because the model is multimodal, it can support image-to-text search, linking images and words, and segmenting and classifying diseases. The goal is to give health care professionals more details in a single scan than ever before, Bhatia said, leading to faster, more accurate diagnosis and treatment.

“The model has significant potential to enable real-time analysis of 3D MRI data, which can improve medical procedures such as biopsies, radiation therapy and robotic surgery,” Dan Sheeran, GM for health care and life sciences at AWS, told VentureBeat.

Already, it has outperformed other publicly available research models on tasks including classifying prostate cancer and Alzheimer’s disease. It shows an accuracy of up to 30% in matching MRI scans with textual descriptions of the image capture – which may not sound as impressive, but it’s a huge improvement over the 3% that capabilities shown in both models.

“It has reached a stage where it is giving some solid results,” Bhatia said. “The implications are huge.”

Doing more with (less) data

the MRI process requires several different types of datasets to support the various techniques that map the human body, Bhatia explained.

The well-known T1-weighted imaging technique, for example, emphasizes fatty tissue and reduces water signal, while T2-weighted imaging enhances water signals. Both methods are complementary and create a complete picture of the brain to help clinicians detect abnormalities such as tumors, trauma or cancer.

“MRI images come in all different shapes and sizes, similar to how you have books in different formats and sizes, right?” said Bhatia.

To overcome the challenges presented by different datasets, the developers introduced a “resize and adapt” strategy so that the model can process and react to different variations. Also, the data may be missing in some areas – an image may be incomplete, for example – so they teach the model to simply ignore those instances.

“Instead of getting stuck, we teach the model to skip the gaps and focus on what works,” Bhatia said. “Think of it as solving a puzzle with some missing pieces.”

The developers also use semi-supervised student-teacher learning, which is especially helpful when there is limited data. With this approach, two different neural networks are trained on both labeled and unlabeled data, with the teacher creating labels that help the student learn and predict future labels.

“We’re now using a lot of self-directed technologies, which don’t require a lot of data or labels to train large models,” Bhatia said. “It reduces dependencies, where you can learn more from raw images than before.”

This helps to ensure that the model performs well in hospitals with limited resources, older machines and different types of data, Bhatia explained.

He also emphasized the importance of multimodality in models. “A lot of technology used to be unimodal,” Bhatia said. “It used to look only in the image, in the text. But now they’re multi-modal, they can switch from image to text, text to image, so you can bring a lot of things that were done in different models before and the work flow will be really integrated.”

He emphasized that researchers use only the datasets they have rights to; GE Healthcare has partners who license anonymized data sets, and they are diligent in following compliance standards and policies.

Used AWS SageMaker to solve computational, data challenges

Undoubtedly, there are many challenges in creating such sophisticated models – such as limited computing power for 3D images that are gigabytes in size.

“It’s a huge 3D volume of data,” Bhatia said. “You have to bring it into the memory of the model, which is a complex problem.”

To help overcome this, GE Healthcare was founded Amazon SageMakerwhich provides high-speed networking and distributed training capabilities on multiple GPUs, and uses Nvidia A100 and tensor core GPUs for large-scale training.

“Because of the size of the data and the size of the models, they can’t send it to a GPU,” Bhatia explained. SageMaker allows them to customize and scale the operations of multiple GPUs that can interact with each other.

Also used by developers Amazon FSx on Amazon S3 object storage, which allows faster reading and writing for datasets.

Bhatia points out that another challenge is cost optimization; with Amazon’s elastic compute cloud (EC2), developers were able to move unused or infrequently used data to lower-cost storage levels.

“Using Sagemaker for training these large models – primarily for efficient, distributed training on multiple high-performance GPU clusters – was one of the critical components that really helped us operate faster,” Bhatia said.

He emphasized that all components are built from a data integrity and compliance perspective that takes into account HIPAA and other regulations and regulatory frameworks.

Ultimately, “these technologies can really streamline, help us change faster, as well as improve overall operational efficiency by reducing the administrative burden, and ultimately drive better patient care – because now you’re providing more personalized care.”

Serves as the basis for other special good models

While the model for now is specific to the MRI domain, researchers see many opportunities to expand into other areas of medicine.

Sheeran points out that, historically, AI in medical imaging has been hampered by the need to create custom models for specific conditions in specific organs, requiring expert annotation for each image used in training.

But that approach is “inherently limited” because of the different ways diseases manifest in individuals, and introduces challenges to generalizability.

“What we really need are thousands of models and the ability to rapidly create new ones as we learn new information,” he said. High-quality labeled datasets for each model are also essential.

Now with generative AI, instead of training discrete models for each disease/organ combination, developers can pre-train a foundational model that can serve as the basis for other specialized goodies. which are targeted models below.

For example, GE Healthcare’s model could be expanded to areas such as radiation therapy, where radiologists spend more time marking organs that may be at risk. It will also help reduce scanning time during x-rays and other procedures that currently require patients to sit in a machine for long periods of time, Bhatia said.

Sheeran marvels that “we’re not just expanding access to medical imaging data through cloud-based tools; we’re revolutionizing how that data can be used to drive AI advances in healthcare.