Imagine being a heart surgeon preparing for a complex case. Traditionally, you rely on stacks of 2D images from CT scans to build a mental picture of your patient’s anatomy. But the heart isn’t static – it twists and pumps blood in a rhythm unique to each person. Now, thanks to a breakthrough at OSF Innovation, surgeons can “see” their patient’s heart beating in 3D before ever stepping into the operating room.

This innovation is called the 4D Heart. Unlike standard 3D models, which capture a single snapshot in time, the 4D Heart recreates every phase of a patient’s cardiac cycle, allowing doctors to watch their heart beat virtually. And it isn’t just a generic animation. Rather, it is the creation of a digital twin.

“This is taking a CT scan of a patient, a retrospectively gated CT scan, and converting a patient’s own heart – in each phase of that cardiac cycle – and making it beat in a virtual environment,” says Matthew Bramlet, MD, who directs the Advanced Image and Modeling (AIM) Lab at OSF Innovation. The lab is focused on improving the understanding of complex anatomy for surgeons and diagnosticians. Dr. Bramlet also specializes in congenital cardiac MRI for OSF HealthCare Children’s Hospital of Illinois and is an associate professor of Clinical Pediatrics at the University of Illinois College of Medicine in Peoria and the University of Illinois Grainger College of Engineering in Urbana-Champaign.

Dr. Bramlet’s efforts began in 2018 with manually segmenting parts of the heart so it could be viewed in virtual reality. It continued with an infusion of grant money two years ago. Using an Innovation for Health (IFH) grant with Bradley University in Peoria, Illinois, a team used machine learning to convert standard medical images into a sequence of 3D models that, when played sequentially, create a 3D beating heart – the 4D heart. The initial process took an entire summer.

That made the technology nearly impossible to use in real world clinical care. But in late 2024, researchers achieved a major breakthrough. With the help of engineers at the University of Illinois Urbana-Champaign, AIM lab engineers trained neural networks on hundreds of high-quality CT scans and automated the entire process. Now, what once took months can be done with a single click.

“Have you ever seen a flip book where you see a drawing move? That’s what we’re doing with these 3D models in a digital, virtual space.” Dr. Bramlet explains, “We’re just turning them off and on in rapid succession until we have that beating heart.”

For surgeons, the benefits are game-changing. They can see not only the anatomy but also how it changes with every beat. This is especially critical for conditions such as hypertrophic obstructive cardiomyopathy, where the problem lies in how the heart contracts. Even more surprising, the AI began rendering valves — structures notoriously difficult to see on CT scans.

“Now we can see that valve which we really couldn’t see before. So, repeat valve procedures are very valuable for this. TAVR or any valve in the cath lab, we’re finding good use with these as well as the hypertrophic cardiomyopathy cases.”

Hospitals across the country are taking notice, sending scans to OSF and receiving beating heart files in return. Some don’t even have the technology to view them yet. Dr. Bramlet says OSF Innovation is pushing the boundaries so far forward, some other health systems will need to get on board with the technology to be able to visualize the data being sent to them.

For now, the ability to view a 4D Heart is already changing how surgeons think about the body’s most vital organ. In some cases, the new view is confirming a surgery plan. In other cases, including a most recent pediatric heart case from another hospital, it changed the approach. Dr. Bramlet says as surgeons interact with the models, they are amazed.

He shares, “I see those lightbulbs go on and there’s nothing like being able to observe and interact with these 3D models in a format, and especially a 4D model, in a way they’ve never been able to see them before, and it will become the standard of care, I have no doubt.”

The ultimate vision is bold: a patient enters the CT or MRI scanner, and their digital twin heart comes out the other side – ready for analysis, surgical planning and even analysis of AI-driven measurements like ejection fraction or blood flow strain to help with diagnosis and a treatment plan.

How long until that happens? Hard to say because as Dr. Bramlet points out, “seeing is believing,” and it might take time for health systems to invest in the technology to make the vision become reality. Government reimbursement for the care will also drive adoption. For now, the work is being covered through research grants, most notably; the Jump ARCHES endowment through the Health Care Engineering Systems Center at the University of Illinois Urbana Champaign and OSF HealthCare.

Dr. Mathew Bramlet

B-roll-4D Beating Heart

In a rapidly evolving technological landscape, machine learning has emerged as a transformative force; it is reshaping countless industries, health care included. Machine learning and artificial intelligence increases productivity, reduces human errors and frees up time for other tasks that can have greater impact. In the case of converting medical scans into 3D or 4D images for a deeper view of organs, tissues and tumors, AI has the possibility of changing how surgeons prepare to operate on patients with complex issues.

Matthew Bramlet, MD, is a pediatric cardiologist at the University of Illinois College of Medicine at Peoria who specializes in congenital cardiac MRI for the Children’s Hospital of Illinois.  His Advanced Imaging and Modeling (AIM) lab at Jump Trading Simulation & Education Center is focused on translation of medical images into 3D and 4D interactive models (for 3D printing for virtual reality) to assist in pre-surgical planning of complex cardiac and cancer cases. 

According to Dr. Bramlet, the heart is a unique organ because it changes shape as it expands and contracts to pump blood through the body. This change in shape becomes a problem when the muscle of the heart grows abnormally and obstructs its own ability to pump blood out of the heart. This is a problem, because no current technology exists that allows a surgeon to view this 4D, dynamic obstruction. 

Rather than relying on years of trial and error of surgical expertise to determine the correct way to cut out muscle to relieve the obstruction, this project seeks to provide the surgeon with a VR view of their next patient’s beating heart in 4D.

The Innovation for Health (IFH) grant program with Bradley University in Peoria, Illinois is designed to inject funding ($50,000) to combine clinical problems with computer science expertise.  Assistant Professor Sam Hawkins, PhD, and graduate students at Bradley University plan to utilize machine learning to convert standard medical images into a sequence of 3D models, that when played sequentially create a 3D beating heart; the “4D heart”.  

This concept has been previously demonstrated by Dr. Bramlet’s AIM lab, but it took an entire summer to convert images of a 4D heart, showing both the expansion and contraction of the heart that could be viewed through a virtual reality headset. Dr. Bramlet says that’s not a sustainable approach.

“How do we put the CT data into a computer and slice by slice, say this is the myocardial tissue so we get an exact replica of the heart. But I don't need just one (3D replica of a heart); I need 20 to make each phase of that heart (he simulates the sound of repetitive beats) into a 4D heart.”

Assistant professor Hawkins believes machine learning can be leveraged to perform the labor-intensive process of converting images. Hawkins explains Computer Science graduate students at Bradley are part of the research effort.

“First, we need to figure out what part of the image is the heart and what part is not. And then we need to combine these tiny images into a 3D image. And then we need to do that many times to get the 4D.”

Improving pre-surgery planning

The solution should allow surgeons to virtually preview the beating heart of their patient, showing the obstruction in 4D, therefore providing a completely new pre-surgical analysis tool. The multidimensional view allows surgeons to zero in on structures of the heart, significantly enlarge elements and get a much better view of the anatomy. This machine learned 4D generation of a beating heart will be a first of its kind technology.

Researchers believe creating an automated solution can reduce the manual process of converting images from months, to hours, and eventually perhaps minutes. That option could eventually be applied to images for other types of complex medical cases in adults, children and the smallest infants.

Dr. Bramlet says new AI-supported software could provide a scaled solution for surgeons everywhere.

“The most immediate impact isn’t going to be the 4D heart. The impact at a grander scale will be the scalability of how any program will be able to create models for pre-surgical planning; 3D printed or for VR with this technology.”

Hawkins believes the technology will lead to other hospitals adopting the approach for pre-surgery planning.

“It has the potential to really remove the barrier of entry for institutions that don’t have the expertise or the time but do have images they wanted to view (more intricately).”

This entire machine learning project has been made possible by Jump Simulation’s long-term investment in sharing 3D insights. Since, 2014, Jump has contributed 3D models of congenital heart disease to Dr. Fauci’s 3D database at the NIH. This library of annotated 3D models provides the rich dataset of raw material needed for machine learned insights into automated segmentation of patient specific 3D models.  

Within a year, this project aims to automate, through machine learned algorithms, the conversion of cardiac CTs into virtual reality. The 4D view will enable surgeons to see the beating hearts specific to individual patients.

Imagine being a surgeon who can plan a surgery in VR by seeing the beating heart of your patient before you walk into the OR. This project seeks to transition this vision from science fiction to science fact.

Dr. Matthew Bramlet

Sam Hawkins, PhD

4D Hearts B-roll