OSF Innovation brings VR beating hearts to life for safer surgeries
Key Takeaways:
- 4D Heart gives surgeons a dynamic, patient-specific virtual reality view of the heart in motion.
- AI automation and analysis created by a collaboration with OSF Innovation, Bradley University, and U of I engineers, makes creating these digital twins fast and practical.
- The technology is already shaping surgical plans and eventually could become standard care.
Imagine being a heart surgeon preparing for a complex case. Traditionally, you rely on stacks of 2D images from CT scans to build a mental picture of your patient’s anatomy. But the heart isn’t static – it twists and pumps blood in a rhythm unique to each person. Now, thanks to a breakthrough at OSF Innovation, surgeons can “see” their patient’s heart beating in 3D before ever stepping into the operating room.
This innovation is called the 4D Heart. Unlike standard 3D models, which capture a single snapshot in time, the 4D Heart recreates every phase of a patient’s cardiac cycle, allowing doctors to watch their heart beat virtually. And it isn’t just a generic animation. Rather, it is the creation of a digital twin.
“This is taking a CT scan of a patient, a retrospectively gated CT scan, and converting a patient’s own heart – in each phase of that cardiac cycle – and making it beat in a virtual environment,” says Matthew Bramlet, MD, who directs the Advanced Image and Modeling (AIM) Lab at OSF Innovation. The lab is focused on improving the understanding of complex anatomy for surgeons and diagnosticians. Dr. Bramlet also specializes in congenital cardiac MRI for OSF HealthCare Children’s Hospital of Illinois and is an associate professor of Clinical Pediatrics at the University of Illinois College of Medicine in Peoria and the University of Illinois Grainger College of Engineering in Urbana-Champaign.
Dr. Bramlet’s efforts began in 2018 with manually segmenting parts of the heart so it could be viewed in virtual reality. It continued with an infusion of grant money two years ago. Using an Innovation for Health (IFH) grant with Bradley University in Peoria, Illinois, a team used machine learning to convert standard medical images into a sequence of 3D models that, when played sequentially, create a 3D beating heart – the 4D heart. The initial process took an entire summer.
That made the technology nearly impossible to use in real world clinical care. But in late 2024, researchers achieved a major breakthrough. With the help of engineers at the University of Illinois Urbana-Champaign, AIM lab engineers trained neural networks on hundreds of high-quality CT scans and automated the entire process. Now, what once took months can be done with a single click.
Have you ever seen a flip book where you see a drawing move? That’s what we’re doing with these 3D models in a digital, virtual space. We’re just turning them off and on in rapid succession until we have that beating heart.
“Have you ever seen a flip book where you see a drawing move? That’s what we’re doing with these 3D models in a digital, virtual space.” Dr. Bramlet explains, “We’re just turning them off and on in rapid succession until we have that beating heart.”
For surgeons, the benefits are game-changing. They can see not only the anatomy but also how it changes with every beat. This is especially critical for conditions such as hypertrophic obstructive cardiomyopathy, where the problem lies in how the heart contracts. Even more surprising, the AI began rendering valves — structures notoriously difficult to see on CT scans.
“Now we can see that valve which we really couldn’t see before. So, repeat valve procedures are very valuable for this. TAVR or any valve in the cath lab, we’re finding good use with these as well as the hypertrophic cardiomyopathy cases.”
Hospitals across the country are taking notice, sending scans to OSF and receiving beating heart files in return. Some don’t even have the technology to view them yet. Dr. Bramlet says OSF Innovation is pushing the boundaries so far forward, some other health systems will need to get on board with the technology to be able to visualize the data being sent to them.
For now, the ability to view a 4D Heart is already changing how surgeons think about the body’s most vital organ. In some cases, the new view is confirming a surgery plan. In other cases, including a most recent pediatric heart case from another hospital, it changed the approach. Dr. Bramlet says as surgeons interact with the models, they are amazed.
He shares, “I see those lightbulbs go on and there’s nothing like being able to observe and interact with these 3D models in a format, and especially a 4D model, in a way they’ve never been able to see them before, and it will become the standard of care, I have no doubt.”
The ultimate vision is bold: a patient enters the CT or MRI scanner, and their digital twin heart comes out the other side – ready for analysis, surgical planning and even analysis of AI-driven measurements like ejection fraction or blood flow strain to help with diagnosis and a treatment plan.
How long until that happens? Hard to say because as Dr. Bramlet points out, “seeing is believing,” and it might take time for health systems to invest in the technology to make the vision become reality. Government reimbursement for the care will also drive adoption. For now, the work is being covered through research grants, most notably; the Jump ARCHES endowment through the Health Care Engineering Systems Center at the University of Illinois Urbana Champaign and OSF HealthCare.