In a rapidly evolving technological landscape, machine learning has emerged as a transformative force; it is reshaping countless industries, health care included. Machine learning and artificial intelligence increases productivity, reduces human errors and frees up time for other tasks that can have greater impact. In the case of converting medical scans into 3D or 4D images for a deeper view of organs, tissues and tumors, AI has the possibility of changing how surgeons prepare to operate on patients with complex issues.

Matthew Bramlet, MD, is a pediatric cardiologist at the University of Illinois College of Medicine at Peoria who specializes in congenital cardiac MRI for the Children’s Hospital of Illinois.  His Advanced Imaging and Modeling (AIM) lab at Jump Trading Simulation & Education Center is focused on translation of medical images into 3D and 4D interactive models (for 3D printing for virtual reality) to assist in pre-surgical planning of complex cardiac and cancer cases. 

According to Dr. Bramlet, the heart is a unique organ because it changes shape as it expands and contracts to pump blood through the body. This change in shape becomes a problem when the muscle of the heart grows abnormally and obstructs its own ability to pump blood out of the heart. This is a problem, because no current technology exists that allows a surgeon to view this 4D, dynamic obstruction. 

Rather than relying on years of trial and error of surgical expertise to determine the correct way to cut out muscle to relieve the obstruction, this project seeks to provide the surgeon with a VR view of their next patient’s beating heart in 4D.

The Innovation for Health (IFH) grant program with Bradley University in Peoria, Illinois is designed to inject funding ($50,000) to combine clinical problems with computer science expertise.  Assistant Professor Sam Hawkins, PhD, and graduate students at Bradley University plan to utilize machine learning to convert standard medical images into a sequence of 3D models, that when played sequentially create a 3D beating heart; the “4D heart”.  

This concept has been previously demonstrated by Dr. Bramlet’s AIM lab, but it took an entire summer to convert images of a 4D heart, showing both the expansion and contraction of the heart that could be viewed through a virtual reality headset. Dr. Bramlet says that’s not a sustainable approach.

“How do we put the CT data into a computer and slice by slice, say this is the myocardial tissue so we get an exact replica of the heart. But I don't need just one (3D replica of a heart); I need 20 to make each phase of that heart (he simulates the sound of repetitive beats) into a 4D heart.”

Assistant professor Hawkins believes machine learning can be leveraged to perform the labor-intensive process of converting images. Hawkins explains Computer Science graduate students at Bradley are part of the research effort.

“First, we need to figure out what part of the image is the heart and what part is not. And then we need to combine these tiny images into a 3D image. And then we need to do that many times to get the 4D.”

Improving pre-surgery planning

The solution should allow surgeons to virtually preview the beating heart of their patient, showing the obstruction in 4D, therefore providing a completely new pre-surgical analysis tool. The multidimensional view allows surgeons to zero in on structures of the heart, significantly enlarge elements and get a much better view of the anatomy. This machine learned 4D generation of a beating heart will be a first of its kind technology.

Researchers believe creating an automated solution can reduce the manual process of converting images from months, to hours, and eventually perhaps minutes. That option could eventually be applied to images for other types of complex medical cases in adults, children and the smallest infants.

Dr. Bramlet says new AI-supported software could provide a scaled solution for surgeons everywhere.

“The most immediate impact isn’t going to be the 4D heart. The impact at a grander scale will be the scalability of how any program will be able to create models for pre-surgical planning; 3D printed or for VR with this technology.”

Hawkins believes the technology will lead to other hospitals adopting the approach for pre-surgery planning.

“It has the potential to really remove the barrier of entry for institutions that don’t have the expertise or the time but do have images they wanted to view (more intricately).”

This entire machine learning project has been made possible by Jump Simulation’s long-term investment in sharing 3D insights. Since, 2014, Jump has contributed 3D models of congenital heart disease to Dr. Fauci’s 3D database at the NIH. This library of annotated 3D models provides the rich dataset of raw material needed for machine learned insights into automated segmentation of patient specific 3D models.  

Within a year, this project aims to automate, through machine learned algorithms, the conversion of cardiac CTs into virtual reality. The 4D view will enable surgeons to see the beating hearts specific to individual patients.

Imagine being a surgeon who can plan a surgery in VR by seeing the beating heart of your patient before you walk into the OR. This project seeks to transition this vision from science fiction to science fact.

Dr. Matthew Bramlet

Sam Hawkins, PhD

4D Hearts B-roll

OSF HealthCare and Bradley University are focusing on cancer prevention and treatment, as well as improving overall cardiac imaging plus pulmonary lung monitoring for babies and young children. The grants are the result of a partnership announced earlier this year between the Peoria, Illinois-based health system and the city’s premiere private university. The awards be given twice a year with joint funding from each organization.

This first round of grants, totaling nearly $200,000, have been awarded to four projects for research into rapidly developing the best, innovative approaches to medical training and health care delivery. The IFH grant awards include co-investigators from both Bradley University and OSF HealthCare.

Here are the Innovation for Health projects for spring 2023:

Toward Automation of a 4D Heart from Retrospectively Gated Cardiac CT Scans

Lead researchers
Matthew Bramlet, MD, OSF HealthCare
Samuel Hawkins, PhD, Bradley University

The goal of this project is to determine the feasibility automating the segmentation of the 18 phases of gated cardiac CT scans to generate patient-specific, 4D beating hearts and that these 4D animations can be placed into and analyzed in a digital stereoscopic format such as virtual reality.

Pulmonary Acoustic Sensor Telemetry Array (PASTA)

Lead researchers
Adam Cross, MD, FAAP, OSF HealthCare
Suruz Miah, PhD, Bradley University
Connor Davey, BA, OSF HealthCare
Reid Jockisch, BS, OSF HealthCare

This venture is an expansion of the Pulmonary Acoustic Sensor Telemetry Array (PASTA) project which is focused on developing a device for use with an algorithm to remotely monitor lung sounds in pediatric patients. The primary deliverable of this phase is a physical device that is easily replicable and records eight channels of audio simultaneously without losing quality due to compression with minimal quality loss due to compression.

Characterization of Changes in Biomechanical Properties and Cell Aggression in Ovarian Cancer Stem Cells Following Exposure to Chemotherapy

Lead researchers
Kalyani Nair, PhD, Bradley University
Craig Cady PhD, Bradley University
Daniel Chan, MD, PhD, OSF HealthCare

The objective of this concept is to determine whether treatment with a chemotherapeutic agent will induce identifiable changes in the biomechanical properties of cancer stem cells which can be correlated with an increase in cancer stem cell aggression. Identifying biomechanical properties linked to an increase in aggression may suggest alternative approaches to suppressing cancer stem cell activation and reducing the recurrence of cancer.

Humanizing Breast Cancer Prevention: A Community-Based Approach to Increase Health Literacy and Address Breast Cancer Health Disparities among Local Underserved Women

Lead researchers
Heather Ford, MFA, Bradley University
Scott Barrows MA, FAMI, OSF HealthCare
Tianjiao (Grace) Wang, PhD, Bradley University|
Rachelle Pavelko PhD, Bradley University
Mary Stapel, MD, OSF HealthCare

This project aims to address health disparities among underserved groups by developing a community-based solution to effectively increase breast cancer health literacy and raise awareness of breast cancer prevention among local ethnic minority women.