AI Combines Chest X-rays with Patient Data to Improve Diagnosis
Released: October 03, 2023
At A Glance
- A new AI model combines imaging information with clinical patient data to improve diagnostic performance on chest X-rays.
- Researchers trained the model to diagnose up to 25 conditions using imaging or non-imaging data, or a combination of both.
- The model using both imaging and non-imaging data outperformed the other models.
- RSNA Media Relations
1-630-590-7762
media@rsna.org - Linda Brooks
1-630-590-7738
lbrooks@rsna.org - Imani Harris
1-630-481-1009
iharris@rsna.org
OAK BROOK, Ill. — A new artificial intelligence (AI) model combines imaging information with clinical patient data to improve diagnostic performance on chest X-rays, according to a study published in Radiology, a journal of the Radiological Society of North America (RSNA).
Clinicians consider both imaging and non-imaging data when diagnosing diseases. However, current AI-based approaches are tailored to solve tasks with only one type of data at a time.
Transformer-based neural networks, a relatively new class of AI models, have the ability to combine imaging and non-imaging data for a more accurate diagnosis. These transformer models were initially developed for the computer processing of human language. They have since fueled large language models like ChatGPT and Google's AI chat service, Bard.
"Unlike convolutional neural networks, which are tuned to process imaging data, transformer models form a more general type of neural network," said study lead author Firas Khader, M.Sc., a Ph.D. student in the Department of Diagnostic and Interventional Radiology at University Hospital Aachen in Aachen, Germany. "They rely on a so-called attention mechanism, which allows the neural network to learn about relationships in its input."
This capability is ideal for medicine, where multiple variables like patient data and imaging findings are often integrated into the diagnosis.
Khader and colleagues developed a transformer model tailored for medical use. They trained it on imaging and non-imaging patient data from two databases containing information from a combined total of more than 82,000 patients.
The researchers trained the model to diagnose up to 25 conditions using non-imaging data, imaging data, or a combination of both, referred to as multimodal data.
Compared to the other models, the multimodal model showed improved diagnostic performance for all conditions.
The model has potential as an aid to clinicians in a time of growing workloads.
"With patient data volumes increasing steadily over the years and time that the doctors can spend per patient being limited, it might become increasingly challenging for clinicians to interpret all available information effectively," Khader said. "Multimodal models hold the promise to assist clinicians in their diagnosis by facilitating the aggregation of the available data into an accurate diagnosis."
The proposed model could serve as a blueprint for seamlessly integrating large data volumes, Khader said.
"Multimodal Deep Learning for Integrating Chest Radiographs and Clinical Parameters - A Case for Transformers." Collaborating with Dr. Khader were Gustav Müller-Franzes, M.Sc., Tianci Wang, B.Sc., Tianyu Han, M.Sc., Soroosh Tayebi Arasteh, M.Sc., Christoph Haarburger, Ph.D., Johannes Stegmaier, Ph.D., Keno Bressem, M.D., Christiane Kuhl, M.D., Sven Nebelung, M.D., Jakob Nikolas Kather, M.D., and Daniel Truhn, M.D., Ph.D.
In 2023, Radiology is celebrating its 100th anniversary with 12 centennial issues, highlighting Radiology's legacy of publishing exceptional and practical science to improve patient care.
Radiology is edited by Linda Moy, M.D., New York University, New York, N.Y., and owned and published by the Radiological Society of North America, Inc. (https://pubs.rsna.org/journal/radiology)
RSNA is an association of radiologists, radiation oncologists, medical physicists and related scientists promoting excellence in patient care and health care delivery through education, research and technologic innovation. The Society is based in Oak Brook, Illinois. (RSNA.org)
For patient-friendly information on chest X-rays, visit RadiologyInfo.org.
Video (MP4):
Video 1. Firas Khader, M.Sc., discusses his research: AI Combines Chest X-rays with Patient Data to Improve Diagnosis
Download MP4
(Right-click and Save As)
Images (JPG, TIF):
Figure 1. Diagram shows an overview of the study. (A–E) Imaging and nonimaging information were extracted from the publicly available Medical Information Mart for Intensive Care data set (A) and an internal data set of chest radiographic and accompanying clinical parametric data (B) . The data sets were split into training, validation, and test sets, and a transformer-based neural network architecture (C) was trained to predict the diagnosis of up to 25 different pathologic conditions. First, the attention mechanism in the transformer architecture (D) was leveraged to provide insight into the decision-making process of the neural network, and it was shown that the predictive performance of the neural network (E) increased for all three data sets when both imaging and nonimaging inputs were provided compared with either imaging (AUC, 0.70) or nonimaging inputs alone.
High-res (TIF) version
(Right-click and Save As)
Figure 2. (A) Schematic shows the model architecture. (B) Line graph shows the epoch duration time for models trained on the same graphics processing unit (Quadro RTX 6000; NVIDIA). (C) Line graph shows graphics processing unit (GPU) video random-access memory (VRAM) consumption as a function of the number of input parameters. MiB = mebibyte, MLP = multilayer perceptron.
High-res (TIF) version
(Right-click and Save As)
Figure 3. (A, B) Box plots show the predictive performance of the trained neural networks in terms of the area under the receiver operating characteristic curve (AUC), positive predictive value, sensitivity, and specificity for the Medical Information Mart for Intensive Care data set (A) and an internal data set of chest radiographs and clinical parameters (B) .
High-res (TIF) version
(Right-click and Save As)
Figure 4. (A) Box plots show performance in terms of the area under the receiver operating characteristic curve, positive predictive value, sensitivity, and specificity of the neural network trained on the Medical Information Mart for Intensive Care data set when a number of clinical parameters (nonimaging information) were omitted. (B) Horizontal bar graphs show clinical parameters that most affected the performance of the neural network for diagnosis of diabetes (without complications), shock, acute cerebrovascular disease, and septicemia. GCS = Glasgow Coma Scale.
High-res (TIF) version
(Right-click and Save As)
Figure 5. Representative radiographs (top), acquired in anteroposterior projection in the supine position, and corresponding attention maps (bottom). (A) Images show main diagnostic findings of the internal data set in a 49-year-old male patient with congestion, pneumonic infiltrates, and effusion (left); a 64-year-old male patient with congestion, pneumonic infiltrates, and effusion (middle); and a 69-year-old female patient with effusion (right). (B) Images show main diagnostic findings of the Medical Information Mart for Intensive Care data set in a 79-year-old male patient with cardiomegaly and pneumonic infiltrates in the right lower lung (left); a 58-year-old female patient with bilateral atelectasis and effusion in the lower lungs (middle); and a 48-year-old female patient with pneumonic infiltrates in the lower right lung (right). Note that the attention maps consistently focus on the most relevant image regions (eg, pneumonic opacities are indicated by opaque image regions of the lung).
High-res (TIF) version
(Right-click and Save As)