| RSNA
RSNA Press Release

High Accuracy AI Improves Lung Cancer Detection

Share: Share on Facebook PDF

Released: June 27, 2023

At A Glance

  • Assistance from an AI algorithm with high diagnostic accuracy improved radiologist performance in lung cancer detection on chest X-ray and increased human acceptance of AI suggestions.
  • Twenty experienced thoracic radiologists and 10 radiology residents assessed 120 chest X-rays without AI, then reinterpreted the X-rays, assisted by either a high- or low-accuracy AI.
  • The high-accuracy AI led to improved cancer detection and more frequent changes in reader determinations—a concept known as susceptibility.

OAK BROOK, Ill. — Assistance from an artificial intelligence (AI) algorithm with high diagnostic accuracy improved radiologist performance in detecting lung cancers on chest X-rays and increased human acceptance of AI suggestions, according to a study published in Radiology, a journal of the Radiological Society of North America (RSNA).

While AI-based image diagnosis has advanced rapidly in the medical field, the factors affecting radiologists' diagnostic determinations in AI-assisted image reading remain underexplored.

Researchers at Seoul National University looked at how these factors might influence the detection of malignant lung nodules during AI-assisted reading of chest X-rays.

In this retrospective study, 30 readers, including 20 thoracic radiologists with five to 18 years of experience and 10 radiology residents with only two to three years of experience, assessed 120 chest X-rays without AI. Of the 120 chest radiographs assessed, 60 were from lung cancer patients (32 males) and 60 were controls (36 males). Patients had a median age of 67 years. In a second session, each group reinterpreted the X-rays, assisted by either a high- or low-accuracy AI. The readers were blind to the fact that two different AIs were used.

Chang Min Park, M.D., Ph.D.

Chang Min Park, M.D., Ph.D.

Use of the high accuracy AI improved readers' detection performance to a greater extent than low-accuracy AI. Use of high-accuracy AI also led to more frequent changes in reader determinations—a concept known as susceptibility.

"It is possible that the relatively large sample size in this study bolstered readers' confidence in the AI's suggestions," said study lead author Chang Min Park, M.D., Ph.D., from the Department of Radiology and Institute of Radiation Medicine at Seoul National University College of Medicine in Seoul. "We think this issue of human trust in AI is what we observed in the susceptibility in this study: humans are more susceptible to AI when using high diagnostic performance AI."

Compared to the first reading session, readers assisted by the high diagnostic accuracy AI at the second reading session showed higher per-lesion sensitivity (0.63 versus 0.53), and specificity (0.94 versus 0.88). Alternatively, readers assisted by the low diagnostic accuracy AI at the second reading session did not show improvement between the two reading sessions for any of these measurements.

"Our study suggests that AI can help radiologists, but only when the AI's diagnostic performance meets or exceeds that of the human reader," Dr. Park said.

The results underline the importance of using high diagnostic performance AI. However, Dr. Park noted that the definition of "high diagnostic performance AI" can vary depending on the task and the clinical context in which it will be used. For example, an AI model that can detect all abnormalities on chest X-rays may seem ideal. But in practice, such a model would have limited value in reducing the workload in a pulmonary tuberculosis mass screening setting.

"Therefore, our study suggests that clinically appropriate use of AI requires both the development of high-performance AI models for given tasks and considerations about the relevant clinical setting to which that AI will be applied," Dr. Park said.

In the future, the researchers want to expand their work on human-AI collaboration to other abnormalities on chest X-rays and CT images.

"Effect of Human-AI Interaction on Detection of Malignant Lung Nodules on Chest Radiographs." Collaborating with Dr. Park were Jong Hyuk Lee, M.D., Ph.D., Hyunsook Hong, Ph.D., Gunhee Nam, Ph.D., and Eui Jin Hwang, M.D., Ph.D.

In 2023, Radiology is celebrating its 100th anniversary with 12 centennial issues, highlighting Radiology's legacy of publishing exceptional and practical science to improve patient care.

Radiology is edited by Linda Moy, M.D., New York University, New York, N.Y., and owned and published by the Radiological Society of North America, Inc. (https://pubs.rsna.org/journal/radiology)

RSNA is an association of radiologists, radiation oncologists, medical physicists and related scientists promoting excellence in patient care and health care delivery through education, research and technologic innovation. The Society is based in Oak Brook, Illinois. (RSNA.org)

For patient-friendly information on chest X-rays, visit RadiologyInfo.org.

Thumbnail
Figure 1. Flowchart shows study design. Using a study sample of 120 randomly selected chest radiographs (60 chest radiographs with 63 lung cancer nodules and 60 normal chest radiographs), 30 readers (20 thoracic radiologists and 10 radiology residents) assessed them at the first session. Then, the readers were dichotomized into two groups with identical diagnostic performance according to the results at the first session. In the second session, the two reader groups reassessed the 120 chest radiographs aided by either full data set–trained artificial intelligence (AI) or 10% data set–trained AI.
High-res (TIF) version
(Right-click and Save As)
Thumbnail
Figure 2. Accuracy of the artificial intelligence (AI) model type based on diagnostic difficulty of the image or in detecting lung nodules. (A) Bar graph shows the per-radiograph accuracy (n = 120). The high diagnostic accuracy AI (blue) and low diagnostic accuracy AI (red) models had per-radiograph accuracies of 1 (95% CI: 0.95, 1) and 0.95 (95% CI: 0.88, 0.99) for images classified as low diagnostic difficulty, 0.85 (95% CI: 0.62, 0.97) and 0.45 (95% CI: 0.23, 0.68) for images classified as intermediate diagnostic difficulty, and 0.39 (95% CI: 0.17, 0.64) and 0.28 (95% CI: 0.1, 0.53) for images classified as high diagnostic difficulty, respectively. (B) Bar graph shows the per-lesion accuracy (n = 63). The high diagnostic accuracy AI (blue) and low diagnostic accuracy AI (red) models had per-lesion accuracies of 1 (95% CI: 0.86, 1) and 0.92 (95% CI: 0.74, 0.99) for nodules classified as low diagnostic difficulty, 0.93 (95% CI: 0.66, 1) and 0.5 (95% CI: 0.23, 0.77) for nodules classified as intermediate diagnostic difficulty, and 0.46 (95% CI: 0.26, 0.67) and 0.25 (95% CI: 0.1, 0.47) for nodules classified as high diagnostic difficulty, respectively.
High-res (TIF) version
(Right-click and Save As)
Thumbnail
Figure 3. (A–C) Chest radiographs obtained as part of a health checkup in a 71-year-old male patient show reader susceptibility to high diagnostic accuracy artificial intelligence (AI). In the first session without AI, a thoracic radiologist with 16 years of experience read the chest radiograph as a normal radiograph (A) . High diagnostic accuracy AI observed potential lung cancer in the radiograph with an 89% CI as indicated by the nodule localization map (B) (as the color changes from blue to red, the probability of the presence of a nodule increases). When presented with the AI suggestion at the second reading session, the radiologist changed the decision and annotated lung cancer in the area that overlapped with the right hemidiaphragm (box annotation) (C) . (D, E) Contrast-enhanced chest CT scans show a 6.8-cm lung mass (arrow) with an air bronchogram in the right lower lobe in the axial (D) and coronal (E) planes. This mass was pathologically proven to be an invasive mucinous adenocarcinoma. Therefore, the reader’s decision was incorrect in the first session but correct in the second session after following the AI suggestion.
High-res (TIF) version
(Right-click and Save As)
Thumbnail
Figure 4. (A, B) Chest radiographs obtained as part of a health checkup in a 65-year-old male patient show reader susceptibility to low diagnostic accuracy performance artificial intelligence (AI). In the first session without AI, a thoracic radiologist with 17 years of experience annotated the mass opacity in the right upper lung zone as lung cancer (box annotation) (A) . Low diagnostic accuracy AI suggested the radiograph as normal (B) . When presented with the AI suggestion at the second reading session, the radiologist changed the decision and determined this radiograph was normal. (C, D) Contrast-enhanced chest CT scans show a 3-cm lung mass (arrow) in the right upper lobe in the axial (C) and coronal (D) planes. The mass was pathologically proven to be invasive adenocarcinoma. Therefore, the reader’s decision was correct in the first session but incorrect in the second session after following the AI suggestion.
High-res (TIF) version
(Right-click and Save As)