Executive Summary

Key Figures

Sources: the paper published in Nature Medicine (2026) and the University of Chicago Medicine press release (September 14, 2026).

I3LUNG is an international collaboration funded by the European Union's Horizon Europe programme. Five centers in Italy, Germany, Greece, Israel, and Spain took part alongside the University of Chicago Medical Center, and together they assembled real-world clinical data on 2,396 patients with advanced non-small cell lung cancer enrolled between September 2012 and October 2023. The authors call it the largest international multi-center multimodal AI study on real-world data to date. The analysis cohort, first-line and later-line patients combined, came to 2,075, split into 1,550 for training and 274 for an independent set, with 251 patients from the geographically separate US center held aside for external validation.

Start with the part that held. On the independent set, the explainable model built on clinical records and blood tests reached an AUC of 0.77. AUC is a score where 1 means a perfect call and 0.5 means a coin flip. Measured on the same patients, the markers a clinic reaches for today came in at 0.53 for PD-L1, 0.62 for ECOG performance status, 0.58 for LDH, and 0.66 for the neutrophil-to-lymphocyte ratio. The model beat all of them by a statistically significant margin. The head-to-head against those markers, though, happened on the independent set. For the external validation cohort at the US center, the paper reports only the model's own scores, and those fell to between 0.55 and 0.72. The lowest of them, predicting disease control, sits at 0.55 with a confidence interval running from 0.48 to 0.62, which contains 0.5.

Trouble shows up on the next layer. The team also built a multimodal model that added radiomic features extracted from CT scans and digital pathology images, and compared it against the first. The 0.88 the press release quotes belongs to that model. Yet the same results section carries another sentence. The multimodal improvement was not consistently reproduced on the independent set, it says, and reproduced only inconsistently in the external validation cohort. The discussion repeats the judgment. Adding imaging-derived data pushed the cross-validated model up to an AUC of 0.88, the authors write, but the gain was not consistently reproduced in the independent and external validation sets, and they attribute that to the limited multimodal data.

The two sentences do not contradict each other. They point at different places. The 0.88 was measured by folding the training data five ways while the model was still being built, and the finding that it did not reproduce is about what happened when that model met patients it had never seen. The diagram below lays the two models over the same three columns to show how far each one got.

Cross-validation splits the data you have into several pieces and takes turns using one piece as the exam paper. Because every piece serves as the exam once, a small amount of data still yields a stable estimate. The catch is that all those pieces come from the same pool of patients, gathered the same way. An independent set works differently. It is an envelope sealed before the model is built and opened once, at the end. External validation goes a step further and brings in data that accumulated at another institution under other habits.

This study's cross-validation sits on the strict end. The pieces were split by center rather than by patient, so even inside training the model had already sat an exam where one whole center was left out and then used as the test. The decision threshold stayed fixed at 0.5 when the model moved to the independent set, untouched. What follows, then, is not a story about a loose development stage producing flattering numbers.

The paper says outright why the multimodal model was only ever compared in cross-validation. The independent set was small. Short as that explanation is, it points precisely at the sorest spot in the design. A multimodal comparison can only use patients who have every required piece of data. Four hundred patients had CT and digital pathology on top of records and blood, and 339 had all four types including genomics. That is 14% of the 2,396 enrolled. The independent set holds 274 patients in total, so filtering it down to those eligible for a multimodal comparison leaves too few to run a statistical test on.

The floor of that comparison matters as much as the ceiling. The paper weighed the multimodal model not against the records-and-blood model on the full cohort but against a records-and-blood model restricted to the same small subgroup. That is a choice made for fairness, and the floor it sets is low. The 24-month survival prediction that scored 0.74 on the full independent set falls to 0.60 in that subgroup's cross-validation. Narrow the subgroup further and the gap widens, so that among patients who received immunotherapy alone, 0.91 faces 0.54. A 0.54 is next door to a coin flip. This is where the question of how far one side rose has to be paired with the question of why the other side fell.

The name the paper attaches to the comparison is equally plain. The results section in question is headed by the claim that multimodal integration showed exploratory gains, and the abstract records that the added benefit was uncertain and did not carry over to the independent and external validation sets. In the statistical methods section the paper goes further. So many subgroup and pairwise comparisons were run, it says, that those analyses should all be treated as hypothesis-generating; no correction for multiple comparisons was applied, so every p value is nominal; and the primary evidence of generalizability is the independent and external validation results. The p value printed next to the 0.88 is one of those nominal ones.

Changing the model architecture changed nothing. Of the two model families the team ran, the deep learning branch gained no benefit at all from the extra data, and the paper records that it saw this consistently across every endpoint and every subgroup. Genomic data carried too many missing values and too much inconsistent encoding to contribute. An additional analysis that re-extracted the pathology features with a different foundation model repeated the same pattern. On that basis the authors point at the data rather than the architecture.

Prior multimodal studies cited directly in the paper enrolled 200 to 300 patients, with complete multimodal cases numbering around 80, and most were single-center work. They reported handsome numbers, AUCs of 0.80 and 0.81, with neither an independent set nor external validation. Another study cited in the same discussion predicted response from tissue slides alone, and the 0.75 it scored on its own data became 0.66 in external validation. Even with a single modality, moving institutions still produces a drop of the same shape. I3LUNG is the first large study to put a model through both gates, and it is at those gates that the multimodal advantage disappeared.

The most widely quoted scene in this study is not a table of model scores but an experiment that sat doctors down. Ten thoracic oncology specialists and ten non-specialists, twenty in all, were given the records of 100 real patients, 80 drawn from the independent set and 20 from the external validation cohort. In the first round each doctor judged without AI whether the patient's disease would be controlled. In the second round they saw the AI's prediction for the same patients along with an explanation of what drove it, and judged again.

The result was clear. Sensitivity at picking out responders rose from 0.72 to 0.87 (p=0.0011), and accuracy from 0.57 to 0.65 (p=0.0431). Non-specialists improved more than specialists, moving from 0.77 to 0.90 against 0.68 to 0.83. The kappa statistic, which measures how far two groups agree, climbed from 0.11 to 0.48, crossing from slight agreement into moderate agreement. The claim that the tool helps most at hospitals thin on specialist staff rests on these numbers.

Other things sit in the same paragraph. Sensitivity rose, and specificity slipped a little. Converted into the odds of a single judgment being right, the rise came to 37%, with a p value of 0.1, which leaves it statistically insignificant. Doctors accepted 74.5% of the AI's correct suggestions, and the rate at which they followed its incorrect ones ran higher among specialists, 72.2% against 63.6% for non-specialists. The share of survival-duration predictions that matched what actually happened went from 16.5% to 22.5%. The direction in which the tool pushed judgment is clear, and the size of the push is written cautiously inside the paper itself.

What matters here for this article is which model went into that experiment. Describing the clinical utility study, the paper nails down one sentence. No multimodal model was used in it. The AI output the twenty doctors saw on screen was the prediction and explanation of a regression-family model running on clinical records and blood tests alone. The images themselves were on screen. Of the 100 patients, 35 had a viewable CT and 42 had a viewable tissue slide, and the doctors had that material in front of them in both rounds. What was missing was the part that reads an image and turns it into a prediction.

3.1Where the Press Release Blended Two Metrics into One Sentence

The University of Chicago Medicine press release carried the passage over like this: using the AI tool improved sensitivity in identifying responders from an AUC of 0.72 to 0.87. Sensitivity and AUC measure different things. Sensitivity is the share of actual responders the doctors found, while AUC is the area under a curve traced by sliding the decision threshold. In the paper, 0.72 and 0.87 are sensitivities, and the release put the name AUC in front of them. The same release also writes that the model adding imaging and digital pathology to records and blood produced 0.88, without writing alongside it that the figure came from cross-validation and did not reproduce on the independent set.

First, the authors point at a difference in population makeup. The US center's cohort was 59% women, where the training cohort was 34.6%. Median survival split too, 22.2 months against 12.4. Baseline characteristics, treatment outcomes, and the pattern of treatment itself belong to a different population of patients. The patients the model learned from and the patients it was examined on did not come from the same distribution.

In the same paragraph the authors attach a caveat. When distributions diverge this far, the predictive relationships can hold while the discrimination metrics still wobble. A lower score on its own, in other words, is not enough to declare the model wrong. The paper also raises, before anyone else can, the possibility that analyzing only patients with complete data invites selection bias, and reports that a sensitivity analysis run specifically to check for it turned up no evidence of systematic bias.

Limitations the paper writes down for itself make a longer list. That list names a retrospective design that carries the heterogeneity of real-world practice as it is, too few complete multimodal cases to trust that part of the analysis, a model architecture that does not bend gracefully around missing data, radiomic features extracted only from the primary lesion, roughly 15% of CT scans not timed to the start of immunotherapy, and external validation resting on a single cohort.

The spread between institutions showed up as a number in the fairness analysis too. Sensitivity at finding patients whose disease would be controlled came to 0.96 at the Greek center and 0.63 at the Italian one, and the difference was statistically significant (p=0.037). Same model, same criteria. The authors put the cause down to data quality and sample imbalance, and leave it standing as an observed fact rather than a diagnosis.

The sentence the authors chose for their conclusion is restrained. Multimodal integration remains promising, they write, but its added value has to be validated. This paper is the retrospective arm of I3LUNG, and a prospective arm enrolling more than 2,000 new patients is under way. A clinical utility study on multimodal models and a randomized trial for regulatory approval are both on the plan, with deployment seen two to three years out. What has come out now is less a verdict that multimodal is useless than a report that the verdict is not in.

From here on we are outside the paper. The authors never pointed at scanner equipment or reading habits to explain why the multimodal gain failed to survive. What they named was the difference in population makeup and the shortage of multimodal data. That data varies by institution, though, is written all through the methods. The design was multi-center and real-world, the paper says, so CT acquisition was heterogeneous, and scans were screened out when the reconstruction kernel was non-standard, when slice thickness fell outside 1 to 5 mm, or when metadata was missing. Manufacturer, slice thickness, and contrast use were tabulated per center. Of 998 pathology slides, 64 were dropped for insufficient tissue, staining quality, and scanner artifacts, and stain normalization was applied to the rest. There was alignment work because there was something to align.

Imaging is not the only thing whose values move. Discussing the sensitivity spread between institutions, the paper notes that ECOG performance status is clinician dependent and maybe institution dependent, and that this drove the variation across centers. ECOG is the variable the explainability analysis flags as the one the model leans on hardest. In the external validation cohort, 2.4% of patients were recorded as ECOG 0; in the training cohort it was 30.3%. The patients may genuinely have differed that much, or habits of writing the same state down differently may have mixed in. The authors hold that center-specific bias is not large, on the grounds that a model trained to guess the institution discriminated poorly.

So we read this result not as a difference between cheap data and expensive data, but as a difference between data whose standards have been settled and data whose standards have not. Records and blood cleared the independent set, and not because that material was clean to begin with. The team started from more than 11,000 fields in the electronic case report forms, kept the 227 that two teams working without sight of each other picked identically, and finally let two senior medical oncologists narrow that down to nine. Validation reconciling missing values, entry errors, and inconsistencies between institutions ran three times with the participating centers. The first sentence of the paper's conclusion does not say real-world clinical data either. It says curated real-world clinical data.

Organizations stacking logs, sensor traces, and images together mostly walk the same order. Build a first model from the data that arrives easily, then bolt on the expensive, heavy data to push performance higher, and in the development environment the score usually goes up. Whether that score survives in production is a separate question, and this paper packed into one publication a case where trying to answer it produced two different answers. How much evidence a hospital actually demands of an AI tool is a question we covered separately.

The questions below move this study's design onto an organization's own work. They are not a checklist printed in any document.

• Where did the evidence come from that the data you are paying to expand improves performance? Cross-validation, or a sealed independent set?
• Each time you bolt on a new data type, how many subjects drop out of the sample that has every field? Is that loss written in the same table as the performance gain?
• Does the same data read the same way at another site? Have you checked how much the equipment, the measurement procedure, and the labeling habits ride along inside the values?
• Have you ever broken model performance out by site? A gap the size of 0.96 against 0.63 can hide behind an overall average.

This is why Pebblous asks about agreement on standards before volume collected when it talks about AI-Ready Data. Two pieces of data carrying the same name are different data to a model, depending on where and how each was made. Add types without settling that difference, and performance rises only in the development environment, by exactly as much as you added.

Thank you for reading this far. Every figure and sentence this article carries over can be checked by anyone in the original Nature Medicine paper. What is the most expensive data in your organization? If you can say whether the evidence that it lifted performance exists outside the development environment, the most eye-catching number in this paper becomes somebody else's problem.

Academic

• 1.Prelaj, A., Miskovic, V., Sacco, S. et al. (2026). "Clinical usability of an explainable AI decision support tool and evaluation of multimodal models in NSCLC." Nature Medicine 32, 3235–3247. — The published I3LUNG study; the primary source for this article.
• 2.Prelaj, A., Miskovic, V., Sacco, S. et al. (2026). "Clinical usability of an explainable AI decision support tool and evaluation of multimodal models in non-small cell lung cancer patients treated with immunotherapy: the I3LUNG study." medRxiv (preprint, not peer-reviewed). — Some figures differ slightly from the published version (e.g., abstract AUC≈0.74 vs. 0.77 in the final paper).

Press

• 3.University of Chicago Medicine. (2026). "AI decision support tool improves oncologists' ability to identify patients likely to benefit from immunotherapy." EurekAlert!. — Labels the physicians' sensitivity gain as "AUC," and does not mention that the multimodal model's AUC of 0.88 came from cross-validation.