Executive Summary

Key Figures

Source: Siddappa, Raju, Al Amin and Ma, Alert-List Concordance for Evaluating Imputation Strategies in ICU Sepsis Prediction (arXiv:2610.22635, October 15, 2026).

The experiment was built to hold everything else still. All ten arms pass through the same four steps: fill the blanks, train the same XGBoost classifier of 100 trees, calibrate the probabilities on 20% of the training data, and raise an alert for the top k% by risk. The classifier's settings, the calibration and the alert rule are all fixed. Only the first step differs.

The scoreboard shows almost no difference. On AUROC, the overall measure of discrimination, ForwardFill stands alone at 0.787, and XGBoostNative, which fills nothing and hands the missing structure straight to the trees, reaches 0.744. The remaining eight sit between 0.722 and 0.733, a band 0.011 wide. That is narrower than the confidence interval of any one of them. Compare the 28 pairs among those eight at the patient level with a Bonferroni correction and 22 of them show no significant difference. If the score is all there is to go on, almost any of them will do.

The other metrics do not read any differently. AUPRC, which does the same job when positives are rare, runs from 0.049 to 0.054 across those eight, and the Brier score, which measures how closely predicted probabilities track observed frequency, spans 0.01733 to 0.01737. The second of those splits only at the fifth decimal. ForwardFill sits at the better end of both, but nothing in any of these metrics puts the other eight in an order. The usual yardsticks for choosing an imputation method are reconstruction error, which asks how closely a held-out value was restored, and aggregate metrics of this kind. Neither asks who was selected.

What the paper does next is the new part. The authors pull out who each pipeline actually named and set the lists side by side. What an alert system hands a ward is not a ranking but a list of people of a fixed size. Take the top 10% of the 12,101 patients in the evaluation set and every method names exactly 1,211. The count is identical. Counting the people who appear in both is where it changes. Of everyone in either list, the share that appears in both, the Jaccard similarity, runs from 0.518 to 0.750.

The pair that overlaps most is DAE and MIDA. Two methods on the same autoencoder architecture, differing only in how noise is applied, and they reach 0.750 with a 95% confidence interval of 0.726 to 0.771. Two near-identical algorithms, and a quarter of the list is different people. Rank correlation hides this. Across all 45 pairs Spearman is high, 0.825 to 0.975, and so is Kendall's tau at 0.633 to 0.872. The overall ordering stays similar, and the methods part company only at the very top of it.

Why filling the blanks carries this much weight is easier to see from the raw material. The PhysioNet/CinC 2019 data comes from the intensive care units of two American hospitals, Beth Israel Deaconess Medical Center and Emory University Hospital. It covers 40,336 patients and 1,552,210 patient-hours laid out in a table at one-hour intervals, with sepsis labeled under the international Sepsis-3 criteria. The model is fed 39 variables: 8 vital signs, 26 lab values, and 5 demographic and admission fields. The patients were split 70 to 30 with the sepsis rate preserved, 28,235 for training and 12,101 held out and never used in training. Those 12,101 are the set whose top 10% was counted in the previous section.

The average missing rate across those 39 variables is 71.9%. The lab values are the worst of it, with 25 of the 26 absent in more than 90% of patient-hours. Glucose is the one exception, and it is still empty 83% of the time. The median gap between consecutive measurements of the same test is 5 hours, with an interquartile range of 2 to 11 hours, and the longest gap is 326 hours.

These blanks are not equipment failures. In an ICU a test is ordered when a clinician judges it necessary. The absence of a value is therefore a trace of clinical judgment, and it carries information about how sick the patient is. A 2019 study in JMIR Medical Informatics, cited by this paper, measured that on its own. On MIMIC-III, the pattern of which values were observed predicts 30-day mortality at an AUROC of 0.68. What someone decided was worth measuring is already a signal.

Filling a blank means writing a number over that signal, and the nine methods compared here write it differently. ForwardFill carries the patient's last observed value forward, and Mean inserts the training-set average for that variable. Early in an admission, where there is nothing yet to carry forward, ForwardFill also falls back on the training average. MICE runs regressions over the relationships between variables ten times to estimate a value. The remaining six are neural networks. DAE, MAE and MIDA are autoencoders, SAITS uses attention and BRITS a bidirectional recurrent network, and GRU-D pulls a value gradually toward the training average as time passes since the last observation. XGBoostNative joins them as the baseline, filling nothing and passing the missing structure to the trees.

ForwardFill stands at an extreme in two places at once. Its AUROC of 0.787 is the highest of the ten arms, and its list overlaps any other method's by 0.40 to 0.45, the lowest of any pairing.

Run the two hospitals separately and the same picture appears. Beth Israel contributes 6,118 patients with 531 cases of sepsis, Emory 5,983 with 349. At one hospital the largest overlap ForwardFill reaches with any other method is 0.409, and at the other 0.440. The pairs that exclude ForwardFill start at 0.466 and 0.454 respectively. The two ranges never touch.

Changing the rule that collapses a patient's hourly risk into a single score does not help either. Everything was recomputed under six of them: the maximum, the mean of the top three hours, the 90th percentile, the last six hours, the first 24 hours, and the overall mean. Among the reconstruction methods the overlap swings widely with the rule, from 0.45 to 0.90, while ForwardFill stays between 0.36 and 0.46 under every one of them. It is an outlier that does not move.

The numbers that reach a hospital dashboard show none of this. At the top 10%, precision across the ten methods runs from 0.257 to 0.298 and recall from 0.353 to 0.410, a precision spread of 0.041. That is roughly four times the 0.073 a random pick would give, and that performance holds whichever method is used. While the list overlap splits almost two to one, between 0.40 and 0.75, the operating metrics move in the second decimal place. A report on a model swap that carries only precision and recall reads as though nothing happened.

Up to here ForwardFill reads like the sensitive one, catching patients early that the others never catch at all. The authors timed the alerts and found the opposite. ForwardFill flagged 111 patients that no other method flagged, and 63 of them crossed the threshold before their first Sepsis-3 label. The median of that head start is 5 hours. For the patients several methods flagged together the same figure is 22 to 26 hours.

The explanation the authors offer is that a property of ForwardFill happens to line up with a property of ICU missingness. Because the method carries the last observed value forward, a bad recent measurement stays in place. And in an ICU, the sicker a patient is, the more often and the more recently they are measured. Where the two meet the score goes up, but that is not evidence of better clinical judgment, as the paper states plainly.

The patients the methods disagree on were examined separately. Among 880 sepsis-positive patients compared across three pairs at the top 10%, the group both methods flagged had the longest ICU stays and the highest risk, with a Kruskal-Wallis p below 0.001 in every panel. The group that catches the eye is the one ForwardFill flagged alone. Their stays are nearly as long as those of the both-flagged group, while the risk scores the other methods assign them are far lower. The number written into each empty cell decided how the same patient was read.

Compute cost points the same way. The sequential neural imputers take 5 to 12 times what Mean takes, even on a GPU. SAITS runs 7.54 minutes against Mean's 0.62, a factor of 12.2. The autoencoders are cheaper at 2.4 to 3.0 times. None of that spending buys an AUROC advantage. Mean imputation is widespread because it is cheap, the authors conclude, not because it has been shown to be clinically superior.

4.1The Lines the Authors Draw Around Their Own Result

What this study does not say is written down in it too.

• • It does not show that patient outcomes change. It shows that the people being flagged change. Whether that shift moves escalation decisions or treatment outcomes is beyond what this data can answer, and prospective shadow deployment is the definitive test, the authors write
• • It is one cohort from two American hospitals. There is no external validation on another cohort such as MIMIC-IV, and whether the same pattern appears at other hospitals is open
• • The time-series imputers were implemented row by row. SAITS, BRITS and GRU-D were designed to see a patient's whole trajectory, and here each row was handled on its own. By the authors' own account this is the comparison's most important limitation, and part of why the neural methods score low
• • Missingness indicators were deliberately left out. Neither the fact that a value is absent nor the time since the last observation was given to the model as a feature. The design isolates the imputed values themselves, and including those features could move where the imputation methods stand against XGBoostNative
• • MICE produced a single dataset. The chained equations ran ten iterations, but only one final dataset was built and nothing was pooled under Rubin's rules. MIDA averages five runs

When a data team changes a model, what it checks is usually the performance metrics: whether the new version's AUROC has held, whether precision and recall are intact. What this paper shows is the question that remains once those checks pass. Metrics holding still does not mean the same people receive the alert. Nothing here changed about the model or the training. Only what went into the blanks.

What makes the preprocessing step unusually dangerous is that it leaves no record. Model architecture and hyperparameters go into papers and internal documents, while which imputation was used often ends as one line somewhere in the code. For a commercial model sold to a hospital, that line is not visible to the customer at all. With no record there is nothing to compare against, and with nothing to compare against the change in the list cannot be discovered in the first place.

There is a well-known precedent nearby. When JAMA Internal Medicine externally validated a commercial sepsis prediction model widely deployed in American hospitals in 2021, it was alerting on 18% of hospitalized patients while missing 67% of sepsis cases. What broke then were the performance figures the vendor had published. This paper goes a step further in. Even when those figures are honest and identical to one another, the people receiving the alert underneath them may have changed.

The checking, fortunately, is the cheap part. No retraining is required. Store the top-k% lists both versions produced and count how much they overlap. For that, three things have to be in the record.

• • Which imputation, under which settings. The name of the method is not enough. With ForwardFill, what filled the stretch before the first observation decides the values; with GRU-D, how fast the decay was set does
• • Whether a missingness indicator was used as a feature. In a domain where an absent value is itself a signal, this one line changes what the model reads
• • The list, not the score. Keep the aggregate metrics and throw away who was selected, and no comparison can be reconstructed later. What has to be stored is the list of identifiers and which version at which time produced it

With those three in hand, the list overlap can be reported alongside every preprocessing change. At 0.9 the two are effectively the same system; at 0.5 they are different systems with the same performance. Miss any one of the three and the question can never be answered. In sepsis alerting the answer arrives at the level of patient names, and in credit scoring or equipment failure prediction it arrives at the level of some other name. The structure is the same.

Thanks for reading this far. The facts in this article were checked against the full text of the paper on arXiv. The data is the public PhysioNet/CinC 2019 dataset, and the paper was accepted to the workshop on data mining in biomedical informatics and healthcare (DMBIH 2026) held alongside IEEE ICDM 2026. If you have ever compared the lists separately when swapping an in-house model or reworking the preprocessing, we would like to hear about it.

Pebblous Data Communication Team
October 20, 2026

Primary Source

• 1.Siddappa, P., Raju, M. A., Al Amin, S. H., Ma, R. (2026). "Alert-List Concordance for Evaluating Imputation Strategies in ICU Sepsis Prediction." arXiv:2610.22635 [cs.LG]. 14th Workshop on Data Mining in Biomedical Informatics and Healthcare (DMBIH 2026), IEEE ICDM 2026.

Background & Cited Studies

• 2.Wong, A., Otles, E., Donnelly, J. P. et al. (2021). "External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients." JAMA Internal Medicine, 181(8), 1065-1070. The Epic Sepsis Model external validation discussed in Section 5.
• 3.Sharafoddini, A., Dubin, J. A., Maslove, D. M., Lee, J. (2019). "A New Insight Into Missing Data in Intensive Care Unit Patient Profiles: Observational Study." JMIR Medical Informatics, 7(1), e11605. Background for the "absence as signal" argument in Section 2.
• 4.Reyna, M. A., Josef, C. S., Jeter, R. et al. (2020). "Early Prediction of Sepsis From Clinical Data: The PhysioNet/Computing in Cardiology Challenge 2019." Critical Care Medicine, 48(2), 210-217. Source paper for the PhysioNet/CinC 2019 dataset used in this study.