Executive Summary

Key Figures

Four numbers belong together in this paper. One of them got better, and the other three got worse.

Source: Li et al., Evaluating Type-1 Diabetes Digital Twins by Decision Quality (arXiv 2610.30577)

Type 1 diabetes is an autoimmune condition in which the body stops producing insulin. It means a lifetime of delivering insulin and checking glucose often, or, in the paper's phrasing, running by hand the control loop a healthy pancreas runs on its own. Three doses have to be kept in balance: the basal insulin that covers the body's background need, the bolus insulin given in proportion to the carbohydrate eaten, and the correction insulin given in proportion to how far glucose sits above target.

A digital twin is a model that tries this control outside the body first. Instead of trying a new dose on the patient, it applies the candidate doses to itself and produces a virtual glucose curve. From that curve it computes something like how long glucose spends in dangerous territory, which puts the candidates in order, and that order supports the clinical judgment. This is where the benefit to the patient comes from. It comes from choosing which option is safer.

The numbers used to grade such models, however, mostly look at something else. They average how far the predicted curve sits from the real one at each point in time, which is what RMSE and MARD measure. What this paper puts on trial is not a model but that habit. If pointwise error goes down, does treatment choice get better along with it? Rather than leave the assumption as an assumption, the authors measured both side by side.

The experiment ran on a simulator rather than on people. The software is simglucose, an open-source Python implementation of the UVA/Padova type 1 diabetes model that the US Food and Drug Administration has accepted. The authors used the 30 virtual patients that ship with it, split evenly into 10 adults, 10 adolescents and 10 children. Because the simulator holds each patient's true response, it is possible to check all the way through whether the order a model produced was right or wrong. That check cannot be run on people.

The 21 candidate treatments were kept simple. Starting from a reference regimen, either the meal bolus or the basal rate, one at a time, was scaled from −50% to +50% in steps of 10%, which gives twenty combinations, and the unchanged reference adds the twenty-first. For each patient the digital twin was fitted on that one observed reference treatment and then asked to predict the glucose curves of the remaining twenty. The predicted curves were scored with the Magni risk kernel to produce the model's ordering of the twenty. The true ordering came from running those same twenty on the same virtual patient in the simulator and scoring them with the same function.

How much of the patient each model got to see differed by method. The three that estimate the parameters of a physiological model (MCMC, SBI, UKF) saw only 264 continuous glucose readings covering 22 hours, while the two that learn directly from data (Linear, Neural) received all 8,640 readings from 30 days. The baseline, Prior, saw none of the patient's glucose at all. That difference returns later, because the method handed the most data is the one that slips the furthest on treatment choice.

There are two scorecards. One holds the accuracy metrics used until now, RMSE and MARD, which measure the distance between the predicted curve and the real one. The other holds decision quality. Spearman correlation asks how far the model's ordering runs in the same direction as the true ordering, and the regret norm converts, per patient, how much worse the model's chosen treatment is than the genuinely best one. Spearman is better near 1, and the regret norm is better near 0.

The six methods scored as follows. The figures in parentheses are the rank on that metric, and the last column is how far a method moved from its accuracy rank to its decision quality rank. The paper also reports MARD, a second accuracy metric, which is left out of the table because the contrast this article follows holds there too. The neural network's 31.5% and the baseline's 52.7% sit apart in the same direction as the gap seen on RMSE.

Put the accuracy ranking on the left, the treatment choice ranking on the right, and join each method to itself, and the mismatch becomes visible at once. A flat line means the two yardsticks are saying the same thing, and a sloping line means they are saying different things.

One point blurs easily, so it comes first. The method that predicted glucose most accurately among the six is not the neural network but MCMC. At 23.0 mg/dL it is first on accuracy, and at a Spearman of 0.861 it is first on decision quality as well. The two yardsticks do not always disagree. The reference point that makes the neural network's 32.9 mg/dL read as "close to half" is not the best score in the table but Prior's 63.6 mg/dL, posted by a model that never looks at the patient.

The mismatch opens up in the lower two rows. The neural network falls from third on accuracy to fifth on decision quality. A Spearman of 0.049 cannot be told apart from putting the twenty treatments in an arbitrary order, and the benefit it gives up, 0.541, is more than six times MCMC's 0.083. On the other side Prior climbs from sixth on accuracy to second on decision quality. That model is the MCMC pipeline with the patient identification step taken out. It replays the candidate treatments over population-average values drawn from the published ReplayBG prior, and reads not one of that patient's continuous glucose measurements.

Raising model complexity also paid off on one side only. Linear and Neural differ in nothing but the model itself, a linear regression swapped for a multilayer perceptron, with the same inputs and the same 30 days of data. The neural network improved clearly over Linear on RMSE, and the difference was statistically significant. What happened on Spearman, though, is not an improvement. The model moved from inverting the ranking (−0.207) to carrying no information about it at all (0.049).

The statistical tests belong here too. A Friedman test rejected equality among the methods on all five metrics. Between the neural network and Prior, the network is 30.65 mg/dL better on RMSE and 0.717 worse on Spearman, with p below 10⁻⁴ in both directions. Stretching "Prior came second" into "Prior beats SBI", however, goes past the paper. The two values, 0.766 and 0.728, cannot be distinguished statistically (p=0.3931).

The paper does not treat this result as an anomaly. It explains the divergence as what follows when two metrics are built to penalize different things. The authors' reasoning runs like this. Regret computes both of its terms on the true glucose trajectories. The twin's prediction enters that computation only through which treatment it selects. So a twin whose predicted glucose is biased in one direction throughout picks up no penalty for that bias, as long as the bias does not change its choice. RMSE and MARD charge for the same bias at every sample. Spearman extends the property to the whole ranking: an error that preserves the order of the predicted scores costs nothing.

That one property explains how Prior reached the upper half. A model that only imitates the population average produces a curve that fits no particular patient well, which is why its pointwise error is last. But the direction holds even at the average level: more insulin brings glucose down, less brings it up. A question that asks only for the relative order among candidates can be answered fairly well from that direction alone. The curve is wrong, but wrong consistently.

The paper attaches a scope to that explanation. The authors write that rankings over these 20 treatments are largely set by the direction of the average dose response, and that personalization refines the ordering rather than defining it. One of the limitations in section 5 meshes with this. Because each candidate moves a single dose axis at a time, the design enlarged the share the average could carry, and whether the same result holds once candidates move both axes together is not something this experiment answers. The baseline's score is therefore not evidence that personalization is unnecessary. It is a record that over this candidate set, personalization contributed less than the accuracy numbers suggest.

The neural network is the mirror image. It was trained on 30 days of data to shrink the error at every sample, so its curve comes close. But nowhere in the loss it minimized is there a term saying that candidate A should score above candidate B. Predictions that are slightly wrong in a different direction for each patient can leave the average error almost untouched and still shuffle the order of twenty candidates. There was never a guarantee that learning to be accurate and learning to keep an order are the same task.

How far this result may be pushed is set out separately in the paper. The first risk it lists applies to the readers of this article as well. It is mistaking a strong score on this benchmark for clinical validation. The experiment happened entirely inside a simulator, and nothing in it gives any model grounds for use on a real patient.

• Glucose was clipped to between 40 and 400 mg/dL for scoring. The purpose is to stop a single deep hypoglycemic episode from dominating the whole score, and the price is that 39 mg/dL and 15 mg/dL receive the same score
• Of the 6,000 simulations run to produce the ground truth, 205, or 3.4%, left the simulator's safe range and terminated early. They were concentrated in 8 patients and were almost all treatments that raised insulin
• The ReplayBG prior that Prior draws on was evaluated on adults, so there may be additional mismatch for the 20 adolescents and children
• Candidate treatments move a single axis at a time. Real clinical adjustments move basal and meal insulin together, and sometimes in opposite directions
• Every result came from one candidate set, one risk function and a 24-hour horizon. How much of the mismatch survives a different objective, such as time in range, or a longer horizon, is something the authors write they do not know

That last item follows the result into any other field it is carried to. The observation that accuracy and decision quality come apart arose in one setting. How wide the gap runs has to be measured, and "it has to be measured" is the point of the paper.

It is a story about a type 1 diabetes simulator, but the structure is common well outside diabetes. The side that builds a model reports performance as a continuous, easily computed number such as prediction error or accuracy, and the side that receives the model uses its predictions to choose something. Which equipment to inspect first, which stock to replenish first, which case a person should look at again. The number being reported is a distance, and the capability actually used is an order.

When the two numbers point the same way, nothing goes wrong. MCMC did exactly that in this paper. The trouble is that when they come apart, the fact does not show up in the report. On a dashboard carrying accuracy alone, the neural network looks like a model roughly twice as good as the baseline. That it is choosing treatments no better than chance appears nowhere on that screen. Nobody gamed the metric. A metric that measures something other than the use was simply being measured, over and over.

If model performance reaches you as a single line of score, four things are worth checking.

• Can you write in one sentence the decision your organization actually makes from this model's output? Until that sentence exists, there is no way to settle which metric is the right one
• Is the score you receive measuring the same thing as that decision? Whether a distance metric has been fastened to a use that asks for an order deserves a separate check at least once
• Are both numbers going up the chain side by side? What the paper recommends is not dropping accuracy but writing decision quality next to it
• Are you comparing against a baseline that uses no information at all? Second place on decision quality in this paper went to a model that read not one individual data point. Without that kind of baseline, there is no way to tell what your model gained from the data

None of the four is about changing the model; each is about adding one more place where the model is measured. In work on data quality and model performance, the most expensive mistake is not looking at a wrong number. It is reading a correct number as the answer to the wrong question. This paper has gathered the cost of that mistake into a single table. It is the scorecard of a model that nearly halved the baseline's prediction error and still held almost none of the treatment ranking.

Thank you for reading this far. The figures in this article were checked directly against the body and tables of the paper published on arXiv, and the passages the authors wrote down as limitations are carried over as written. If the model score in front of you is one number, we would like to hear how you check whether it runs in the same direction as the quality of the actual decision.

Pebblous Data Communication Team
October 26, 2026