Executive Summary

Medical AI privacy has long taken comfort in a single number. Average the question "does this model leak patient information?" across the whole dataset, and the answer came back barely different from a random guess. A paper from a Technical University of Munich (TUM) team, published in Nature on June 24, 2026, shows that this very average lies. When the researchers audited seven real clinical datasets one patient at a time, the group average stayed at random-guess levels — yet certain patients were identified almost perfectly.

The attack is called a membership inference attack (MIA): it guesses whether a given person's record was used to train the model. The crux is who is vulnerable. Patients with rare diseases or atypical clinical presentations, and patients underrepresented along axes like race, sex, or insurance, were exposed far more sharply, because uniqueness itself becomes a signal. More uncomfortable still: the larger the model, the more such high-risk patients there are.

This article builds the intuition for why an average hides individual risk, shows how a single bit ("was in the training set") can become sensitive medical information, and explains why standard defenses like differential privacy are not a cure-all. It closes on why recording rights, consent, and provenance into data is not an ethical slogan but measurable management of an attack surface.

Key Figures

Sources: Nature (Knolle et al., 2026), Inside Precision Medicine

Four numbers carry the weight of this finding. With the same model and the same data, the group average and the individual value point in opposite directions. They also capture how far down the study went to measure risk — one patient at a time — and the counterintuitive result that smarter models do not reduce the risk but expand it.

~50%

group-average attack success

Across the whole dataset it is nearly a coin flip — which is why it looked safe

near-perfect

per-patient identification

Yet isolate one patient and membership is revealed almost perfectly

per-patient

patient-level audit

Risk measured one patient at a time, not as a group average — a first of its kind for medical AI

capacity ↑ → risk ↑

the bigger the model

The larger the model, the more the absolute number of high-risk patients grows

1

The Average Said Safe — Until You Looked at One Person

For a medical AI developer, "does our model leak patient information?" is not an abstract ethics question but a practical checklist item. And that check usually ended in a single line. You measured attack success across the whole dataset, took the average, and if the value sat near a random guess (about 50%), you wrote down "safe." Low average, low risk — that was the logic.

In their June 2026 Nature paper, Moritz Knolle and colleagues at the Technical University of Munich confronted that method head-on. They assembled seven large-scale, real clinical datasets spanning medical imaging, electrocardiograms (ECG), and electronic health records (EHR). For each dataset they trained roughly 200 versions of the AI model, then measured attack success not as a group average but for every individual patient — unfolding the distribution that a single average number had been hiding, down to each person.

The result split in two. The dataset-wide average stayed at random-guess levels, as before. But hidden inside that average were patients who could be identified almost perfectly. While many ordinary records showed low risk and pulled the average down, a small number of extremely vulnerable patients were buried in that same average and vanished from view. In the authors' words, group-level privacy metrics can severely underestimate individual privacy risk.

Per-Patient MIA Success Rate Distribution Patient count MIA success rate (%) ~50% 50–55% 55–65% 65–75% 75–85% 85–100% Population avg ≈ 50% Hidden high-risk patients Most patients (random-level risk) Vulnerable patients (high risk)
▲ Per-patient MIA success rate distribution — the group average looks random-level while a tail of vulnerable patients can be identified almost perfectly | Pebblous original diagram (Fig. 1 reinterpretation, Knolle et al., Nature 2026)

The point: Saying the average is safe means "most patients are safe," not "every patient is safe." Privacy risk lives in the tail of the distribution, not at the mean — and in medical data, the people in that tail are usually the patients who most need protection.

2

Membership Inference: Just Querying the Model Can Leak Who It Trained On

The formal name of the attack used in this paper is a membership inference attack (MIA). It asks just one thing: was this person's record used to train the model, or not? The answer is one of two, in or out: a single bit. Academically, the standard is the likelihood ratio attack (LiRA) family: the attacker separately trains several shadow models and compares how the model's output changes when a record is in the training set versus when it is not.

The part worth dwelling on is that the attacker never needs to get hold of the patient's record itself. It is enough to query the model and observe its responses. By using shadow models to learn in advance how a model's output shifts when a record is included in training, the attacker can then tell membership on the target model from output patterns alone. Even if the original data is encrypted or access-controlled, the trained model itself becomes the channel that leaks the trace. That is why the reassurance of "we protected the data with anonymization and access control" falls apart in front of the model.

How a Membership Inference Attack (LiRA) Works Shadow Models Model A Model B Model N… Learn output patterns Target Model Query Observe API output only No patient record needed Bypasses anonymization ← the core risk LiRA → Decision Member Non-member ① Learn patterns ② Query (no data access) ③ Membership verdict "Did my data train this model?" — now answerable with statistics
▲ LiRA attack flow — the attacker needs only query access to the model, never the patient record itself | Pebblous original diagram (Fig. 2 reinterpretation, Knolle et al., Nature 2026)

2.1Why One Bit Becomes Sensitive Information

You might ask what one bit — "included in training" — really matters. For a model built on the general population, it genuinely doesn't: almost everyone is in some dataset somewhere. The danger appears when the population the model handles narrows.

Consider a model trained on a narrow disease- or institution-specific cohort: a model built from a cohort of HIV-positive patients, a model trained only on patients with a particular rare cancer, a model trained on the records of a single psychiatric hospital. The moment "my record was included in training" is confirmed for such a model, it becomes a direct proxy for "I am HIV-positive," "I have that rare cancer," "I was treated at that hospital." The single bit of membership is translated into a diagnosis, the most sensitive information of all.

Why it matters: The risk of membership inference is decided not by the type of model but by the deployment context. The same attack can be harmless on a general-population model yet, on a narrow-cohort model, turn a single bit of membership into a diagnosis. That is why the authors urge against asking "is this model safe?" uniformly, and instead recommend evaluating, model by model and context by context, exactly what an attacker could actually learn.

3

Why the Average Lies: Uniqueness Becomes a Signal

So why is one patient identified almost perfectly while another stays as safe as a coin flip? The answer fits in a single word: uniqueness. A model generalizes its training data, but the farther a case sits from the distribution (the more atypical it is), the more clearly the model leaves its trace in the output. A common record gets buried among other similar records and cannot be told apart as "in or out," but a one-of-a-kind record changes the model's response itself, and so it gives itself away.

The paper adds one condition peculiar to medical data: a single patient often leaves several similar records. When repeat scans, follow-up exams, and consecutive ECGs from the same person pile up in a dataset, that patient's pattern stands out even more, and the model remembers them more strongly. The closer to the edge of the distribution, and the more a single person's traces overlap, the sharper the boundary the likelihood ratio attack draws between inside and outside the training set.

3.1Disparate Risk — The Most Vulnerable Are the Most Exposed

That is why the word "disparate" sits in the paper's title. The best-identified patients are those with rare diseases, those with atypical clinical presentations, and subgroups underrepresented along axes like disease, race, insurance, sex, and imaging protocol. The very fact that they are few in the data makes them more unique, and the more unique they are, the higher their exposure risk. It is a structure in which privacy risk falls most heavily on groups that are already socially vulnerable.

The most counterintuitive result concerns model capacity. As a model's parameters and expressive power grow, diagnostic performance improves, but the absolute number of high-risk patients rises considerably at the same time. A smarter model means one that memorizes individuals better. Since the very choice that lifts performance also widens the privacy attack surface, the assumption that "bigger model = better model" deserves a second look in the medical context.

MIA Vulnerability by Patient Type Typical patients ~50% Rare disease ~90%+ Underrepresented ~78% Repeat records ~72% random (50%) Model Capacity ↑ → High-Risk Patients ↑ High-risk count Small Large Perf↑ Risk↑
▲ (left) MIA vulnerability by patient type — rare and underrepresented patients are identified far more sharply. (right) Larger models increase the absolute count of high-risk patients | Pebblous original diagram (Fig. 3 reinterpretation, Knolle et al., Nature 2026)

The intuition: Privacy attacks feed on uniqueness, not ordinariness. So the rarest, the most atypical, the most underrepresented people in the data are the ones most easily identified. The lens of the average is exactly what renders them invisible.

4

Can't We Just Use Differential Privacy?

At this point a natural objection arises: can't we just train with differential privacy, specifically DP-SGD? Fair enough. DP-SGD adds noise during training to mathematically bound the influence any single record has on the model, and it is a powerful defense that drives the likelihood ratio attack down to random-guess levels. The problem is that this defense treats every record identically.

As the previous section showed, risk differs from patient to patient. An ordinary patient carries a low, random-level risk; a rare patient carries a high, near-perfect risk. Yet the protection DP-SGD applies is uniform. It acts more strongly than necessary on ordinary patients, shaving off diagnostic accuracy, while it may not be enough for exactly the rare patients who most need protection. This is where a defense that looks at the average diverges from the disparate risk borne by individuals.

Individual Risk Levels (Disparate) High risk Rare patient Low risk Typical patients DP-SGD Protection Level (Uniform) ⚠ Under- protected Over-protected → accuracy loss Rare patient Typical patients Same ε protection → too much for some, too little for others
▲ DP-SGD uniform protection vs disparate individual risk — the rare patient remains under-protected while typical patients face accuracy-shaving over-protection | Pebblous original diagram (Fig. 4 reinterpretation, Knolle et al., Nature 2026)

On top of that, strong differential privacy (a small ε) degrades diagnostic performance. Because medical AI is especially sensitive to accuracy loss, "just add more noise" is no answer. So the authors recommend a context-specific risk assessment before any uniform defense: weigh what sensitive information an attacker could actually obtain from the model and its deployment environment, and protect high-risk models with verifiable risk-mitigation strategies and strict access control.

In one line: Differential privacy is powerful but not a cure-all. If risk differs per individual while protection is uniform, it is too much for some and too little for others. In medical AI, measuring "who is at how much risk" comes before deciding "how much to protect."

5

"Did My Data Train This Model?" Now Has a Statistical Answer

The message this paper sends to the data side is clear. The question "did my data train this model?" is no longer an abstract worry but a measurable fact you can answer with statistics. Privacy risk has moved from the realm of ethical debate into the realm of a number: attack success rate. And that number is computed not as a dataset average but one patient at a time.

At this point, recording the rights, consent, and provenance of data becomes a precondition for risk management rather than an ethical slogan. If you cannot trace and prove which patient's data entered training, under what scope of consent, and by what path, you cannot even gauge how much the model exposes whom. Membership inference attacks bore straight into that untraceability. Conversely, only an organization whose data lineage is clear has a starting line from which to identify high-risk cohorts, control access, and verify mitigation strategies.

This is exactly where Pebblous's emphasis on inscribing rights, consent, and provenance into data connects. AI-Ready Data is not merely clean, refined data; it is data that can explain whose what entered under which promise. This paper shows that the absence of that explanatory power is not an abstract flaw but an attack surface measured one patient at a time. Adjacent themes in data governance continue in other Pebblous pieces — the data moat of medical AI, the Alzheimer's digital twin, and how synthetic data contributes to quality.

To close: The average says "most are safe," but privacy should be judged not by the average but by the risk to the single most vulnerable person. Recording rights, consent, and provenance into data is the most basic infrastructure for identifying and protecting that one person.

R

References

Academic

Industry & Press

  • 4.Inside Precision Medicine. (2026). "Medical AI Model Privacy Risks." Inside Precision Medicine. — Reporting that explains the Nature paper, summarizing key figures and author commentary.