Executive Summary

Key figures

Source: Moon, Tamura & Guha, arXiv:2608.11491 §2, §3 and supplementary Table 3

Algorithmic fairness has long treated group gaps as a defect in the data and the model. Correct the training data, match calibration across groups, impose constraints that equalize error rates, and the gap shrinks. What the authors point out is that this conversation rests on an unstated assumption that resources are sufficient. It asked whether the algorithm lines people up correctly, but not what happens when there is almost nowhere to put the people it lined up.

The ground this argument stands on was cleared by sociology long ago. Charles Tilly described how organizational procedure turns small differences between groups into durable advantage, and Robert Merton called the process by which early advantage accumulates the Matthew effect. Algorithmic research has likewise noted that accurate prediction is no guarantee of better outcomes, and that the context a tool lands in decides the result alongside the tool. Even where bias was confirmed, as in the 2019 Science study that exposed racial disparity in a health risk prediction algorithm, correction stayed focused on adjustments inside the algorithm and never moved to the point where accuracy meets a shortage of resources.

In homelessness services, child welfare, and cancer treatment referral, demand exceeds supply by orders of magnitude. Under that condition allocation becomes rationing rather than matching. The system has to rank and then draw a cutoff that removes most of the queue, and selection happens in the extreme tail of the score distribution rather than around its middle. The statistics of the tail behave differently from the average statistics of classification.

The intuition the authors offer runs like this. Inaccurate ranking scrambles selection near the cutoff because of noise. That scrambling leaves a nonzero selection probability even for the group the allocation structure places lower. When the system sharpens, the randomness vanishes and selection in the tail converges, in expectation, on the group sitting closer to the cutoff. Inaccuracy has been working all along as an unintended equity buffer, in the paper's phrasing.

The diagram below shows how the composition above the cutoff changes when the same amount of resource is allocated and only ranking precision differs. Four people are selected on either side.

The result also changes how bureaucratic discretion looks. Caseworker discretion has long been treated as inefficiency, and the move to structured assessment tools has been justified as its correction. Without denying the harms discretion has caused, the authors say the noise those tools removed was doing structural work the tools themselves cannot replace. Justifying high precision ranking under scarcity in the name of equity turns out to be harder than it looks.

The model is simple. The population splits into two groups A and B, and each group's latent risk Z follows a normal distribution that differs only in its mean. The difference between the two means is the structural gap Δ. What the allocator actually sees is not Z but a score Y, written as Y = ρZ + √(1−ρ²)ε. Here ρ is the correlation between the score and true risk, and the paper reads it as rank discrimination fidelity rather than calibration. Cutoff selection responds only to relative order, not to the absolute value of a score.

The relative gap D is the ratio of the two groups' selection probabilities, P(Y_A > t) / P(Y_B > t). Since t is the cutoff that scarcity sets, it gets pushed further into the right tail as σ, the share of the population that can be served, approaches zero. Using a tail approximation based on Mill's ratio, the authors obtain ln D = t·ρ·Δ + O(1), and exponentiating gives the following form.

That the three terms multiply inside the exponent is the whole of this formula. When resources are ample and t is small, raising accuracy moves the gap only gently. When resources are severely short and t is large, the same increment of accuracy produces a far larger gap, because sensitivity to accuracy works out to ∂D/∂ρ ≈ t·Δ·D. A precision gain that would go unnoticed in abundance is amplified exponentially under rationing, and that is what the name accuracy trap means.

How large t grows is set by the share that can be served. If the score distribution is close to standard normal, the correspondence the paper notes makes t the z value at the point that cuts off the top few percent. When half the population can be served t sits near zero, and when only 5 percent can it climbs toward 1.6. Inside this formula, a budget cut shows up as one of the multiplied terms in the exponent growing larger.

To check that the result is not an artifact of the normality assumption, the authors also solve the case where scores follow a lognormal distribution. There the relative gap becomes a power law, D ∝ T^(ρΔ). The functional form changes and the direction does not. Raise scarcity or raise accuracy and the gap grows monotonically. A link useful to practitioners sits in the appendix as well. Under the equal variance normal model AUC = Φ(ρΔ/√2) holds, so a deployed model's AUC together with a measured Δ lets you back out ρ.

The appendix also answers the objection that the formula holds only under ideal conditions. In practice it is the share that can be served rather than the cutoff that is fixed, so the cutoff moves as accuracy rises. The authors solve that case separately and find that tail behavior dominates in the scarce regime, leaving the mechanism intact. They also compared the approximation as t goes to infinity against exact values at finite cutoffs and confirmed convergence. Earlier work treated the same phenomenon for finite allocation cases with discrete combinatorics, and this paper extends that result into a continuous asymptotic form.

This validation involved no deployment of a new algorithm in the field. The authors injected noise after the fact into scores already in operation, forming Y = ρR + √(1−ρ²)ε. ρ = 1 is the deployed model's current operating point, and lowering ρ produces a less precise ranking. It is a procedure for measuring how much allocation outcomes swing as ranking precision changes.

There are three stages of validation. First came a Monte Carlo simulation of 20,000 people. Two groups of 10,000 each were given latent risk means of 0.8 and 0.3 with a standard deviation of 1, and ρ was moved from 0.2 to 1 with 25 repetitions at every combination with scarcity. Because the assumptions hold exactly there, the confidence intervals are tightest and the curve sits closest to theory. The other two use scores in live operation, and whether those two curves follow the simulation's rising line was the crux of the validation.

3.1Canadian child welfare

The data are 37,201 narrative case notes on 583 families collected between 2022 and 2025 at the Children's Aid Society of Toronto, with research ethics board approval. A local LLM, Meta-Llama-3.1-8B, classified whether each record showed progress toward service goals as 1 or 0, and each family's risk score was defined as one minus the share of notes aligned with the goal. The further from the goal, the higher the score. This classifier was reviewed against 6,031 notes by child welfare practitioners and researchers in earlier work.

The inner Toronto team, which covers the urban core, carried families with a mean risk score of 0.443 (95 percent confidence interval 0.408 to 0.478), against 0.385 (0.363 to 0.406) for the outer Toronto team. The measured structural gap Δ is 0.058. Toronto is a city where poverty clusters by district, and earlier work identified district as a significant predictor of variation in child welfare service, so the difference between the two regions serves as a proxy for Δ. Relative gap estimates were stabilized with 1,000 bootstrap replications.

The resource being divided here is caseworker time and support services. A record showing no progress toward the goal reads as a signal that safety concerns remain unresolved, and putting staff on those families first is what this score is for. The sample splits into 178 inner Toronto families and 405 outer Toronto families. Converted to an effect size, the difference between the two regions is 0.261 (0.084 to 0.441), not a large value, but the confidence interval does not contain zero. The gap is already visible in the data before any allocation analysis runs.

3.2Breast cancer treatment referral in the United States

Breast cancer mortality rises when treatment is delayed, and short resources lengthen that delay. Deciding who gets referred first is therefore an act of allocation.

The second field is the U.S. SEER cancer registry. A gradient boosting model trained on breast cancer cases diagnosed between 2012 and 2021 produced a five year breast cancer specific mortality risk score. Mean predicted risk was 0.325 (0.324 to 0.327) for 118,662 non-Hispanic white patients and 0.418 (0.414 to 0.422) for 16,820 non-Hispanic Black patients, giving a Δ of 0.093. The same post hoc noise injection was repeated across 25 independent simulations. Two fields that share neither country nor population nor model structure traced the same rising trajectory as the 20,000 person Monte Carlo simulation.

That trajectory was not drawn by feeding in a race variable. The SEER model left race out of its training features. Age, stage, grade, tumor size, hormone receptor and HER2 status, number of nodes examined and number of positive nodes went in, and race did not. The choice was made to establish that the observed gap comes from the interaction of accuracy and scarcity rather than from taking race as a variable. On held out data the model's AUC was 0.8471 (0.8370 to 0.8564).

The table below shows how the two groups' selection probabilities separate as the share that can be served falls, measured at the deployed model's operating point. When 95 percent of the caseload can be served the two groups are barely distinguishable. In the band where only 5 percent can be served, the child welfare gap widens to 2.86 times.

The table shows a single operating point of the deployed model. Redraw the same picture at lower accuracy and, across all three domains, the gap curve for a low ρ model lies almost flat as resources shrink, while it steepens as ρ rises. This is where the sensitivity formula shows up in real data, and where it becomes clear that accuracy, and not scarcity alone, sets the slope of the gap.

In the extreme scarcity band both groups have small selection probabilities, so the ratio may look unstable. The authors answer that concern by publishing the absolute probabilities alongside. Even at the 5 percent band the child welfare figures are 0.091 against 0.032 and SEER is 0.088 against 0.045, so the denominator does not sit at zero. They set down the limits themselves as well. The child welfare confidence intervals are wider than in the other two domains because the extreme tail holds a small sample, and both validations sit inside public welfare systems, which leaves open whether the same phenomenon appears in market allocation or at moderate scarcity.

The authors also state up front that the model uses a single shared ρ for both groups. In reality a group with sparser records often has lower measurement quality and therefore a lower ρ. They expect the trap to deepen rather than ease in that case, and offer the current shared ρ result as a lower bound on the gap real systems would show. In an appendix simulation, lowering ρ for group B alone pushed B's observed score mean toward zero and separated the two groups further.

The easiest thing to misread in this paper is the definition of groups A and B. The paper defines A as the group the allocation system structurally favors, and states that this definition may diverge from marginalization in the historical sense or invert it. In both empirical cases A is the side with higher risk scores. In child welfare it is the families carried by the inner Toronto team covering the poverty dense urban core, and in SEER it is Black patients with higher predicted mortality risk. Under a rule that gives resources to high risk first, they are the ones standing above the cutoff.

The trap is therefore neutral about the normative character of the resource being allocated. Two things outside the model decide who benefits and who is harmed by the amplified gap. Whether the resource is a benefit or a burden, and whether the group in question sits above or below the cutoff. If the intervention is support, the lower group goes without the help it needs. If the intervention is a burden such as surveillance or investigation, the upper group absorbs that burden in concentrated form.

This distinction matters in practice because it changes the direction of the check. Confirming that your model does not assign lower scores to a particular group will not catch this trap. What has to be asked is not the direction of the score but where the cutoff is drawn, who is left below it, and whether what gets cut off is support or burden.

The authors' choice of relative rather than absolute gaps comes from the same place. Under rationing every group has a small selection probability, so an equity metric built on absolute differences reports a small number no matter how the system allocates. The politically and morally important question is not how often a given group is selected but how the chance of selection flows across the population. The authors draw here on earlier work arguing that equity should be treated as a matter of power and resource distribution rather than a statistical constraint. When D moves from 2 to 10, what the lower group experiences is a partly open system closing in practice, while the change in absolute probability may look unremarkable.

Public sector algorithm audits today mostly look at one thing. Whether error rates or score distributions differ across protected groups. The authors call that audit necessary but not sufficient. A fairness intervention that equalizes false positive rates somewhere in the body of the score distribution says nothing about what happens in the tail, where rationing actually occurs.

What they propose adding is an audit of allocative volatility. It refers to how sensitively relative access probability responds when accuracy and supply change. At the operational level it becomes a matter of computing D(σ, ρ) over a range of plausible scarcity and accuracy combinations and putting the result in the report. It is a stress test that shows in advance where equity breaks when budgets fall.

Once exponential sensitivity is visible there are two options. One is to introduce deliberate imprecision. Banding is offered as an example, grouping everyone above a given quantile at the same priority and setting the order within the band by lottery. Fine grained risk discrimination falls, and so does the exponential amplification the paper predicts. The other is to touch scarcity itself by increasing supply. Whether to accept that trade is a policy question rather than a technical one, and the authors write that the paper's role ends at making the choice visible. Optimizing the ranking algorithm alone cannot resolve a gap that arises from the interaction of precision and scarcity, reads the closing sentence.

The argument for adding randomness did not originate here. A position paper arguing that machine learning based allocation of scarce resources should be randomized was already on the table in 2024, and what this paper adds is the speed. Once you can compute where and how fast the gap opens, whether to mix in a lottery moves from a matter of taste to an argument over numbers.

Computation is not the obstacle. The simulation and both empirical analyses in this paper ran on a single laptop, in a form that can be rerun with fixed random seeds. Measuring allocative volatility is less a question of having the infrastructure than of deciding to measure.

If you build these scores, or sit on the review that approves them, three questions are worth asking right now.

• Is this score used for classification or for allocation? A score used to identify a risk group and a score used to divide limited slots cannot be evaluated by the same metric. That distinction can be written into the deployment review document first.
• Where is the cutoff now, and where does it move if supply falls? When budgets are cut or the waitlist grows, t gets pushed toward the tail. Recording the current share that can be served alongside the worst case turns it into an audit item.
• How does the next accuracy improvement change allocation outcomes? With a deployed model's AUC and a measured Δ you can back out ρ, then plug in a target and compute the change in D in advance.

Earlier articles along the same line include our look at how hiding the audit sample changed the cost of faking fairness, the EU exception that unsealed sensitive data for bias correction, and the right to question an AI decision about hiring and benefits. The paper itself is available at arXiv:2608.11491.

Primary Source

• 1.Moon, E. S., Tamura, M., & Guha, S. (2026). "The Accuracy Trap: Structural Scarcity Amplifies Relative Inequality in Algorithmic Allocation." arXiv:2608.11491. — This article's primary paper. Source of the scaling law and the Canadian child welfare / U.S. SEER breast cancer validations.

Academic Papers

• 2.Moon, E. S., & Guha, S. (2026). "The Paradox of Prioritization in Public Sector Algorithms." Proceedings of the 2026 ACM Conference on Fairness, Accountability, and Transparency (FAccT). — The paper's direct precursor, a discrete combinatorics version of the prioritization problem.
• 3.Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). "Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations." Science, 366(6464), 447–453. — The canonical prior case of racial bias in a health risk prediction algorithm.
• 4.Kasy, M., & Abebe, R. (2021). "Fairness, Equality, and Power in Algorithmic Decision-Making." Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT), 576–586. — Grounds the framing of equity as a matter of power and resource distribution.
• 5.Tilly, C. (1998). Durable Inequality. University of California Press. — Sociological background on how organizational procedure turns small group differences into durable advantage.
• 6.Merton, R. K. (1968). "The Matthew Effect in Science." Science, 159(3810), 56–63. — Describes how early advantage accumulates, background for why Δ does not vanish.