Executive Summary

Key Numbers

Source: Sasli and Coughlin, Anomaly-ranked confirmation slows new-class learning, arXiv:2611.03250 (2026)

Two models usually stand side by side wherever a new kind of thing has to be found. One is a classifier trained on the known classes. The other is an anomaly detector that points at inputs the classifier finds strange. The detector orders the candidates by how suspicious they are, and an expert works down from the top. At an observatory that means spectroscopy, at a hospital a read, on a factory line a visual inspection. Each confirmed example joins the training data for the new class, and the classifier ends up knowing one class more.

Sasli and Coughlin asked about one step in that flow. Can the ranking built for discovery also serve for teaching? They took five datasets, hid one known class in each so it would behave like a new one, raised the number of confirmations, and measured how well the classifier came to recognize the hidden class. The comparison was the same number of candidates confirmed at random.

The question came out of their earlier work. FLARE, the transient pipeline the two of them built, also fed confirmed examples of a hidden class back into training and tracked recognition, but there the examples to confirm were drawn independently of the anomaly ranking. In a real follow-up program driven by anomaly detection, the detector also shapes which examples receive labels. This paper picks up the point where those two conditions part.

The five datasets share no field. Gravity Spy sorts the noise in a gravitational-wave detector into 22 shapes. PTB-XL is a public set of 12-lead electrocardiograms. The light curves come from the ZTF Bright Transient Survey. CIFAR-100 and Food-101 cover photographs. The mix is there to show that the result is not the habit of a single field.

The first measurement looks at individual examples. Members of the hidden class were split into those the detector flagged and those it missed, and both groups were scored with the same classifier. That classifier had been promoted using random confirmations, so the comparison turns on which members get recognized rather than on which members did the teaching. In all five datasets the flagged side was recognized after fewer confirmations.

In Gravity Spy, recognition after a single confirmation was 0.69 for flagged members and 0.24 for the rest, and the two groups took about 40 confirmations to meet. CIFAR-100 reached 0.86 after one confirmation, while the unflagged side sat at 0.79 even after 40. Food-101 shows the same shape at 0.72 and 0.57. In the ECGs, recognition of flagged members fell as confirmations accumulated, yet it stayed above the unflagged side throughout.

All of this reads easily as a sign that the detector is good at its job. The paper reads it from the other side. The region the detector picks is where the class shows its features most sharply, which is also where the classifier already learns with little help. Spending the confirmation budget there again and again removes any chance to see the rest of the class. Ten confirmations teach a different picture depending on where inside the class they come from.

The second measurement aims at the selection rule itself. Confirmation targets were chosen three ways, the top n by the detector, n at random, and the bottom n by the detector, with the learner and the evaluation held fixed. This comparison ran in three of the five datasets: Gravity Spy, CIFAR-100 and the ECGs.

At ten confirmations the detector's choice trailed random in 20 of 22 Gravity Spy classes, in all 13 CIFAR-100 classes and in 10 of 11 ECG classes. The mean gap in Gravity Spy was 0.17, with a 95% bootstrap interval from 0.11 to 0.24 averaged over three training seeds. Every one of the three datasets has a range where that interval excludes zero: from five confirmations onward in Gravity Spy, at every measured budget in CIFAR-100, and at five of six budgets in the ECGs. Stretching to twenty confirmations leaves the gap in place, 0.66 for the detector's choice against 0.82 for random.

Converted into labeling cost the number grows. Reaching the recognition rate of ten random confirmations takes about eighteen detector-ranked confirmations in Gravity Spy and about nineteen in the ECGs. CIFAR-100 is worse. Twenty detector-ranked confirmations did not reach what five random ones delivered.

The suspicion that a few extremes at the top of the list produce all of this has an answer ready in the paper. Drawing n at random from the top 2n keeps the result, and all eight Gravity Spy classes tested that way still trailed random. Choosing from the least suspicious end instead lands near random or a little below. The top of the ranking is the part that hurts, and reversing the ranking does not fix it.

4.1The Classifier Takes the Next Turn

The first fix changes who owns the ranking. Only the top two candidates from the detector are confirmed, enough to promote the new class, and from there the confirmation targets are drawn at random from the candidates the promoted classifier itself alerts on. At ten confirmations Gravity Spy recovered 63% of the gap between detector and random, and CIFAR-100 recovered 72%. The ECGs showed no improvement. Letting the same classifier pick only its highest-probability members did worse than sampling at random inside its alerts.

Conditions closer to the field do not overturn the result. In a stream that mixed candidates for the target class with candidates from another hidden class and a fifth of the existing training data, the detector's top of the list did hold more true members of the target class. At twenty confirmations that came to 15.6 against 12.9 in Gravity Spy, and 10.8 against 7.2 in CIFAR-100. Recognition still ran lower: 0.57 against 0.66 in Gravity Spy and 0.70 against 0.72 in CIFAR-100, with random ahead in both. The twenty-confirmation numbers in the previous section came from an unmixed stream, so the two sets do not line up for a direct comparison. Across both datasets the highest recognition belonged to the hand-off to the classifier.

4.2Distant Neighbors Instead of Close Ones

The second fix touches the known classes rather than the new one. For each of 21 Gravity Spy classes the authors built one taxonomy from the three nearest existing classes and another from the three most distant, holding everything else fixed. With the distant neighbors, the confirmations needed to reach 80% recognition fell in 18 of the 21 classes, and the median went from 4.3 to 1.6. In CIFAR-100, replacing a class's four semantically close siblings with four classes from a different superclass raised the score in 99 of 100 classes.

The authors draw a line under this. The intervention shows that the makeup of the known classes governs the cost of learning, and it is not a law stating how much distance buys how much saving. The correlation between the increase in directional distance and the drop in cost came to −0.04, effectively nothing. Distance tends to make the learning cheaper, and that is as far as the experiment goes.

That is where the paper ends, and its edges are marked by the authors themselves. The member-level analysis is the exploratory stage that prompted the experiments after it, and the selection-rule comparison ran in three of the five datasets. Hiding a known class so it behaves like a new one may differ from meeting a genuinely new population, a caveat the text states outright. Predeclared criteria that went unmet are reported as unmet, while noisy labels and classes whose confirmation costs differ are left for later.

Outside the paper, this structure is not confined to observatories and hospitals. The same question stands wherever the number of items a person can look at is fixed and somebody decides what fills it. Quality inspection on a line, fraud review, medical reads and document review queues all work that way. Most of them push the strangest items to the top. Few teams have put that order next to a random sample and compared the two.

So what should we ask first? Three questions are worth taking in turn.

• What orders your queue? An anomaly score on its own, or something else alongside it?
• Is there a record of that order being compared against a random sample? Without one, filling a fixed share of the queue at random is itself an experiment worth running.
• Is this review meant for discovery or for teaching? Nothing says the work of finding problems today and the work of teaching the next model have to share one order.

The last question matters most. The paper measured the cost of teaching a new class, not the speed of finding anomalies. Where an accident has to be caught today, the most suspicious item still belongs at the top. The loss accumulates quietly when both purposes share one queue.

Thank you for reading this far. The paper is at arXiv:2611.03250, and the authors have published their experiment code in the label-cost repository. If your review queue already carries a random share, we would like to hear how you settled on the proportion.

• 1.Sasli, A., & Coughlin, M. (2026). "Anomaly-ranked confirmation slows new-class learning." arXiv:2611.03250 [astro-ph.IM].
• 2.Sasli, A. et al., & Coughlin, M. (2026). "Photometric classification and out-of-taxonomy detection of ZTF transients with FLARE." arXiv:2610.30656.