Executive Summary

Key numbers

Source: Marinho, Object-independent evaluation of small-data astronomical image classifiers, arXiv:2611.06484 (2026).

The comparison set an ImageNet-pretrained ResNet-50 and an ImageNet-pretrained ViT-B/16 against a custom CNN–Transformer hybrid trained from scratch. Three seeds were fixed in advance. On the test set, every prediction was written down once before an independent scoring step was allowed to open the labels. Taking the same exam several times and keeping the best attempt was ruled out by design.

Scores are kept in balanced accuracy, the mean of the four per-class recalls, so a class with more examples cannot pull the number toward itself. Random guessing gives 0.25. On the adaptive view, which leaves each object's own field of view intact, the three models reached 0.773 ± 0.014, 0.874 ± 0.054 and 0.866 ± 0.023. Only the model trained from scratch comes in a step lower; the two pretrained references land together.

The two are 0.007 apart. The 95% cluster-bootstrap interval around that difference runs from −0.048 to 0.062. Since it contains zero, nothing supports calling either model the better one, and across nine randomization tests under Holm correction not a single comparison rejected its null.

The visible gap lies elsewhere. The model trained from scratch trails at 0.773, and the class where it trails most clearly is nebulae. It got 27.5% of them right and sent 51.0% of them to the open-cluster class. On that same class ResNet-50 managed 76.5% and ViT-B/16 80.4%.

For that gap, several bootstrap intervals excluded zero while no test survived Holm correction. Marinho does not treat the mismatch as a contradiction. Unadjusted intervals around individual contrasts and a family-wise error rate across nine tests answer different inferential questions. The gaps therefore stay descriptive effect estimates rather than adjusted evidence that one architecture is superior.

Nor can the gap be credited to architecture. Only the custom model started out without ImageNet weights. Whatever advantage the two pretrained references hold mixes architecture together with parameterization, optimization and transfer learning. That is the sentence the abstract closes on: the comparison does not isolate architecture from pretraining.

Images reach the models in two forms. The first keeps the original 512×512 field and shrinks it by exact area averaging. This is the adaptive view, and in it each object keeps the field of view it was photographed with, anywhere from 5 to 120 arcminutes. An arcminute is a small unit of angle: the full Moon is about 31 arcminutes across, so 5 arcminutes is a sixth of the Moon and 120 is four Moons laid side by side. The second form resamples every object onto a common 4-arcminute field, which wipes those differences out.

The masked controls black out the middle of each form. In the adaptive view the central 128×128 pixels are set to zero; in the common field the central 4×4 cells of the effective 16×16 grid are zeroed. Even the percentile stretch that sets the contrast is computed on unmasked pixels only. The place where the subject stood is cut out of the photograph and the background is left behind.

Sorting those cut-out photographs into four classes gives a balanced accuracy of 0.731 to 0.800. That is close to three times chance, and not far from the 0.87 the intact images produced. Background and empty sky carry a good part of the class label on their own.

Model by model, the loss is uneven. ViT-B/16 slipped from the 0.866 it scored on intact images to 0.800. Erasing the object outright cost it less than 0.07. The masked common field fell further, to between 0.608 and 0.667, which is still more than twice chance. Only when field of view and object are taken away at once does the score bend noticeably.

The limit of this control is written down alongside it. Diffuse objects such as nebulae and open clusters can spill past the mask, so what the experiment offers is a context control rather than certified blank sky. The caveat narrows what the control proves; it does not disturb the finding that most of the score survives.

The plainest control hands the model no photograph at all. Its inputs come to five: the log of the field of view, the log of the pixel scale and the log of the catalogue extent, plus a flag for a missing extent value and a flag for a field of view that hit the cap. Object names, file paths and index numbers were excluded. Leaving a name in would let the model memorize the name instead of the picture, so the most obvious shortcut was cleared away first.

The models are light as well. A multinomial logistic regression with balanced class weights and a depth-3 decision tree, both fit on the training objects only. No network, no pretraining, no pixels. The pair scored 0.721 to 0.735. A model that has never looked at a single photograph, knowing only how the photograph was taken, already stands most of the way to what ResNet-50 reached with every pixel in hand.

What the control actually measured is narrowed down by the author. Among the five inputs, field of view, pixel scale and the cap flag describe how the photograph was acquired, while the catalogue extent and its missingness describe a property of the object itself. The 0.72, then, is the predictive power of those two kinds of information combined rather than a score won from acquisition settings alone, and on its own the number does not prove that leakage caused anything.

Lay the three conditions on one scale and the size of the gap becomes visible. Everything there is to win on this task runs from 0.25 up to the 0.87 of the intact images, and the condition with no image at all already sits three quarters of the way along.

How the acquisition record came to hold the answer is a matter of observing practice. A sprawling object has to be framed wide to fit on the plate, while a small dense one does not. Line the catalogue's images up by class and the distributions of field of view come apart. Table 1 of the paper puts numbers on that split.

All four classes share the same minimum of 5 arcmin. The upper tail is where they separate. Source: Table 1 of the paper.

Galaxies and globular clusters both sit at a median of 5 arcminutes and barely grow from there. Nebulae sit at 15 arcminutes and open clusters at 27.2, and both of them run out to 120. One number compresses the whole split: class membership explains 0.5827 of the descriptive variance of log field of view. Knowing the field of view alone, without looking at the image, finishes more than half the job of naming the class.

So does erasing the differences in field of view bring the score down? On the common field described in the previous section, the model trained from scratch reached 0.704, ResNet-50 reached 0.737 and ViT-B/16 reached 0.793, all three below the adaptive view. Marinho warns against reading that drop as the price of removing an acquisition cue. The common field is a coarse 16×16 effective grid that blurs compact targets and can truncate extended ones before the result is stretched back out to 224×224. The difference between the two representations mixes angular context with scale normalization and with outright information loss, so it is not a clean intervention on field of view.

There is already a name for a cue that travels with the label while having nothing to do with the task. Shortcut learning entered the vocabulary through Geirhos and colleagues in Nature Machine Intelligence in 2020, and that article is where class-dependent acquisition choices are named as possible shortcuts. This re-evaluation is a case of the same claim measured out in astronomical data.

Cutting 370 objects into 222, 75 and 73 followed a single rule: things that came from the same object are never cut apart. Different bands, different renders, different crops and different observations of one target do not amount to independent physical examples. Components whose fields overlapped were merged into indivisible assignment units, and those units were assigned whole to training, validation or test. Seventeen non-trivial components were preserved that way.

Once things are grouped, the file count and the object count part ways. This collection holds 1,110 FITS files across the blue, red and near-infrared bands of DSS2, and 370 distinct astronomical targets. Count by file and the data looks three times larger; count by object and it returns to its own size. Which of the two you count decides the unit of the split, and the unit of the split decides what the report card means.

What goes wrong without the rule is easiest to see by breaking it. Under a random split by image, one object's blue band lands in training while its red band lands in test. At test time the model is not meeting an unseen target; it is meeting yesterday's target again. Generalization and memorization blend into the resulting score and no method remains for pulling them apart. Kapoor and Narayanan set this out in 2023, naming leakage and the wrong split unit as recognized threats to reproducibility in machine-learning-based science, and this re-evaluation leans on their account.

The recommendations in the conclusion come to three. Split by physical object. Control for shortcuts explicitly. Test once, through a procedure fixed in advance. All three follow from one sentence: high accuracy on unseen objects is evidence of generalization within the collection, and not by itself evidence that the model learned the astronomical morphology it was meant to learn.

From here on we are outside the paper. Lifting the astronomy away from this experiment leaves a familiar scene. Organizations choose models off a benchmark table, and swap architectures over a difference in the second decimal place. This re-evaluation shows how small that decimal place gets beside the way the evaluation data was gathered. Changing the model bought 0.007. Erasing the subject left 0.73 standing.

A few articles on this blog have picked at the same problem in astronomical data. The detection gap in LSST at Rubin Observatory looked at where the composition of the answer key decides a model's score, and BRAHMa, the AI that measures bars in galaxies, showed that an accuracy built on a wobbling answer key wobbles with it. Both of those looked at the labels. This one looks at the inputs. Labels can be exact and the report card will still read the observing habits baked into the pictures.

Answers leaking into a benchmark dataset is no astronomical specialty either. Anomaly detection benchmarks carried identical rows across the train and test sides. Asking whether a score was made by the model or by the data is a question that crosses fields. Any organization holding evaluation data can check three things today.

• What counts as one item in our evaluation data? An image, a file, or the real-world object sitting behind them?
• Have several copies drawn from one object been separated across training and test? Photographs of one thing from different angles, records of one patient on different days, and different versions of one document all belong here.
• With the content stripped out and only the metadata left in, what score comes back? This control runs on a single line of logistic regression, and one run tells you where the floor of your own score sits.

Thank you for reading this far. The full text is at arXiv:2611.06484, and the reproducibility archive holding the protocols, configurations and prediction records is in the author's public repository. If your team has already moved its evaluation split onto object units, we would be glad to hear how far the score fell.