Executive Summary

Key figures

Source: Polat et al., Separating perception from reasoning in vision–language models arXiv:2609.00663v1 (2026-09-01), Results and Methods

Vision–language models already sit in the reader's seat for figures, spectra and rendered structures, and they turn up as components inside materials-design pipelines. The flagship multimodal benchmark for the domain reports that models handle basic perception of chemistry and materials images well and then fail at spatial reasoning and multi-step inference. Crystal structures are the sharpest case in that domain, because they reach most readers as pictures, and a model that could read those pictures would inherit a century of crystallography that was drawn and never tabulated.

The difficulty starts once a model gets one wrong. Re-scoring after the image is swapped for a model-written description, serialising the visual stream into ground-truth text, a best-case ensemble, a selector fitted to the labels: those are the routes in use. The paper points at what they share. The reference against which a model's deficit is read has a second model inside it. Attribution inherits that model's errors, and the apparent headroom shifts whenever the evaluation artifacts change.

Without a reference, counting the remaining headroom also falls back on convention. The unsolved fraction of a benchmark is usually read as one minus the best score, which amounts to assuming the ceiling is 1. Attempts to measure the ceiling rather than assume it keep a model inside all the same, either a best-case ensemble over several models or a selector computed against the labels, and swapping that evaluation artifact moves the fraction counted as unsolvable by enough to change conclusions. An earlier crystal-structure stress test from the same group supplied the source crystallographic file alongside the image; the gap narrowed and did not close, and the yardstick it was read against was the best observed score rather than a measured bound.

For natural photographs there is no way around any of this. Recovering scene geometry from a single photograph is ill-posed, an infinite family of three-dimensional scenes projects to the same picture, and every practical reader supplies a learned prior in place of the missing constraint. Rendered scientific figures sit differently. A crystal-structure render is an orthographic projection of a structure whose coordinates are known exactly, taken by a camera set the experimenter chose. Invert those cameras, re-solve the correspondence between views, and the answer the images support arrives with no model at any step. The paper calls that quantity the render ceiling.

The structures come from the Materials Project in conventional unit-cell form. To block the shortcut of answering from composition, a composition-disjoint split holds out thirteen elements for evaluation only, and the 1,820 released structures divide into 1,610 for training and 210 for evaluation. The label is the seven-way crystal system that spglib returns at a fixed tolerance. Structures whose space group wobbles across a tolerance sweep were quarantined, and on a stratified sample disjoint from both splits this label agreed with the source database's space group on all 220.

The render protocol is plain. The conventional cell is tiled 2×2×2 and drawn at 768 px by five orthographic cameras, three on principal axes and two off-axis. The pictures carry no shading and no bonds, only flat discs sorted by depth, with the cell edges left as dashed lines. The whole design rests on one property: each camera's projection matrix is a value the experimenter already holds.

The oracle runs that value backwards. Ground-truth atom positions are projected into each camera, the correspondence between views is deliberately discarded, rays from same-species discs are intersected to rebuild positions, and every surviving point is checked for visibility in all remaining views. Feeding the survivors back into the same algorithm that produced the labels returns the answer. The procedure is deterministic end to end, with no learned part anywhere in it.

The theorem says this procedure can fail one way only. Two distinct atoms whose projections happen to coincide can manufacture a phantom point that passes the check in every view. Phantoms are fixed by the structure and the cameras alone, which means they can be enumerated exhaustively for each sample. The theorem's separation assumption, the minimum distance between same-species atoms, came to 1.0951 Å on the 210 evaluation structures and 0.7501 Å on the 1,950 scale-up structures where the full phantom census was run, both far above the 0.02 Å required. At the certified setting the oracle returned the correct label on both samples, reconstructing positions to 3.9×10⁻¹⁵ Å, with no atom dropped and no phantom accepted. That is where 1.0000 comes from.

Whether the instrument measures the images or merely assumes them perfect is settled by taking cameras away. Running the same census over camera subsets does turn up phantoms, and the ceiling does drop below 1. At the certified tolerance the mean ceiling is 0.9214 with two cameras and 0.9967 with three. The phantom rate reaches exactly zero at four cameras, so the fifth adds reconstruction fidelity rather than emptying a set the fourth left open.

Where the ceiling gives way is not the image but the extraction stage. Re-reading the same oracle while widening the tolerance used to merge points traces a frontier: 1.0000 at the certified value, 0.9810 at five times it, and 0.9524 at the fifteen times the released pipeline used. Every structure lost along the way still recovers the correct atom count and has its closest same-species pair more than eleven times the tolerance apart, so the loss is a label-tolerance failure inside the symmetry algorithm rather than distinct atoms being merged together. Two questions part company here: whether the answer is identifiable from the images, and whether an actual pipeline extracts it.

One real extractor was run end to end, and that record makes the split visible. A connected-component blob detector working on colour thresholds was placed in front of the oracle under a pre-registered gate: if triangulation fails on more than 5% of structures, report the failure rate and do not score it. The detector tripped the gate. It recovered zero atoms on 19.0% of the evaluation sample and 22.4% of the separately drawn expansion sample. The run therefore carries no score, only diagnostic values of 0.0762 and 0.0857, below every model that appears later in the paper.

Nor do the errors accumulate in one direction. When a disc goes missing in one view, a ray from a different atom can pass the cross-view check and manufacture a site that was never there, and 44 of the 210 structures came back with more atoms than exist. Simulating better detection while holding the rest of the pipeline fixed, per-view recall has to reach somewhere between 0.98 and 0.999, depending on the sample, before the diagnostic catches the best vision–language model's 0.7333. Measuring the ceiling brings that point into view as well: where the binding constraint stops being the model under test and becomes our own reading.

The same geometry hands benchmark builders a rule. Assuming exact extraction, the frozen protocol, an off-axis perturbation of it and the released tiled geometry each return all 210 labels, so no camera choice in this family hides part of the task. What separates them is conditioning. The frozen cameras carry a conditioning constant of 2.7321 against 8.4258 for the off-axis perturbation, and under injected Gaussian noise on the centroids the frozen protocol tolerated 2.1 times as much. Robustness comes from separating the camera directions, not from adding views. Two of the three predictions the authors registered in advance missed, though: the three-way ordering that included the tiled arm, and the absolute noise budget, with the observed tolerance ratio falling short of the predicted one.

Once the ceiling is fixed, a model's deficit splits by subtraction. Three rungs of the ladder are in play: answering from the pixel renders alone, answering from exact fractional coordinates, species and cell parameters handed over as text, and the oracle's ceiling. The second rung minus the first is what doing the perception for the model recovers. The ceiling minus the second is what remains after the perception has been done for it. Because the ceiling holds no model, both subtractions run against a fixed quantity rather than against another model's output.

Handing over the geometry as text lifted all fourteen. None reached the ceiling, and the best landed at 0.8524. The median perception share of the total deficit came to 0.2901, and for 13 of the 14 the post-perception residual was the larger of the two components. Grok 4.5 is the single exception. Re-read against the ceiling at the released tolerance the count is 12 of 14, so the verdict holds wherever on the tolerance frontier it is taken.

The two ends of the roster put numbers on the split. Gemini 3.6 Flash, the best pixel reader, went from 0.7333 to 0.8524, recovering 11.90 points and leaving 14.76, a share of 0.4464. GPT-4.1 mini, the weakest, went from 0.3667 to 0.4143, recovering 4.76 points and leaving 58.57, a share of 0.0752. The two shares stand a factor of 5.9 apart. The natural strengthening, that a stronger model should show a larger perception share, is not supported: share and pixel accuracy are unrelated by rank correlation, and the raw lift runs the other way, correlating with pixel accuracy at −0.6439. The better a model reads pixels, the less exact geometry buys it.

Separate controls rule out the lift being an effect of the text format. Dropping the image and passing the chemical formula in an otherwise identical prompt collapsed every scored model toward chance, a mean of 0.1487 against seven-way chance at 0.1429, with the paired difference significant for 10 of 13; the three nulls are the weakest models, two of which sat at chance even with images. The explanation that the renders withhold information is excluded as well. Removing informative occluders from the oracle's input changed no symmetry classification, while exact atom-count match on the expansion sample's four-view condition moved from 204 to 194 of 210, so the intervention did reach the reconstructor.

Whether the residual is a deliberation deficit was controlled on one model. Claude Opus 5 run at a minimal reasoning budget and at the provider default scored identically, 130 of 210 under both, while the default spent 5.4 times the output tokens and 3.4 times the wall time. The two settings agreed on 169 structures and broke even on the discordant ones, 13 against 13. For this model, the distance to the ceiling is not something more thinking closes. Whether that holds for the other thirteen was not tested. From there the authors take one step toward post-training: a recipe that decouples perception from reasoning would improve a stage other than the one holding most of the roster back.

A large deficit on its own does not exclude the explanation that the images were never readable in the first place. So the authors put a generic vision architecture with no language component in front of the same renders. ResNet-50 and ViT-small started from ImageNet weights and were fine-tuned on the 1,610 released training renders and nothing else. No augmentation was applied at all, since a rotation, flip or aspect-ratio distortion can invert the very label being predicted. The answer comes from averaging logits over the five views.

ResNet-50 got 188 of 210 right for 0.8952, within 0.1048 of the ceiling and above the 0.7333 of the best pixel reader among the vision–language models. ViT-small reached 0.8333 and also cleared all fourteen. The conditions ran against them: both networks trained and scored at 224 px, below the resolution the vision–language models saw. Training took 153 and 152 seconds on a single consumer graphics card.

One more row belongs in that table, a reading that never looks at the image. Nineteen numbers pulled from the conventional cell, fed to a random forest, get the same 188 of 210. Those nineteen are the six lattice constants and the volume, three ratios of sorted edge lengths, four angle deviations and the angle range, the variances of the angles and the edges, and the site count and density. To four decimal places it matches the pixel model at 0.8952. The section therefore establishes that the pixels are readable. It does not establish that pixels win.

The conclusion is narrow and firm. A model with no crystallography-specific inductive bias and no language reads most of what the vision–language models left on the table, so the deficit measured earlier is not the product of information the images cannot carry in principle. The vision model has its own limits, though. On the separately drawn expansion sample of 210 the same checkpoint fell 0.1428 to 0.7524. Part of that drop belongs to the sample: the shape-free baseline falls from 0.5286 to 0.2476 on the same draw, so ResNet-50's margin over the baseline actually widens to 0.5048. Of the 22 errors on the original sample, 7 swapped hexagonal for trigonal, the single largest confusion pair, and that pair is a cell-metric degeneracy rather than a weakness specific to one model.

The hexagonal–trigonal degeneracy reaches the contrast between the two readings. Splitting the evaluation sample by whether the cell numbers alone determine the crystal system leaves 140 sufficient and 70 ambiguous structures, and 60 of the 70 are hexagonal or trigonal. Remove that one pair and ten structures remain where the pixel-against-numeric contrast genuinely carries information. There is a limit on the other side too. At the finer space-group granularity the tabular classifier drops from 0.8952 to 0.6810, because point groups and space groups turn on atom positions that the lattice constants do not carry. The oracle is flat across all four labels, since a reconstruction is either right or wrong and all four labels follow from it.

Choosing what to hold the models' scores against also needs this yardstick. A shape-free baseline that sees only atom count, density and cell volume scores 0.5286, and 13 of the 17 scored vision–language model arms sit below it, with the four that clear it all current-generation. The authors also record that 0.5286 is an outlier of how the released sample was drawn. On 1,933 structures generated under identical rules the same baseline is 0.2054, and across 446 resampling trials under three regimes it never exceeded 0.52.

Held against a stage in the middle of a pipeline, the same yardstick surfaces a failure a leaderboard cannot see. The authors promoted a strong model to the extraction stage, prompting it for species and positions with the symmetry question withheld, then passed that text to a weaker model to answer from. Final accuracy came out exactly at the majority stratum's base rate. A pipeline that scores only its endpoint walks away from this with a number and no measurement.

The intermediate is where the information sits. Scored against ground truth, the emitted coordinates had a median recall of 0.0000. Of 206 structures, 105 had not one emitted atom within tolerance of a real one. Four structures produced no parseable atom list at all and dropped out at the gate, so the denominator here is 206 of the 210, and those four count as unanswered in the full-sample figure. And yet the model emitted a median of 48 well-formed atoms per structure. The blob detector from Section 2 reaches 0.400 on the same task. Syntax perfect, content empty.

The discussion assigns this finding the widest reach. Agentic materials workflows are assembled from exactly these intermediates: a structure read off a figure and handed to a property predictor or a synthesis planner, checked, where it is checked at all, by a learned verifier that cannot separate a fabricated intermediate from a mistaken one. For any stage whose forward map is known, the intermediate can be scored against a reference with no model inside it, and the paper calls that the audit primitive these pipelines currently lack.

The move from here to evaluation-dataset design is not something the paper tested. It is this article's reading of the result in the language of data quality.

How far the instrument reaches is bound by three conditions the paper sets out: the forward rendering has to be known, it has to be invertible, and the demonstration covers one task in one domain. The oracle answers from exact centroids, so it states what the geometry determines rather than that any reader recovers it. Certification happens at a single tolerance per sample, and on the separately drawn expansion sample one structure is already lost at the certified setting. The phantom set is not monotone in that tolerance, so emptiness always holds only at the tolerance where it was measured, and no growth rate with cell size is proved.

The decomposition leaves its own residue: the model arms were not re-run on the scale-up sample, and the conditioning constant orders camera placements without setting a noise budget. The roster was frozen before the current generation, which makes the leaderboard a snapshot rather than a ranking, and a stronger roster could narrow the gap. The transfers the authors name are rendered molecular conformers and figures plotted from computed numbers: band structures, densities of states, phase diagrams. Natural photographs, whose cameras are unknown, are not on that list.

Inside that scope, the idea reaches straight into how data gets built. We write the answer into an evaluation set and we do not write down the ceiling on that answer. So when the scores come in low, the meeting splits. Nothing in the data settles whether the model fell short, whether the image never carried the information, or whether the label moved. This paper computed that ground into the dataset in advance. With the ceiling at 1.0000 the whole deficit belonged to the model, and taking cameras away lowered the ceiling for real, which showed the yardstick was measuring the images rather than flattering them. Before release, a builder can know whether the pictures ask the question the labels answer, and at what point their own pipeline's reading binds before the model does.

For a team building an evaluation set out of renders or synthetic data, four things are worth checking now.

• Can the generation protocol be inverted to recover the answer? If it can, that value is the ceiling on what the data supports, and model scores belong next to it.
• Have you looked for the conditions under which the ceiling drops below 1? Cutting back to two cameras produced phantoms on 14.5% of the evaluation structures, and which settings make the data hide its own answer is a countable property.
• Are the pipeline's intermediates scored against ground truth? Scoring only the endpoint books a well-formed fabrication as a reasoning failure.
• Do the prompts and the documentation describe the picture accurately? Even here, the render module's own docstring and the question shown to every model both call the output ball-and-stick, while the emitted renders carry no bonds at all. The authors disclosed the mismatch themselves, and it is the kind that can pull a reader toward bond geometry the image does not contain.

The question of what a benchmark score is actually measuring took the same shape in a drug-discovery case where models called binding correctly 98% of the time and could not point to the binding site. The wider problem of getting data ready sits in the conditions for AI-Ready Data.

Thank you for reading this far. The full text is at arXiv:2609.00663, and the code and data are public at KurbanIntelligenceLab/render-ceiling. Every number in this article was checked against the paper's own results and methods. If your team already computes a ceiling alongside the answers in an evaluation set, we would be glad to hear what form you keep it in.

Pebblous Data Communication Team
September 9, 2026

References

Academic paper

• 1.Polat, C., Kurban, M., Serpedin, E., & Kurban, H. (2026). Separating perception from reasoning in vision–language models: a model-free render ceiling for crystal structures. arXiv:2609.00663

Data & tools

• 2.Jain, A. et al. (2013). Commentary: The Materials Project: A materials genome approach to accelerating materials innovation. APL Materials, 1(1), 011002. doi.org/10.1063/1.4812323
• 3.Togo, A., Shinohara, K., & Tanaka, I. (2024). Spglib: a software library for crystal symmetry search. Science and Technology of Advanced Materials: Methods, 4(1), 2384822. arXiv:1808.01590
• 4.Kurban Intelligence Lab (2026). render-ceiling (code and data release). github.com/KurbanIntelligenceLab/render-ceiling