Executive Summary

Key figures

Source: Saha et al., New Evidence, Same Choice: Testing Physical Experiment Selection in Vision Language Models, arXiv:2609.11022v1 (2026-09-10), results and appendix

Physical reasoning benchmarks usually score the final answer alone. The decisions a model passed through on the way to that answer leave no trace in the score. The paper's example is simple. If half the questions can already be answered, a model that always stops and answers takes half the points without judging anything. Average accuracy writes that half down as skill.

The half that average accuracy books as skill shows up as numbers in the records of all six models. Pixtral 12B chose to stop on 97.6% of all its decisions, and its minimum cost choice accuracy is 48.8%, close to half. Qwen2.5-VL 32B is the highest of the six on the direct protocol at 49.5%. Read as single numbers those look like coin flips. Counted by pair, Pixtral has no fully correct pair at all and the 32B reaches 0.3%.

Earlier attempts exist. TRAPSBench pairs sufficient and insufficient physical evidence to study restraint, and OMD-Bench corrupts the input modalities to test whether abstention stays calibrated. This paper asks for one thing more: which of the missing measurements to choose.

The tasks are three mechanics settings. How far a pushed block coasts before it stops, how much energy an object keeps after it bounces, and how long a mass on a spring takes to complete an oscillation. Each question has four candidate worlds, built from two masses crossed with two values of a second property. That second property is the friction coefficient, the coefficient of restitution, or the spring stiffness.

The model receives one 672×672 image of four panels carrying a ruler, marked positions, and times or loads. This first measurement reveals one property and cuts the candidate worlds from four to two. If the two survivors point to the same answer, nothing further needs measuring; if they point to different answers, one more measurement is required. A test of either property costs one unit, the combined report that hands over both properties costs two, and free fall before contact costs one while revealing neither unknown.

Pairs are built two ways. In the first, the question stays fixed and only the image is swapped, so the prompt text is identical down to the byte. In the second, the image stays fixed and only the threshold in the question moves, so the figure and the action menu survive untouched. Either way the reference action splits inside the pair. The candidate worlds and the prices are fully known, so which action is correct follows from calculation.

The scale is 144 independent physical families, 48 of them in each of the three tasks. That yields 576 decisions and 288 pairs. The authors also ran a deterministic reference program on the same problems, and it obtained a correct result on all 9,216 of its records. That program also sees the calibration values, the panel boundaries, and the item identifiers, so the paper states plainly that it confirms the procedure rather than supplying a score to set beside the models.

Whether the construction came out as designed was counted separately in the appendix. All 288 fixed-question pairs carried identical text, and the 1,152 measurements that cannot separate the hidden worlds matched down to the image bytes. When a detector read the marked positions back, the largest error was 0.456% of the panel span, and the smallest separation between features that must be told apart was 54 pixels against the 8 pixels the implementation requires. The tightest threshold still had 17.2% of margin, and no check failures were reported. This verification covers the original rendered sheets, not the resized and tokenized input each model actually receives.

The models under test are Qwen2.5-VL at 3B, 7B, and 32B, SmolVLM2 2.2B, Idefics3 8B, and Pixtral 12B. Weights were neither modified nor quantized. Under the direct protocol, the share of the 288 image-changed pairs that drew the same action runs from 95.1% for SmolVLM2 up to 100.0% for Pixtral. The reference action never repeats inside a pair by construction, so this number says the change in evidence did not reach the decision. Pairs where only the threshold in the question moved land between 95.5% and 97.9%.

The failure is not one lump. The paper counts two things apart: whether a model stopped properly on the 288 questions with a known answer, and whether it chose the right measurement on the 288 questions still open. Each model breaks in a different place.

In four of the models, stopping when stopping is right comes out far better than choosing what to measure. Pixtral is the extreme case: it stopped on 97.6% of the answerable questions and never once picked the right measurement. A habit of almost always stopping is enough to produce that first column. The opposite direction also appears. Qwen2.5-VL 3B stopped on only 2.1% of the questions where stopping was right, and its 16.0% on choosing a measurement is the second best of the six. The three Qwen sizes do not rise and fall together either, and the paper states that no scaling law can be drawn here.

The choice distributions in the appendix make that habit visible. The authors counted what the six models picked on the 288 already-answerable questions and on the 288 still-open ones, and printed the two as separate tables. The reference action is opposite across the two groups. In the first, stopping alone has minimum cost; in the second, measuring the missing property is cheapest. Yet the distributions recorded under the direct protocol nearly coincide. Pixtral stopped 281 times and re-bought an already measured property 7 times in both groups, with all six cells identical between the two rows. Qwen2.5-VL 32B stopped 216 times and 218 times, Qwen2.5-VL 3B re-bought 233 times and 232 times, Idefics3 re-bought 93 times and 95 times. Whether the answer is already settled barely enters the choice.

The habit follows the decision to buy a measurement. Of the 278 single-property purchases Qwen2.5-VL 3B made on still-open questions, 83.5% asked again for the property the first image had already measured. Idefics3 is at 86.4% of 110 and SmolVLM2 at 80.9% of 199. Qwen2.5-VL 32B almost never makes this mistake, and instead simply stops on 75.7% of the still-open questions.

The second prompt asks for two short sentences of justification with the answer on the last line, capped at 256 tokens. With it attached, the action repetition rate drops to somewhere between 70.8% and 93.8%. Actions change with the evidence noticeably more often. The best pair-level score also comes from here, 5.9% for Qwen2.5-VL 7B, with a 95% interval of 3.1 to 9.0. Whether the justification written down is correct is a separate matter. In a sliding case the authors reproduced from the same model, the action was right while the justification said the image “provides direct evidence of the block’s stopping distance.” The authors’ reading printed beside it differs: “The image starts at a known speed. The target starts after an impulse.”

Accuracy did not move in one direction. On minimum cost choice accuracy, Qwen2.5-VL 3B went from 9.0% to 25.0% and Idefics3 from 33.2% to 41.0%. Under the same prompt Qwen2.5-VL 32B fell from 49.5% to 21.0% and Pixtral from 48.8% to 17.2%. The drop is steeper on stopping when stopping is right: the 32B went from 75.0% to 20.1% and Pixtral from 97.6% to 20.5%. The paper spells out why more varied behavior cannot be read as closer to correct. Some of the responses that look changed are invalid outputs or parser misreads.

Choosing what to measure and using a value once measured are separate failures, and of the cautions the paper attached this one weighs more. The authors supplied a useful measurement result free on the still-open questions and asked again for a final answer. The best of them, Qwen2.5-VL 3B, reached 34.6% and SmolVLM2 reached 15.3%, while the other four stayed under 1%.

Those four did not write down wrong answers. Of the 4,608 answer requests that follow a supplied measurement, undetermined came back 4,608 times for Pixtral, 4,595 times for Idefics3, 4,582 times for Qwen2.5-VL 32B, and 4,537 times for the 7B. That is where the near-zero scores come from even when the evidence has become sufficient. The authors draw a line around this. It is a property of the recorded answers, not proof that every numerical calculation failed.

With the image removed and both property values handed over as numbers, no model reached the 75% requirement written into the plan. Scores stay low even with the calculation pulled out and measured by itself. Two of these probes offer only two options, so guessing alone returns 50%, and all six models cluster near that line on both. Recovering which value the first image points to runs from 49.7% to 53.5%, and the balanced true-false calculation question tops out at 50.4% for Qwen2.5-VL 32B. Ruler reading itself varies widely: Qwen2.5-VL 7B sits at 83.7% and Idefics3 at 2.8%.

Pixtral scored 0.1% in the direct both-given condition because only one request out of 1,152 was parsed into an answer at all. Many generations began a derivation and hit the token cap, and the same condition returned 45.5% once a short reasoning step was allowed. A low score is not always missing knowledge. Prompt changes are no common cure either. When the ruler reading is spelled out in text, four models gain 9.0 to 14.2 percentage points, Qwen2.5-VL 32B stays where it was, and Pixtral loses 35.8 points. The remaining three variants are flatter. The prediction equations, handed over in full, lift only SmolVLM2, by 5.2 points, and the interval covers zero for the other five. An instruction to count the remaining possibilities moves that same single model, by 10.1 points. A redrawn figure with different colors and line styles changes little beyond 2.8 points for Qwen2.5-VL 7B.

One last experiment puts a price on the defect. The authors left the stored response records untouched and rescored them under different choice rules. When Qwen2.5-VL 3B chose for itself, it spent a mean cost of 0.99 and got 18.8% right. Under a rule that takes only the cheapest sufficient measurement, the cost drops to 0.50 and accuracy rises to 31.7%. The rule that buys the complete report spends 2.00 and stalls at 32.1%. SmolVLM2 shows it more sharply. Its own choices cost 0.79 for 12.0%, the minimum cost rule costs 0.50 for 19.4%, and the complete report spends four times as much to come back down to 15.6%. That leaves room to measure less and get more right. This comparison rescores answers already stored, so it does not test how a model responds when a new price list is put in front of it.

The paper's scope covers six particular open checkpoints, three idealized mechanics families, discrete properties with only two values, and at most one extra experiment. Natural video, sensor noise, physical robots, and open-ended experimental design all sit outside it. Selection and answering are separate calls, so the side that answers cannot see why that measurement was chosen.

The same question comes back outside the lab. In science and manufacturing, experimental design and instrumentation are data collection. More systems now seat a model where the decision about what to measure next gets made, and both habits found here, re-measuring a known value and stopping while the answer is still open, fold into the average when only final accuracy is scored. The question of what a benchmark score is actually attached to came up again in a crystal-structure benchmark whose answer ceiling was computed without a model, and the reasons researchers keep experimental design in their own hands run through the work scientists refuse to hand to chatbots.

Thank you for reading this far. The full text sits at arXiv:2609.11022, and every figure in this article was checked against the body and the appendix tables. If your team already lets a model pick which data to collect, we would like to hear what safeguards you put around it.

Pebblous Data Communication Team
September 14, 2026

References

• 1.Saha, S., Dipta, S. R., Sarwar, N., Saha, S., Jiang, Y., Li, S., & Wang, Q. (2026). New Evidence, Same Choice: Testing Physical Experiment Selection in Vision Language Models. arXiv:2609.11022