Executive Summary

The paper opens on a bar chart. It is an ordinary figure, a row of categories with their values, except that the researchers laid a gray patch over the label reading "Academic Funding" and ran a Gaussian blur across it. Not even the outline of the letters survives for a human reader. Then they asked: what does this bar's label say?

GPT-5.2 answered "Customer Support." That category appears nowhere in the figure. The answer carried no hedge, no note that the text was blurred or uncertain. The axes, the legend, and every other bar were perfectly legible, so the model had not failed to read the chart. What differed was what it did when it hit the one spot it could not read.

Filing this under OCR error or recognition failure misses what kind of failure it is. Recognition failure means misreading blurred characters as different characters. This is something else: putting a plausible value in a place where there is nothing left to read, and not saying that you put it there. The first is a question of accuracy. The second is a question of behavior. Chart understanding benchmarks have overwhelmingly measured the first.

CharXiv (2024) took roughly 2,300 real charts from arXiv papers and scored accuracy separately for descriptive and reasoning questions. SciFIBench (2024) posed 2,000 multiple-choice items on scientific figure interpretation. Both answer the question of how accurately a model reads. Nor is SciFigBench the first to include questions that cannot be answered. ChartQAPro (ACL Findings 2025) explicitly built in an unanswerable question type and reported scores broken out by answer type, and ChartHal (2025) measured chart hallucination sorted into categories such as nonexistent elements, nonexistent values, and contradictions. Getting this lineage right is what makes it possible to say precisely where the new contribution sits.

The broader proposition that accuracy and judgment are different axes is one we have covered twice recently: a report on GeneBench Pro, a biology benchmark that asked whether models withhold conclusions in messy data, and an analysis of the causal reasoning benchmark CausalDS, where a model tied for first on answer accuracy fell to fifth overall. What separates this piece from those two is not the modality but the question. Those asked when a model should hold back. This one asks whether a model that is already filling gaps can tell the fillable ones from the rest.

To measure what is missing, a human has to know what was removed. The team drew a stratified sample of 250 figures from 187 arXiv papers published between 2023 and 2025: 99 bar charts, 99 line charts, and 52 pie charts. The paper calls them "scientific figures," but the sample is drawn from NLP, machine learning, and computational linguistics papers. There are no wet-lab, chemistry, or physics figures, and no scatter plots, heatmaps, or network diagrams. The authors say as much in their limitations section.

Five kinds of stress are applied to those 250 figures: perceptual distortion (noise, low contrast, 15-degree rotation), a context condition that embeds each figure back into its source page, 100 figures carrying captions with false claims, 750 false-premise probes, and the centerpiece of the study, selective blurring.

How the selective blurs were made carries most of the benchmark's credibility. EasyOCR extracted text regions from each figure, GPT-4o proposed which ones to erase, fuzzy matching aligned the coordinates, and a gray blend plus Gaussian blur removed the content. Then the step that matters: a human opened every one and confirmed it. The resulting targets split into two groups: 228 where nothing in the surrounding figure could reconstruct the erased content, and 215 where something could. That split is the entire subject of section 5 below. Human labor across the project runs past 600 hours, and the description baseline was written independently by two trained annotators with a third adjudicating disagreements, reaching 94% agreement.

▲ The structure of SciFigBench. The caveat that the sample is limited to bar, line, and pie charts from AI-field papers travels with every number that follows | Source: arXiv:2608.13267

Scoring borrows Multidimensional Quality Metrics (MQM), a framework from translation evaluation, and applies it to figure descriptions. Each chart type gets a hand-built checklist, violations of accuracy, completeness, and clarity draw penalties, and the result is normalized to a 100-point scale. Accuracy violations are weighted roughly four times as heavily as completeness violations. The design says that saying something wrong is worse than leaving something out, which is the same judgment this article is built around.

The researchers say they measure three behaviors, and they borrow the names from electrical circuits: admittance, resistance, and inductance. The metaphor is memorable but loose. Inductance in a circuit is the property of opposing a change in current, whereas inductance in this paper means inference from partial information. Since the names do not explain themselves, each axis needs a definition of its own.

3.1Admittance: does it say it doesn't know?

Admittance is the share of cases where the model says it cannot read an illegible element. It is measured two ways. The active condition asks pointedly about the blurred spot. The passive condition lets the model describe the whole figure and checks whether it volunteers that part of it is unreadable. The fact that these two move in opposite directions for different models is the seed of the reversal that comes later.

3.2Resistance: does it fall for a false premise?

The resistance probes come in three types: questions that presuppose an element the figure does not contain, questions that plant a number 20 to 30% off the true value and ask the model to reason from it, and questions the figure simply cannot answer. Scoring runs on three levels: 1.0 for an explicit refusal, 0.5 for a vague deflection, and 0 for accepting the premise or inventing an answer. A separate 100-figure trial with falsified captions sits alongside these.

The first probe traces back to Loftus's 1975 eyewitness memory experiments. Ask "did you see the broken headlight?" and the definite article smuggles in a premise that leads people to remember objects that were never there. A falsehood works better presupposed than asserted. The second probe transplants Tversky and Kahneman's anchoring bias directly. What the researchers confirmed is that the ordering that works on people works on models in much the same way.

3.3Inductance: how often the filled-in value is right

Inductance is the axis most easily misread. It is not the accuracy rate across all questions but the share of invented answers that turn out to be correct. It is a conditional measure, computed only within the cases where fabrication already happened. A high score is therefore not automatically good news. A model that filled a gap it should have left alone still climbs on this axis if it guesses lucky. The number only means something read next to admittance.

▲ The three axes and the operational failures they map to. Inductance is conditional on fabrication having occurred, so reading it alone inverts its meaning.

Asked to describe 250 clean figures, the top two models are effectively tied. GPT-5.2 leads at 91.6, Gemini 3.1 Pro follows at 90.2. The 1.4-point gap is statistically significant (p=0.009) but small in effect size. The middle of the field clusters around Llama 4 Maverick at 81.4 and Qwen3-VL 235B at 80.8, and Phi-4 Multimodal sits last at 62.2. On the 1,000 reasoning questions the order flips slightly: Gemini scores 81.0% against GPT-5.2's 78.4%, with third-place Qwen3-VL 235B more than 20 points back at 58.4%.

Then the blurred spot gets named in the question, and the two models split. Gemini 3.1 Pro said it did not know 71% of the time. GPT-5.2 did so 8% of the time. The other six models all sit at 19% or below. Running an eye down the third column of the table below is enough to see how differently this axis moves from the others.

▲ Scores for the eight models. The scales are mixed, so do not compare across columns: MQM runs 0–100, resistance runs 0–1, and admittance and reasoning are percentages. An em dash marks a cell where the paper does not report a per-model value; no estimates were filled in. | Source: arXiv:2608.13267

Put the contrast on one screen and the study's claim resolves into a single picture. On describing figures they can see, the two models are 1.4 points apart. On how they behave in front of a spot they cannot see, they are 63 points apart.

One piece of arithmetic needs care. In the same condition, GPT-5.2 fabricated a value 96% of the time, and adding that to its 8% admittance gives 104%. This is not a scoring error. The judge marks admission and fabrication as independent items, because a response like "this is obscured, but it appears to read X" both admits and fabricates. So the inference that "only 8% admitted, therefore 92% fabricated" does not hold. The two numbers sit side by side and are read separately.

4.1Ask directly and it admits less

The stranger finding is the direction. Left to describe the whole figure freely, GPT-5.2 mentions the unreadable region 23% of the time. Point at that exact spot and the rate drops to 8%. The more specific the question, the less it admits. Gemini moves the other way, from 59% to 71%. The authors call this an answer-at-all-costs bias: a pointed question raises the pressure to produce an answer, and that pressure crowds out abstention.

This paper is not the first to observe that pressure. Meta FAIR's AbstentionBench measured abstention across twenty datasets and reported that scaling models up does not bring the ability along with it. The sharper finding came next: models put through reasoning fine-tuning abstained less, not more. Polish a model to reason well and it becomes less willing to leave a gap alone. That study tested text models, though. That the same tendency appears in front of a figure is not AbstentionBench's claim but an observation this benchmark adds.

The ordering of probe difficulty tells the same story. Models defended best against unanswerable questions (0.92–0.95), counterfactual figures came next, and the probe that worked best on them was a sentence presupposing an element that isn't there. Llama 4 Maverick holds at 0.76 when an incorrect number is stated outright, but slides to 0.63 when the same falsehood is buried in a premise. That is the ordering Loftus observed in people.

4.2The model that invents least invents most where it cannot see

Here is the sharpest edge of the result. In the baseline condition of describing clean figures, GPT-5.2's hallucination penalty was 0.01, the lowest of all eight models. On figures it can read, it is the model least likely to say something that isn't there. That same model filled in 96% of the spots it could not read. This is a signal that hallucination is not one fixed trait that shows up at a steady rate across situations. Change the condition and the behavior changes with it.

The caption trial paints yet another picture. Resistance to falsified captions ties GPT-5.2 and Gemini at 0.89, with no statistical difference between them (p=0.44). Phi-4 Multimodal scores 0.05, effectively transcribing whatever the wrong caption says. What is interesting is that Gemma 3 27B IT scores below Phi-4 on description quality yet reaches 0.38 on caption resistance, far higher. That is where the authors' reading comes from: susceptibility to captions looks less like a capability gap and more like a habit left behind by instruction tuning. One more note on distortion tolerance: the heaviest damage came from 15-degree rotation, which cost 19.4 points on average, while noise and low contrast barely moved the scores. A scan shot at a slight angle can hurt more than a blurry one.

As noted above, a human sorted the 443 blur targets into two piles. The criterion works like this. On an axis with monthly ticks running in order from January, erase July and the remaining ticks still tell you what was there. That spot counts as recoverable. Erase a category name that appears exactly once in the legend, and nothing anywhere in the figure can bring it back. That spot counts as unrecoverable. To a human reader the two situations are plainly different.

The odds that a model's filled-in value is correct diverge sharply between them: 14–66% where the surrounding evidence survives, 5–14% where it is gone. In the active condition, Gemini 3.1 Pro reaches 66%, GPT-5.2 59%, Llama 4 Maverick 34%, and Phi-4 Multimodal 15%. These numbers are conditional. They are not accuracy across all questions but the odds that a value the model chose to fill in is right.

The problem is not the scores but the behavior. The scoreboard says the two kinds of gap are different; the models treat them identically. They fill both. And in neither case do they mark that they filled anything. There is no switch inside the model that separates "there is enough here to estimate, so I will" from "there is nothing here, so I will leave it blank."

Missing data is hardly an edge case either. An analysis of more than 1.7 million records in the US National Cancer Database found 54.9% missing or incomplete on at least one variable. That imputation error propagates downstream has been quantified in the literature as well, with reports that the negative correlation between imputation error and downstream predictive performance strengthens as the missingness rate rises or the sample shrinks.

There is counterevidence pointing the other way. One line of argument holds that imputation creates no new information, so a sufficiently strong classifier extracts the signal regardless of imputation quality, and a related hypothesis suggests inaccurate imputation may even act like regularization. So the claim here is not that imputation is dangerous. It is that unmarked imputation is dangerous. Once a filled value sits in the same cell in the same format as an observed one, nobody downstream can tell them apart again.

Everything so far comes from a single preprint. But the same behavior was written down eight months earlier, in a different register, by the people who built the model.

In the GPT-5 system card of August 2025, OpenAI described its deception mitigations this way: they gave the model tasks that were partly or entirely impossible and rewarded it for honestly admitting it could not do them. One of the evaluation categories they listed is the subject of this report. "Tasks where the model is missing key information, such as a user asking about an image that is unavailable to the model, where previous models would fabricate an answer or claim to have completed the task."

Four months later, in December 2025, the GPT-5.2 system card update recorded what happened next.

The answer-at-all-costs bias the paper observed, then, is not an accidental defect but a priority the vendor documented first. Two independent observations point the same way. Elsewhere in the same document, OpenAI reports factuality on real-world prompts as on par with or slightly better than the previous model. The study's claim, that factuality metrics can improve while behavior in front of missing evidence gets worse, reproduces inside the vendor's own paperwork.

6.1The vendor removes the whole image; production loses one label

The vendor's test and this benchmark are built differently, though. What OpenAI used for its abstention evaluation was a CharXiv variant with the image removed entirely, plus Meta's AbstentionBench. In both, the input is conspicuously empty and easy for a model to notice. SciFigBench does the opposite: it leaves the figure intact and erases only the element being asked about. The model can still see the axes, the legend, and every other bar. It has to notice the absence on its own.

▲ A comparison of designs only. The three tests measure different concepts under different conditions, so their scores cannot be lined up and no numeric column is given.

What production systems actually meet is the bottom row. Figures rarely vanish outright. The figure is right there, and one of its labels has been smeared by a scanner, clipped by a crop, or mushed by a low-resolution render. Between conspicuous absence and inconspicuous loss lies the gap where that 96% lives.

6.2A documented policy does not guarantee deployed behavior

There is one more asymmetry. The Gemini 3 model cards carry no dedicated section on abstention or expressing uncertainty, just a line noting the general limitations of foundation models, hallucination among them. Which means the lab that documented explicitly training and measuring abstention scored 8% on this benchmark, and the lab that documented nothing of the kind scored 71%.

Reading that contrast as "documentation is meaningless" draws the wrong lesson. The accurate reading is the other one. A published policy does not guarantee deployed behavior, so the adopting organization has to measure it on its own input distribution. If the vendor's test conditions differ from what your pipeline meets daily, the vendor's conclusion is not yours either.

Regulators already have language demanding this behavior. Article 13(3) of the EU AI Act (Regulation 2024/1689) requires the instructions for use of a high-risk system to state not only the tested and validated level of accuracy and its metrics but also the known and foreseeable circumstances that may affect that expected accuracy. Article 15(4) requires resilience to errors, faults, and inconsistencies. Blurry scans, clipped figures, and mismatched captions are precisely "known circumstances affecting expected accuracy." For a high-risk use, documenting behavior on degraded input reads less like an option than an obligation. The high-risk provisions took full effect on 2 August 2026.

Before taking any of these figures at face value, look at the weak links the benchmark discloses about itself. The largest one is the judge.

Agreement among human annotators is high. Scores from different people agree at around 0.9 on every reliability measure used, with a mean absolute difference of 7.6 points. The automated judge is another matter. This benchmark's GPT-4o judge scored 15 points below humans on average, and up to 25.6 points below on GPT-5.2's responses. The decisive number is fabrication detection: of the fabrications humans caught, the judge caught 7%. It waved most plausible-sounding inventions through as valid answers. Model rankings held up reasonably well (rank correlation 0.80), but the bluntness has to stay in mind when reading any absolute number.

Treating this as a flaw peculiar to one paper misses the lesson. In a study asking GPT-family judges to detect hallucinations and omissions in mental health chatbot responses, hallucination recall came in at 9.3%, and collapsed to 1.6% when the dataset changed. There are findings in the other direction too. On short factoid question answering, judges have reached recall above 0.95 against human labels. The difference comes from the nature of the task. Judging breaks down as the task becomes more subjective, more domain-specific, and more multi-turn, and judging fabrication in figure descriptions belongs squarely on the hard end. The conclusion is not that judges cannot be trusted but that nothing is guaranteed until you have checked recall on your own task against human labels.

The paper's headline claim needs a qualifier as well. Compute rank correlations between axes across all eight models and every pair comes out at 0.83 or higher. The table itself states that values below 0.70 would indicate separate dimensions, and not one value falls below 0.70. Across the population, in other words, models that perform well generally behave well. The claim that quality and behavior are separate axes does not come from statistical separation. It comes from a rank reversal between two frontier models. Blur that distinction and the claim becomes an overstatement.

The run conditions carry several limits too. Gemini 3.1 Pro was a preview build rather than a general release. Every condition was run exactly once at temperature 0, with no repeated sampling, so the individual percentages come with no information about run-to-run variance. The sample is 250 figures, and as noted twice already, those figures are bar, line, and pie charts from AI-field papers. Given that arXiv alone receives more than 30,000 submissions a month, each carrying roughly ten figures, 250 is a very small window. Finally, the paper is a preprint posted on 13 August 2026 and has not been peer reviewed.

No news coverage or community discussion of the study turns up in search since it went up on arXiv. This is not a study that made waves; it is one nobody has verified yet, which is why every number here should be read as a value this study reports. As for how far a benchmark's target capability can drift from what it actually measures, we watched that play out in the retirement of SWE-bench Verified.

A pipeline that pulls figures out of documents and feeds them to a model runs with degraded inputs and mismatched metadata as constants. That is a measured condition, not an assumption. Even an extraction tool tuned specifically for astronomy literature reaches only 90.9% F1 on figures and 92.2% on captions, which means roughly one figure in ten gets attached to the wrong caption. Reports keep finding that conventional PDF parsers miss complex visual elements and fail to bind captions reliably to the elements they belong to.

So the caption-bias probe this study built is not a hypothetical attack. It is a condition document pipelines manufacture daily. Multimodal RAG research has a name of its own for the failure where a system cannot find the answer in the designated image and pulls in unrelated text evidence instead, and it has long been observed that retrieval quality degrades noticeably once OCR error passes 5%. Phi-4 Multimodal following false captions 95% of the time shows in advance what happens when a model like that is dropped into that environment.

8.1The diagnostic target moves from data to model behavior

The question DataClinic asks when it profiles a dataset is a simple one: was this value observed, or was it filled in? In this study the same question is aimed at a model's output. The question is unchanged; only the target moved. The yardstick built for measuring data completeness can be laid against how a model handles what is missing.

Move toward Physical AI and the same structure grows more urgent. When a sensor is occluded in a frame and the model paints in a plausible scene instead of reporting that it cannot see, that judgment becomes motion. In a document pipeline a wrong value stays in a table. On a robot it comes out as movement.

8.2Six questions to ask before adopting a model

For anyone choosing a model from benchmark scores, here is what this study says to add to the list.

• What does this model do with unreadable input? Has that behavior been measured separately from accuracy metrics?
• Does our pipeline record filled values separately from read values? Without that distinction, nothing downstream can undo it.
• Can we control how much captions and metadata are trusted? Caption resistance in this benchmark spread from 0.05 to 0.89.
• Do bad premises arriving from a user or an upstream agent get filtered? Sentences presupposing something that isn't there worked best of all the probes.
• Have we checked against human labels whether our own evaluation judge detects fabrication? In this study it caught 7%.
• Have we reproduced the vendor's documented policy on our input distribution? Documentation and deployed behavior are not the same thing.

One idea runs through all six. The most dangerous value in a pipeline is not the wrong one but the one whose origin has been erased. Wrong values get caught in validation. Quietly imputed values wear the same face as observations, travel downstream, and become fact. If the ability to call missing data missing is never measured on its own, you will pick a model that lacks it on the strength of its accuracy alone.

All that happened was one label being erased from one figure. Yet what a model does at that spot determines the trustworthiness of a system reading a million documents unattended. If accuracy is the only thing being measured, there is no way to know what is happening on the axis nobody measured.

Frequently Asked Questions

A few points tend to snag readers. What situation the 96% actually refers to, why adding 8% to it comes to 104%, and whether the result is grounds for switching models. Pinning down what the numbers mean keeps the conclusions steady.

Every SciFigBench number in this article was verified directly against the arXiv preprint, body and appendix included. Because the paper has not been peer reviewed, all values should be read as values this study reports. Vendor quotations come from the original system card PDFs.

The study under discussion

• 1.Oamen, P. O., Osei, O.-B., Mukherjee, A., Greisinger, C., Eger, S., Onobhayedo, P., & Zhao, W. (2026). "How Do VLMs Behave When Blind or Misled? Behavioral Evaluation of VLMs on Scientific Figures." arXiv preprint (2026-08-13). arxiv.org/abs/2608.13267

Prior benchmarks

• 2.Wang, Z. et al. (2024). "CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs." NeurIPS 2024 Datasets & Benchmarks. arxiv.org/abs/2406.18521
• 3.Roberts, J., Han, K., Houlsby, N., & Albanie, S. (2024). "SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretation." NeurIPS 2024 Datasets & Benchmarks. arxiv.org/abs/2405.08807
• 4.Masry, A. et al. (2025). "ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answering." Findings of ACL 2025, 19123–19151. arxiv.org/abs/2504.05506
• 5.Wang, X. et al. (2025). "ChartHal: A Fine-grained Framework Evaluating Hallucination of LVLMs in Chart Understanding." arxiv.org/abs/2509.17481

Abstention, honesty, and judge reliability

• 6.Kirichenko, P., Ibrahim, M., Chaudhuri, K., & Bell, S. J. (2025). "AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions." Meta FAIR. arxiv.org/abs/2506.09038
• 7.Wen, B. et al. (2025). "Know Your Limits: A Survey of Abstention in Large Language Models." TACL 13:529–556.
• 8."Blending Human and LLM Expertise to Detect Hallucinations and Omissions in Mental Health Chatbot Responses." arxiv.org/abs/2604.06216
• 9."The Illusion of Progress: Re-evaluating Hallucination Detection in LLMs." arxiv.org/abs/2508.08285

Vendor documentation and regulation

• 10.OpenAI. (2025-08-13). "GPT-5 System Card." cdn.openai.com
• 11.OpenAI. (2025-12-11). "Update to GPT-5 System Card: GPT-5.2," §3.7 Deception. cdn.openai.com
• 12.Google DeepMind. "Gemini 3.1 Pro — Model Card." deepmind.google
• 13.Regulation (EU) 2024/1689 (AI Act), Art. 13(3)(b), Art. 15(3)(4). artificialintelligenceact.eu/article/13 · /article/15

Pipelines and missing-data handling

• 14.Oracle AI. (2026). "Lightweight and Production-Ready PDF Visual Element Parsing." arxiv.org/abs/2604.23276
• 15."RAG-Anything: All-in-One RAG Framework" (Cross-Modal Misalignment). arxiv.org/abs/2510.12323
• 16.Figure and caption extraction alignment in astronomy literature (figure F1 90.9% / caption F1 92.2%). arxiv.org/abs/2209.04460
• 17."Gradient Importance Learning for Incomplete Observations" (downstream propagation of imputation error). arxiv.org/abs/2107.01983
• 18."Classification of Datasets with Imputed Missing Values" (counterevidence on imputation quality and downstream performance). arxiv.org/abs/2206.08478