Executive Summary

Key Numbers

Source: Mostafa, A. et al. (2026), arXiv:2609.18583v2 · per-field scored counts from Figure 5 of the same paper

The first sentence of the abstract states the problem the study set out to solve. The scientific literature holds decades of experimental measurements, and those measurements remain hard to reach as structured data for modern AI and data-driven research. Half the difficulty is that the information never sits in one place. The measurement lives in a figure, what the figure measured lives in the caption, how the sample was made lives in the body text, and the rest of the conditions live in a table. Each piece has to be found, joined to the others and checked against the source before any of it becomes reusable.

X-ray diffraction shows that difficulty at its sharpest. A diffraction pattern taken to confirm a crystal structure usually gets published as a single graph, and the array of numbers behind that graph goes nowhere. A human reader looks at the peak positions and accepts the conclusion; a machine receives a pile of pixels. Hundreds of papers can accumulate on the same material without any way to gather their measurements into one table.

Mining data out of the literature is not a new ambition. The paper traces the line from rule-based and natural-language-processing tools such as OSCAR and ChemDataExtractor, through domain-specific language models such as MatScholar and MaterialsBERT, to recent agentic approaches such as nanoMINER and DIVE that give a language model tools and let it work through a document in several passes. One thing runs through all of them. Most stayed with the body text.

The authors also draw a line between their work and opXRD, which shares a similar goal. Efforts like that provide valuable experimental data resources, they write, while ERAF4XRD takes on the much larger challenge of recovering the X-ray diffraction data and its context that sit inside the literature in a form no machine can read. One job gathers data that already exists somewhere in digital form. The other turns something that only ever existed on the page back into data.

ERAF4XRD runs in four steps with no human hand in the loop. Knowing what each step is responsible for changes how the numbers later in the article read.

2.1Collect the documents, then throw most of them out

The pipeline starts by downloading open-access and Creative Commons papers through APIs. Four sources are wired in at the moment: arXiv, Elsevier, Springer and CrossRef. Queries pair an element with a technique. To take the paper's own example, a search for copper-related XRD work asks for documents that mention Cu or Copper and also X-ray diffraction or XRD.

Reading every downloaded document in full would cost more than it is worth. A screening agent therefore loops through observe, reason and act, opening only the parts it needs. Each document gets six loops by default, and a document that produces no decision within those six is rejected. For collections that need closer inspection, the budget and the tool list can both be widened.

The authors also tested whether looping is worth it. On a 35-publication subset, GPT-5.2 reached the same 97% accuracy whether it looped or answered in a single pass, and the single pass used far fewer tokens and cost far less. Smaller models went the other way. GPT-4.1-nano climbed from 77% to 89% once its agentic budget grew. The weaker the model, the more the agent loop buys.

The same subset settled a second question, about how much of a document to hand over. Feeding in only the XRD-relevant passages scored 97%; uploading the whole paper scored 94%. More input made things worse. Irrelevant text such as literature reviews, bibliographies and acknowledgments dilutes the XRD-relevant evidence and eats the context window, the paper explains, and the problem bites hardest in agentic mode, where the model needs room to accumulate evidence across iterations.

2.2Find the figure and read what it shows

Getting figures out of a PDF takes two routes at once. Raster figures come straight from the embedded image objects, and vector figures get reconstructed by grouping the drawing primitives that make up the plot. A lightweight filter then screens the candidates by type, area, file size and proximity to XRD-relevant text before anything reaches a vision-capable model.

One experimental result here transfers directly to practice. Performance peaked when the model saw a high-resolution crop of the figure together with the full page it sat on. The two inputs complement each other, the authors explain: the crop shows the shape of the plot itself, and the full page shows the context the plot sits in. Give the model only one of the two and the judgment gets worse.

2.3Attach the scattered conditions to the right figure

Regular expressions pull candidate values out of the body text and tables, and an agent then decides which value belongs to which figure. The paper's own example shows what kind of work that is. Suppose a document reports Cu Kα radiation for powder XRD and Mo Kα1 radiation for single-crystal measurements. Both values are genuinely in that paper. Only the context around each figure decides which one attaches where. A metadata value may be correct in isolation, the authors write, yet incorrect for a particular experimental record if it is linked to the wrong figure.

Every field the system keeps carries the passage that supports it and the path by which it was extracted, so that any value can be traced back to the source later. Once the linking is done, every trace the agent left behind gets deleted. Tool-call logs, gathered evidence, decision records: none of it moves on. The point is to keep the validation agent from leaning on the reasoning that came before it.

2.4An independent check asks each value for its evidence

The last step has two layers. In the first, a validation agent goes back to the source document, looks for the evidence again, and marks each field supported, unsupported or unclear. This agent gets a default tool-call budget of twelve per metadata block, twice the six allowed during screening. A different model from the earlier steps can be seated in the validation chair.

The second layer uses no language model at all. The paper explains why the rules are needed: because the framework treats the source document as ground truth for as-reported metadata, a value can be copied correctly from the source and still break basic crystallography. Three pairings get checked. Does the space group match the crystal system, could the lattice parameters come from that crystal system, and do the radiation source and the wavelength belong together. A cubic space group such as Fm3̄m attached to a tetragonal system, or Cu Kα paired with a wavelength of 0.7107 Å, gets flagged. A field caught by these rules drops from supported to unsupported, and the inconsistency is recorded for a human to look at.

The benchmark is 273 hand-inspected publications. It splits into eight subsets and mixes material families from pure metals through high-entropy alloys, aluminum alloys, molybdenum and oxides. Those papers yielded 3,150 candidate figures, of which humans confirmed 282 as genuine X-ray diffraction plots. That is one real figure for roughly every eleven candidates, which is why the authors call this an imbalanced classification task. But the ratio swings hard between subsets. The oxide set collected from CrossRef ran at one real figure per 3.4 candidates, while the iron and copper set from arXiv ran at one per 52. A fifteenfold spread. No single model came out best on every subset, and the arXiv iron and copper set was hard for all four.

The 98.7% accuracy belongs to GPT-5.2, with Claude Sonnet 4.5 behind it at 98.1%. That number is more generous than it looks. When nine candidates in ten are not XRD, answering "no" to everything already clears 90%. So the paper also reports figures that exclude the correctly rejected negatives: for GPT-5.2, 94.8% precision, 90.8% recall and an F1 of 92.8%. Close to one figure in ten is already being lost at the identification stage. Gemini 2.5 Flash had the highest recall at 96.1% but only 79.7% precision, and Grok 4.1 had the lowest precision at 73.3%. Lean toward missing nothing and you take in more that does not belong.

Reaching for a newer model did not solve it either. On the generic subset collected from arXiv, moving from GPT-5.2 to GPT-5.5 raised figure-identification accuracy from 95.9% to 96.2%. Three tenths of a point. Speed does not follow the accuracy order. Per paper, Claude Sonnet 4.5 was fastest at 0.96 minutes, then GPT-5.2 at 1.29, Gemini 2.5 Flash at 1.72 and Grok 4.1 at 2.10. The fastest model also carried the highest API cost. For work at the scale of a literature corpus, the biggest and newest model is not automatically the answer.

The case for trusting these values comes from what the validation step actually changed. Across the full 273-paper benchmark the first pass produced 1,471 values, and validation disposed of them as follows.

Breaking the 71 removals apart shows what this step prevented. The validation agent caught 59 of them. Of those, 44 were values that appeared nowhere in the source document, which is to say the model had invented them; 11 were attached to the wrong figure; and 4 contradicted the source outright. The remaining 12 were caught by rules that use no language model: 11 radiation and wavelength mismatches, and one lattice parameter that did not fit its crystal system. Without this step, those 44 invented values would have entered the database as they were.

The results were also checked against a standard outside the papers. For five randomly chosen benchmark publications, the authors compared six fields against the crystallographic information files held in an open crystallography database: crystal system, space group, lattice parameters, chemical formula, formula units per cell and radiation type. Of 22 scorable fields, 18 agreed with both the source paper and the crystallographic file. Set aside one edge case with no extractable diffractogram and it becomes 16 of 17. The authors list the five-paper sample size among their own limitations.

How usable the final validated records are was settled by hand. The authors took 443 fields from 47 publications and matched them one by one against a curated reference. 402 agreed, 6 were wrong and 35 were missing. No value showed up that the source did not support. Precision of 98.5% and recall of 90.7% come from this exercise. It draws on a different sample from the 1,400 in the previous section, so the two sets of numbers should not be mixed.

Precision and recall say different things. Precision of 98.5% means you can trust what the system wrote down. Recall of 90.7% means one field in ten that ought to be there is empty. An empty cell looks safer than a wrong one, until you ask where the empty cells fall. The paper reports that the misses occurred mainly in lattice parameters, radiation type, wavelength, Wyckoff positions and X-ray density, and it counts them field by field in the body text.

That same figure records how many values were scored in each field. Lattice parameters lost 10 out of 48 scored values, one in five. Wyckoff positions lost 4 of 8, and X-ray density lost 4 of 6. Half, and two in three. Counted by volume, lattice parameters lead; counted by rate, the fields that appear rarely in papers are the ones that go blank most often. Anyone asking which column they can trust needs the ranking by rate.

Some fields were almost never missed. Crystal system, temperature, the software used, the databases referenced, step size, peak positions, full width at half maximum and formula weight came back at close to complete agreement. The 6 values that were present but wrong fell in space group, space-group number, radiation type, wavelength, Miller indices and scan range.

The line between the two lists is sharp. The fields it catches are the ones that can be read off the plot or that travel with the caption. The fields it loses belong to the experimental conditions. Which X-ray source was used, what the wavelength was, what the lattice parameters refined to: those tend to live in one sentence somewhere in the body text rather than in the result graph, and a sentence that goes unfound leaves a blank. The results survive and the conditions drop out.

Summed across models, the figure-identification stage took 208 non-XRD figures for XRD and missed 92 real ones. More than twice as many errors of inclusion. The paper sorts them by type. On the inclusion side: diffraction patterns that are not X-ray, ordinary graphs that happen to have peaks, misread axes and legends, and two-dimensional maps that look like diffraction. On the missing side: unconventional ways of presenting the data, low-resolution figures, map-style diffraction, and problems in the structure of the PDF itself.

Failure has a different shape in each model. Broken out by model, GPT-5.2 accounts for 14 of the 208 false positives, the fewest, and all of them in a single category. Claude Sonnet 4.5 produced 33, mostly by misreading axes and legends. Gemini 2.5 Flash produced 69 and Grok 4.1 produced 92, spread evenly across all four categories. The 92 misses reverse that: Gemini has the fewest at 11, Claude and GPT-5.2 have 26 each, and Grok has 29. When failures cluster in one place, a human only has to guard that place. When they scatter, everything needs watching.

The most interesting decision in this framework is a design principle rather than a performance number. The paper says it prioritizes source-supported information and validated data-metadata links over maximum completeness. The records that remain are the ones with solid evidence in the source, and information that is uncertain or could not be extracted stays missing rather than entering as an unsupported record. Filling the table was not the goal. Filling only what could be trusted was.

To anyone who has worked on data quality for a while, that choice is both familiar and rare. The pressure to raise a completeness metric is strong everywhere. A database full of blanks looks unfinished, and a database full of values looks done. But a value filled in without evidence cannot be told apart afterwards. Had those 44 invented values gone through, they would have sat in the table in exactly the same shape as the 1,400 that survived. The paper gives the same reason: in experimental databases built for quantitative analysis and machine learning, incorrect individual records can propagate into erroneous downstream scientific conclusions.

Which is why the most valuable output here, for practitioners, is Figure 5 rather than the 98.5% precision. It is the figure that counts and publishes which fields go empty most often. Knowing that lattice parameters came up empty ten times tells anyone using that database what to expect from the lattice-parameter column. A blank you know about and a blank you cannot explain are entirely different objects when it comes to reuse. Scientific data gets reused on the strength of a map of its absences, not on precision alone. The team put the benchmark, the ground-truth data, the framework outputs and the analysis spreadsheets on Zenodo and released the code as well. When you publish a map of the gaps, somebody else can redraw it.

Where the missing fields landed is worth a second look too. Lattice parameters, radiation type, wavelength and Wyckoff positions are the first things you reach for when you want to repeat a measurement or set it beside another one. If peak positions survive well and the conditions that produced those peaks fall out, the resulting database serves search but comes up short for reproduction. That is not a question of accuracy. It is a question of which way the data leans.

A team trying to recover measurements buried in the literature could start by checking four things. This list is our reading of the paper's design and limits, translated toward practice.

• Are the side that produces values and the side that checks them kept apart? A number like zero unsupported values is hard to reach when one model grades its own answers.
• Has the miss rate been counted field by field? A single overall recall figure says nothing about which column you can rely on.
• Are the contradictions a machine can catch written down as rules? The 12 caught by rules in this study are the ones the language model let through.
• Do blank, zero and not-measured get recorded as three different things? Once they share a cell, there is no way back.

Thank you for reading this far. Every number and design decision this article cites can be checked by anyone in arXiv:2609.18583. We would be glad to hear how your organization recovers measurements that survive only in documents, and how you mark the blanks once you have.

• 1.Mostafa, A., Ratcliff, W., Billinge, S. J. L., Abdolrahim, N. (2026). "ERAF4xrd: A multimodal agentic framework for constructing validated experimental X-ray diffraction databases from scientific literature." arXiv:2609.18583v2.