Executive Summary

Key Numbers

All four numbers are values the paper published. The accuracy was measured in the mode that audits one named facility, and the token figure is an average over the papers used in testing.

Source: arXiv:2610.32135, posted October 23, 2026.

A large observing facility, once built, runs for decades. Decades of operating budget, the upkeep of the archive, and the decision whether to build the next instrument all hang on one question. How much has this facility actually contributed to peer-reviewed papers? That is the motive the researchers give for building the system. These are among the most expensive scientific installations humanity has ever put up, and the indicator that speaks for what they produced is, in practice, a literature tally and nothing else.

The way it is taken, though, is plain. If a facility's name appears somewhere in a paper, the paper counts toward that facility. The title of the paper aims at exactly this habit. The original reads Beyond Mention-as-Usage, which is to say: get past the arithmetic that treats a mention as a use.

Under that arithmetic, a paper whose introduction notes once that such a survey exists and a paper that drew every observation behind its conclusion from that facility are both a single count. What the two papers owe the facility is nothing alike. And then budgets are divided by that number.

The scale the system assigns has six levels. There is one criterion: how far the data out of that facility holds up the paper's scientific argument. It does not count how many times the name appears. It asks whether the conclusion still stands without that data.

Source: the L0 through L5 definitions in arXiv:2610.32135.

What the old tally counted as one splits five ways on this scale. L1 and L5 were the same single line in the ledger, but measured by what the facility contributed to the paper, they sit at opposite ends.

Assigning these levels by hand means reading the full text. Whether the name written in the introduction comes back in the data section, whether the figure carrying the result was drawn from that data: you have to follow all of it before the level is settled. Set against the number of papers a single observatory has to comb through in a year, no team of hands can keep up.

The route the system takes is to not push the whole paper into the model. It first cuts the PDF into a page-bounded outline and tags each section with a role, such as background, method, or result, and with whether a facility is mentioned there. A tool-calling model then pulls out only the passages it needs, when it needs them. The evidence gathered that way goes into an explicit decision tree, which settles the level. It is a structure that leaves a record of which sentence in which section was the basis for that level.

Which part of that structure carries the performance is something the paper shows by stripping the components out one at a time. Remove the ability to go back and read passages over several passes and macro-F1 falls from 79.8% to 62.9%; remove the tree-structured index over the outline and it lands at 66.6%. Those drops, 16.9 and 13.2 points, are the two largest among the four components tested. The paper records that these two were decisive above all for telling L3 through L5 apart. The stretch where you have to judge whether the conclusion genuinely rests on that data is exactly where the structure was needed.

The last step shows what kind of system this is. Every verdict carries a coverage-calibrated confidence score, and cases with a low score are routed to human review. There are two modes. One names a single facility and assigns only its level in that paper; the other first finds every facility the paper names and then assigns a level to each. The first is where 89.1% came from.

People built the exam first. From 8,184 papers published in ten journals between 2022 and 2025, they paired ten facilities against those papers to draw 79,407 paper-facility pairs. The facility list is not only LAMOST, FAST, and EP. The Hubble Space Telescope, Keck, the VLT, Subaru, Gemini, the GTC, and SDSS are in there as well. This is not a scale built around Chinese facilities alone.

Two people read each full text separately and assigned the levels, and a third adjudicator decided the cases where they split. Taking 2,000 of the pairs aside and measuring how often the two picked the same level gave a Cohen's kappa of 0.87. On a six-way split, that is how far people agreed with each other. Of the 79,407 pairs, 39,358, covering 4,093 papers, were used for testing.

The result is 89.1% accuracy and 79.8% macro-F1. Set beside the three approaches compared on the same exam, it reads like this.

Source: arXiv:2610.32135. The accuracy gaps are 5.8, 8.3, and 12.1 points respectively.

The paper points to L2 and L5 as the places where the gap opens widest. A paper that only talks about the instrument, and a paper that leaned on that one facility and nothing else: these are the two ends that even people cannot sort out without reading the full text. The paper does attach a caveat alongside, that L2 cases are rare enough on their own that no statistically firm conclusion can be drawn about them.

The values at those two ends run as follows. At L5 the F1 is 63.9%, where the best of the three compared approaches reached 50.0%. At L2 it is 62.3% against 48.5%. Ahead is the right word, but the fact that both values sit well below the overall macro-F1 of 79.8% has to be read alongside it. Splitting the call six ways is still on the hard side for machines too.

Two modes mean two scorecards. In the mode that first finds every facility a paper names, picking them all out is itself the test. Precision at that step is 86.5%, recall 80.4%, and F1 83.3%. That is the same number as the 83.3% in the table above, and it is not a typo. That one is the accuracy of feeding the whole paper to a model; this one is the F1 for listing facilities without missing any. Macro-F1 for assigning levels among only the pairs it listed correctly was 78.6%. In this mode too it came out ahead of all three compared approaches.

The system has not stayed a prototype built for a paper. It has been deployed at the National Astronomical Data Center, where it handles literature data for three facilities: LAMOST, FAST, and the Einstein Probe (EP). All three are large observing facilities China operates.

That does not mean people have taken their hands off it. Filtering once, before a bibliographer reviews the case, is all this system is asked to do. Verdicts that come back with low confidence go up to a person.

One design advantage the paper puts forward is that moving the system to a new facility calls for no retraining. It judges by level definitions and a decision tree rather than by a classifier tamed on data, so there is nothing to train in the first place. The limitations section of the same paper, though, records that this transfer has not yet been tested. What has been confirmed so far is performance within those ten facilities.

The same ledger exists outside astronomy. Anywhere that how much a resource was used gets counted by how many times its name was written down.

Provenance in AI training data is one such place. That a dataset's name sits in a model card or a datasheet says almost nothing about which of the model's abilities that data grew, and by how much. A small corpus that got mixed in once and the corpus that decided the model's character are the same single line on the list. Records of public data use look much the same. Count use by page views and downloads, and opening a file once and closing it counts as much as building a service on top of it.

Look again at what FacilityAudit-Agent did, and the larger part is less that it built a new model than that it redefined how to count. It put grades between a name on the page and a conclusion that stands on the data, and it got a machine to assign those grades consistently. What is worth taking from the astronomy case is that order.

• • Define the levels first. If what counts as L1 and what counts as L5 is not written down, neither a person nor a machine can repeat the same verdict
• • Have people label an exam set. The procedure in which two read separately and an adjudicator decides where they split, and the step of putting their agreement out as a number, belong here
• • Publish how far the machine's verdicts and people's verdicts agree. 89.1% is useful not because it is high but because which exam it came out of is written next to it
• • Settle in advance where ambiguous cases go. Without a route that sends low-confidence verdicts up to a person, even a six-slot scale ends up used like a one-slot ledger

When Pebblous talks about AI-Ready Data, what it puts next to quality is provenance and lineage. The state in which you can trace back which data contributed how much to which result. This paper shows, against 39,358 labels that people assigned, that once you turn that tracing into a graded scale, a machine can assign it on your behalf. Without the grades, all that is left is the fact that a name appeared.

Thank you for reading this far. The facts in this article were checked against arXiv:2610.32135, posted on October 23, 2026, and the deployment and the center's name were checked against the National Astronomical Data Center's own site. The analogy in Section 6 is not something the paper says; it is this article's reading from the side that works with data. We would be glad to hear how many slots your own organization counts data use in.

Pebblous Data Communication Team
October 28, 2026