Executive Summary

When we talk about an AI model's score we usually look only at the score. Which test, how many points, how much better than the last model. But a score needs a test paper first, and a test paper has someone who made it. Who wrote the items, who fixed the correct answers, who assigned the points. Those three are properties of the evaluation material rather than of the model. Which is to say that an evaluation is a dataset before it is a ruler, and a dataset ought to come with a quality history.

We already ask that of training data. Where did it come from, who labeled it, what was it derived from. We almost never ask the same of an evaluation set. A preprint that Chao Wang, an independent researcher, posted to arXiv on 15 September 2026 turns that question into a form. The study gathers papers that introduce or substantially update an evaluation resource and records the design each one chose in seven fields.

1.1The form that records a design

Here are the seven. What each one records is defined in a single appendix table, and that definition holds up every number in the study. The two this article follows are the fifth and the sixth, material source and scoring source.

The seven coding fields (paper, Table 1). The two in bold are what this article follows. The scoring field carries one qualification in its definition: it counts what produces the item-level score, not how those scores are later averaged together.

1.2From 1.15 million records down to 14,767

The study did not read every paper on arXiv one by one. It is a selective pipeline that filters at each stage, and the author puts that in the paper himself: "This selective pipeline may miss eligible papers before full-text assessment." The methods notes released with the repository are blunter still. The top-15% cut is "a conservative development choice, not an estimated population-recall guarantee."

Two things attach here. First, 14,767 is a count of paper records and not a count of distinct benchmarks. The paper nails this down in a sentence: a follow-up paper in the same benchmark family enters as its own record. Second, this distribution is not an adoption rate. The author states that nothing is weighted by citations, popularity or actual usage. So reading the 40.3% that comes later as "40% of the tests the industry runs" gets it wrong. It means that among proposed designs, that many chose that mechanism.

Leaving out the weighting was a choice rather than an omission, and the introduction gives the reason. Some evaluation resources barely appear in later experiments and still leave a record of a proposal about what models ought to be able to do. "Its value as evidence of a research proposal is therefore distinct from its influence as a measurement tool," the author writes. So this corpus is a record of what researchers asked for, not of what the industry uses, and every share this article counts is a distribution of demands.

What gets into the corpus carries a condition too. The thing being scored has to be an LLM. The coding instructions are explicit: "A model used only to create data, annotate, embed, judge, or process material is not the benchmark target." Carrying the word benchmark in the name, having open code, running many baselines — none of those qualify a paper on their own.

The limits of the selection also point in a direction. The author writes that monthly routing, the classifier's older training target and selective abstract triage may disproportionately miss early terminology and resources released alongside other contributions, and concludes that "differences across years or domains may partly reflect selection." Part of the difference between years or domains may be an artifact of the filtering rather than a real change. That qualification attaches to every trend below.

There is a boundary on scope as well. The corpus holds public arXiv proposals only. In the paper the collection covers identified arXiv proposals, "not all public or private evaluation resources." An evaluation set a company runs internally was never in here. And what the study looks at is design, not whether a test measures the concept it claims to measure. The author hands that question to other work in the limitations section, and the piece this blog ran ten days ago, The bias check AI labs report ranks models like a reasoning test, is the work on the receiving end of that handoff. This article asks about the design of the test paper. That one asks whether the name on the test paper fits.

The clearest change in the study is in the field recording what produces the score. The seats visibly swapped over the last two years. The figure below takes January through August only, so the years are cut to the same length, and shows what percentage of papers each of the five scoring mechanisms appears in. One paper can use several, so the five values in any one year add to more than 100%.

Put the three cohorts side by side and two mechanisms rise while three fall. A model reading the answer and assigning a score went from 25.8% to 40.3%, and running code or an environment and taking the outcome as the verdict went from 16.9% to 28.4%. Matching against an answer key written in advance went the other way, 72.6% down to 59.6%. It remains the most common mechanism by a distance, and it gave up 13 percentage points in two years.

The figure starts at 2024 so that only the cohorts with large denominators sit next to each other. Push further back on the same January-to-August basis and the spread widens a great deal. Model scoring went from 0.0% in 2022 through 13.6% in 2023 to 40.3%, and reference or metric scoring came down from 88.4% to 59.6%. The author attaches a qualification to that 0.0% himself: it is an observation about this selected corpus, not evidence that such scoring was absent from the field at the time. Execution scoring, meanwhile, sat still across three years at 17.4%, 16.0% and 16.9% before jumping to 28.4% in the last two.

Several scoring labels on one paper do not mean several judges look at one answer. The author's example is WebArena. On some tasks that evaluation checks the state of a website programmatically, and on others it asks GPT-4 whether an answer means the same thing as the reference. The paper states the consequence: "Multiple scoring labels can therefore describe different tasks within one benchmark, rather than multiple judges assessing every response." Two labels usually means the grading differs from task to task inside one test, not that a response gets checked twice. That distinction turns out to be decisive when we read the mixed designs in section 4.

The decline carries a qualification too. Reading the 13-point drop in reference and metric scoring purely as that mechanism being abandoned overstates it. The coding instructions say that assertions checking execution or interaction state belong on the execution side even when they use deterministic comparisons, and then the rule closes: "do not count that same checker again as reference_or_metric." So while execution scoring climbed from 16.9% to 28.4%, part of what reference and metric gave up may be the same checks filed in a different field rather than checks that disappeared.

There is a reason this particular movement is unusually solid. The author ran a worst-case scenario in which 5% of the labels in each year are wrong. Even under that assumption the change in LLM scoring stays between +4.53 and +24.49 percentage points. The range is wide, and the sign does not move. That is exactly what will contrast with the material side later.

A second scenario of a different character exists as well. Rather than flipping labels, it removes up to 5% of records per year as possibly ineligible, concentrating the removals in the most unfavorable direction. The increase in LLM scoring still does not fall below +10.02 percentage points. The author notes that he did not compound the two scenarios, and that neither one covers papers missed upstream. Shaking the result two different ways and getting the same direction is still worth recording.

2.1People did not get pushed out; they moved seats

Seeing human judging fall from 9.8% to 4.7%, it is easy to read that as people leaving evaluation. That reading goes wrong in two places.

First, the field counts direct scoring only. The coding instructions spell it out: "Human creation or review of references is not human judging." Writing the answer key or checking the reference answers files under reference and metric, and the material itself lands under human or real-world content. Those two values are still 59.6% and 79.0% in 2026. The human hand moved from the grading desk over to writing items and answer keys, and it gave up a majority in neither place.

Second, what shrank is the slice rather than the amount. Papers where people graded directly went from 165 to 268, a rise of 62%. The corpus itself more than tripled over the same span, which is what turns that into a halved share. LMArena, the most widely cited public leaderboard, still ranks models on human votes. The count of votes on a leaderboard and the share of papers in this study measure completely different units, though, so there is no putting the two numbers side by side and calling one of them larger.

2.2Agents are not what did this

The first explanation that comes to mind for more LLM scoring is agents. A system that uses tools and moves through an environment cannot have one answer written down for it, so it needs a judge, and evaluations of that kind grew, so judges grew with them. The author separated that explanation out numerically. Within papers about agents, LLM scoring went from 26.3% to 44.0%. Within papers that are not about agents, it went from 25.7% to 38.8%. It rose in both groups.

Splitting the total 14.51-point change in two makes the picture sharper. Only 0.56 points come from agent papers growing as a share of the corpus, and the remaining 13.95 come from change inside each group. This decomposition is an arithmetic allocation rather than a causal analysis, and the author says so in his claim ledger: "A symmetric arithmetic allocation under this grouping, not a causal effect." Stated precisely: the portion that the growth of agent evaluation accounts for is 0.56 of a point.

That this conclusion does not rest on one pair of years is visible in a table shipped with the repository. The author ran the same decomposition over ten windows, and in every one the composition share stays under a point. Take the longest window, January–August 2022 to January–August 2026: of a 40.27-point total change, composition accounts for 0.69 and the remaining 39.58 is within-group change. Over those four years and eight months, agent papers went from 1.4% of the corpus to 28.3%, a factor of twenty. The board was remade that thoroughly, and what it accounts for still does not reach one point.

2.3Twelve of thirteen domains

Cut the corpus by domain and the direction barely changes. In 12 of the author's 13 domains the LLM-scoring share rose, by somewhere between 9 and 25 percentage points. Medicine and biology moved most, 28.3% to 53.4% for a gain of 25.1 points, followed by language and communication at 23.7 and natural sciences at 21.0. In medicine that means one in two evaluation papers now hands the grading to a model.

Engineering is the one domain that went the other way, 32.8% down to 30.5%, a fall of 2.3 points. Blowing that single case up into a remarkable exception would be inaccurate. The 2024 engineering cell holds 61 papers, which is small, and what matters here is that one domain went the opposite way while twelve rose together. Code being run directly to grade it may already have been established in engineering, and this study's data cannot settle that reason.

The author's error scenarios back up that caution. Assume 3% of labels in each year are wrong and only two of the 13 domains keep a negative value, general-purpose and language and communication. Engineering's −2.3 points does not hold its sign under that assumption. We can say one domain went the other way. We cannot say from this data that model scoring actually declined in engineering.

The other field records who made the evaluation content. What happened at the grading desk did not happen here. But reading "did not happen" as "AI writing questions is still rare" inverts the fact. Most of it had already happened.

In 2022 it was one paper in seven. By 2024 it was one in two. More than tripled in three years. And then it stopped. The wording is exact: "an early expansion of model involvement in material creation coexists with a much less pronounced recent change." The early expansion came first, and what has happened recently is far less pronounced by comparison. This is saturation rather than absence.

3.1A value you cannot even call a rise

Match the January–August cohorts of 2024 and 2026 and the change in this field is +1.81 points. Small, but positive, which tempts you to write that it rose a little. Apply the error scenario from the previous section here and the range runs from −8.17 to +11.79 points. The sign flips. The author's sentence covers it: the four large increases stay positive under that assumption, while "Smaller material, fixed-collection, and English-only differences can change direction under the same scenario."

So the asymmetry this article is built on is most accurately written this way. On the scoring side the rise is certain. On the material side we cannot even say whether there was a rise. That is why the two roles cannot be put on a single automation dial and described as climbing together.

Could a loose definition be inflating that 51.3%? The opposite. The coding instructions attach this label only when a generative model created or substantially rewrote the evaluation content, and then they close the door: "Faithful copying, extraction, OCR or transcription of existing content is not generative authorship, even if an LLM performs it." Copying does not count even when a model does it. Human design of the rules or the generation prompts, and human curation and correction of generated items, do not by themselves make the material human either. Which makes 51.3% closer to a floor than to a generous count.

Some things are excluded entirely. The instructions place training data, construction instructions and judge prompts outside this field. So a model writing the rubric leaves the material field untouched. Model involvement on the ruler rather than on the items appears nowhere in that 51.3%. A good deal of the risk this article comes to later happens to live on the ruler.

3.2The items are still written in advance

The design that would show automated item writing most dramatically is a different one: producing new items during the test itself. That share is 6.7% in 2024, 7.0% in 2025 and 7.5% in 2026. Three years running it does not reach one in ten. Fixing the items in advance, meanwhile, sits at 95.7%, 95.3% and 95.0%, which is effectively motionless. Material written by people or recorded from the real world did fall from 86.3% to 79.0%, and it is still an overwhelming majority.

There is a common misreading here. Over the same span, evaluations requiring interaction tripled, 9.5% to 28.2%. Fixed items holding at 95% while interaction grows looks contradictory, and both values are true at once, because the coding instructions have the two labels judged independently. A collection assembled in advance can require closed-loop interaction. So the data does not yield a narrative about the era of static evaluation ending and the era of interaction arriving. What changed is how items get answered, not when they get made.

The author gives a worked example. Asuka-Bench fixes 50 web development tasks in advance and then lets a code agent take feedback from how its work ran in a browser and revise its own implementation. The task collection is fixed and the process of solving a task is interactive. Papers carrying both labels went from 7.8% in January–August 2024 to 25.1% in the same months of 2026. Fixed items stayed bolted at 95% while the way through them changed underneath.

None of this says that using a model-written test paper is itself the problem. When a synthetic evaluation set holds up and when it falls apart is a separate thing to measure, and this blog took that up once in Measuring the Quality of the Synthetic Data That Grades AI Agents. What this section says comes before that: the practice already reached half, and it has not climbed since.

The last two sections looked at each field on its own. The scene the common line describes appears where the two overlap: a test whose items a model wrote and whose points a model assigned. How much of that is there?

4.1One in five papers uses both

Across January 2022 through August 2026, among the 14,763 papers where both fields are valid, 3,272 contain model-generated material together with LLM scoring. That is 22.2%, roughly one in five. Taken as a number alone it looks like the common line is right.

Open those 3,272 up and the situation differs. Of them, 2,530 also use material written by people or recorded from the real world. And 2,020 also use a scoring mechanism other than an LLM. That is 77.3% and 61.7%, and the two groups overlap. The author's verdict is short. The combination is "better described as mixed design than as fully autonomous evaluation." Then he adds a sentence: mixture alone establishes neither supervision nor validity.

What that 61.7% actually consists of is written out by label combination in the aggregate file the author released. Counted again from the raw data, the figures match.

Combination counts from the author's repository file roles.json, recomputed from the raw data. One paper can use several mechanisms, so the first four rows sum to more than 2,020. The row in bold is the one to watch in this table.

The most common companion is a reference answer or a metric, and papers that also have a human judge number 226, which is 6.9% of the group. One in fifteen. The material side is lopsided too. Of the 3,272, some 2,163 use model material and human material together, two thirds of the group, and 367 bring in a program or simulator as well. The typical mixed design the author names does not look like people coming back in to grade alongside the model. It looks closer to a test paper mixing model-written and human-written items, graded partly by a model judge and partly by an answer key.

Widen the scorer side slightly and the count grows. Include learned scorers that are not LLM judges, such as embedding-based scoring, and papers overlapping with model material go from 3,272 to 3,696, from 22.2% to 25.0%. That is the second ruler the author built into his calculation code.

4.2Filter again and 359 remain

So what happens if only the designs that are not mixed are kept? Papers whose material source is exactly one thing, a generative model, and whose scoring source is exactly one thing, an LLM judge. No human material, no program or simulator, no reference answer, no execution check, no human judge. This value does not appear in the paper's text. It takes opening the aggregate file the author put in the repository and the code that produces it.

359 papers. That is 2.43% of all 14,763, and 10.97% even within the 3,272 where the two fields overlap. Two in a hundred, or one in ten if you count inside the overlap. And even that 2.4% is an upper bound. The paper gives the reason: "Paper-level labels do not identify the same model, component, or item across roles and therefore do not establish a self-evaluation loop." Inside those 359 there is no telling whether the model that set the items and the model that graded them were the same. So the number cannot be read as the size of a self-evaluation loop, and writing it as at most 2.4% is the right form every time.

Looking at how that number was produced attaches a qualification of a different character. The 359 counts papers whose material field is exactly one generative-model label and whose scoring field is exactly one LLM-judge label. The calculation code says so literally. But both of those fields accept multiple labels, and the author writes repeatedly that a label not being selected does not mean the property is absent. He writes it in the paper's body, in the appendix, in the data dictionary, in the methods documents, and inside the coding instructions themselves. The material-source section of those instructions ends this way: "Select supported origins; an unselected origin is not proof of its absence."

So 359 is a count built on fields being empty. And a field being empty means the paper was read that way, not that the test in question actually had no human material and no answer key. Where the previous qualification presses the number down, this one shakes it in both directions. For the same reason every share before and after this one comes with the warning the author attached. An unselected label is not evidence of absence.

4.3The two roles move in opposite directions

Split the combination of the two fields four ways and the asymmetry in this article shows up most cleanly. Each paper falls into exactly one of the four, so this time the values sum to 100%.

Papers using generated material alone fell from 32.6% to 26.0%, and papers using LLM scoring alone rose from 8.9% to 15.0%. That is the simplest evidence that the two roles went opposite ways. And papers where a model entered neither role are still one in three, at 33.7%.

4.4The coupling did not tighten

Papers using both went from 16.8% to 25.3%. Does that license saying the two roles increasingly travel together? The author blocks that reading in advance. As each field becomes more common on its own, the chance of the two overlapping by coincidence rises with it. Measure actual overlap against the amount coincidence would predict and the ratio falls: 1.32 in 2024, 1.24 in 2025, 1.22 in 2026. Co-occurrence grew while the pull between the two weakened.

That decline survives a change of window. Count by full calendar year instead of matched January–August cohorts and the ratio runs 1.39 in 2023, 1.31 in 2024 and 1.25 in 2025. The author adds that most of the rise in co-occurrence itself comes from the baseline rising as the two labels each became common. Overlap did grow, and most of the reason it grew is that coincidence got bigger.

What is remarkable is how the author recorded this. The repository contains a file called claim_ledger.csv, a machine-readable table of the claims he can make and the ones he cannot, each with a grade. The status value on row C06 is stronger_interpretation_not_supported. The statement reads: "Joint prevalence increases, but lift changes from 1.32 to 1.22; an increasingly strong association is not established." The very belief this article set out to check, that the loop where AI writes and AI grades is tightening, carries a not-supported tag the author attached himself.

This report measured the same question once more with a different ruler. Taking the author's released raw data, we split the 359 by year and attached denominators. By raw count the jump looks dramatic, 12 papers in 2023 to 160 in January–August 2026. Over the same span, though, the number of valid papers per year went from 861 to 5,734. With denominators attached, the share of the whole runs 1.39% to 2.79%, a doubling in three years that still does not reach 3%. And inside the papers where the two fields overlap it is 14.81% in 2023, 11.11% in 2024, 10.59% in 2025 and 11.03% in 2026, which is standing still.

4.5So why is it a problem anyway

Read only this far and you could land on nothing to see here. That conclusion is early. The 359 counts whether the two roles are separated on paper, not whether the two judgments are independent of each other. The risk the author raises sits exactly there: "nominally separate model roles need not provide independent judgments." Roles separated in name carry no guarantee of independent judgment.

He has an experiment to point to. The preference leakage study Li and colleagues put out in 2025, accepted at ICLR 2026. Unlike the main paper, which is still a preprint, this one went through peer review. It measures in a controlled setting what happens when the model that produced the training data and the judge model doing the scoring are related to each other. It defines three kinds of relation: the same model, an inheritance relation, and the same family.

The sharpest scene is in one appendix table. The same pair of answers, shown to two different judges.

Taken from Appendix C.2 of the preference leakage study (arXiv:2502.01534v3, ICLR 2026). The pair of answers being compared is identical across the two rows. Only the judge changes, and the win rate on the same answers splits 55.1% against 36.8%.

Same answers, and the winner flips depending on who judges. Each judge picked the side trained on data from its own family. The study also reduces that tendency to a single score. Pairing GPT-4o with Gemini-1.5 produced 23.6% and 27.9% across two kinds of student model. And one qualification has to travel with those numbers. Pairing GPT-4o with LLaMA-3.3 produced −0.1%, meaning no leakage was observed. This does not happen with every pair. Leaving that out would commit the overstatement this article has spent its length refusing.

The author goes one step further from there. Citing what Messeri and Crockett published in Nature in 2024, he carries their warning that broad reliance on AI tools can narrow the questions, methods and viewpoints of science over into evaluation. Carried across, it becomes a conditional risk: if tasks that models can readily generate and judge receive disproportionate attention, and if the scores that come out then guide model selection and optimization, preferred behaviors could be reinforced. The author immediately draws a line, saying his own study does not establish that this feedback is happening, and moves instead to what to check. Those prescriptions are the three in section 6.

None of this is a blanket knock on LLM judges either. The MT-Bench work reported that a strong judge reaches agreement above 80%, the same level as agreement between people. The problem is independence rather than consistency. That same study named position bias and verbosity bias alongside self-preference bias, and preference leakage adds another layer on top. How a minor choice in judge design manufactures a score is something this blog covered in The LLM Judge Bias a Rating Scale Floor Can Manufacture, and the generous marks that appear when the model that made the data and the model doing the grading coincide were the subject of An AI Wrote the Answer Key That Grades Other AIs.

To put it together: fully closed designs are at most 2.4%, and that share has not hardened. That does not make the remainder independent evidence. Inside the 22.2% of mixed designs where both fields appear, the model that made the material and the model that produces the score may well be from the same family, and a paper-level label will not tell you. What counting established is scale. What counting cannot establish is independence.

Every share carried across so far has a stage behind it. Somebody had to fill in seven fields for 14,767 papers. No person did. One model did. If this is an article asking whose ruler produced an AI scorecard, the question has to be turned on its own evidence.

5.1One model read the full texts and filled the fields

The model that did the coding is GLM-5.3-Flash. The same model handled both stages, abstract triage and full-text coding. The author wrote down the settings too: reasoning effort high, sampling off, output as JSON in a fixed schema, output capped at 3,200 tokens. And there is one more constraint, on the input. The model receives extracted text rather than page images. The author concedes the consequence in the limitations: "PDF extraction may also lose information from tables, formulas, or layouts."

What the model read had already been filtered, though, and the thing doing the filtering hardly qualifies as a model. The local classifier from section 1 is a 1.5-megabyte file, character TF-IDF with weighted logistic SGD on top. The freeze record the author left says what it is. It also records zero API calls for the freeze and no GPU. That is the gate that dropped 980,000 of 1.15 million records without an abstract being read.

The basis for the top-15% line is in the same file. During development there were 222 hand-labeled records, 83 of them clear positives, and the classifier caught all 83 within its top 10% alone. The author writes that he added a five-percentage-point margin on top and set the line at 15%. The qualification carried in section 1, that this value is not a population-recall guarantee, is attached right there in that record. The sample behind it is 222 records drawn from three held-out months, and that is all of it.

The gate and the main review were also holding different definitions. This classifier was trained toward "Papers whose primary or dominant contribution is a reusable, scored benchmark release," while the final eligibility criteria instruct the opposite, not to rank the paper's contributions and to accept a benchmark that arrives alongside other work. The author notes that the mismatch may affect recall and is not corrected by the full-text review downstream. A paper the gate dropped in error never comes back at any later stage.

None of this is grounds for taking the author down. Hand-coding four years and eight months of corpus is not something one researcher without an institution behind them can do. What matters is what came next. He built a table himself, and published it, showing how far the labels made this way have been verified.

5.2What was verified and what was not

The author drew samples at four points in the pipeline and read them again. The results sit in one appendix table.

Appendix Table C4 of the paper. Re-reading 100 of the finally included papers confirmed 99 as eligible, and drawing 100 records from the uncertain pool whose full text was never read turned up 1 that should have been in.

There is a calculation you want to do here. One in 100 out of 25,079 uncertain records, so roughly 250 papers were left out. The author explicitly blocks that calculation. The sample is sparse, other exit points are not covered, and "Population estimates would require the recorded sampling weights." This article carries only the fact that re-reading 100 records turned up 1 that belonged.

So how should that estimate be made? The author already wrote the answer down. He just did not run it. The required-validation entry in the classifier freeze record specifies the whole procedure: leave June 2024 entirely untouched by model development; take every record the classifier retained from that month, plus 500 near the threshold and a random 2,000 from the excluded side, and put them through the frozen abstract triage; then read the full text of every survivor on the low route. The claim-boundary entry in the same file is one line. Population recall remains unknown until probability-sample auditing runs on that untouched month.

This report pulled the public repository and found no file holding the results of that audit anywhere in it. The plan was recorded and released unexecuted. You can read that as a defect. Then again, research that writes down what it does not yet know, in the form of a procedure someone else could run in its place, is not something we see often. When a gap has a name and a method attached, the next person can start from there.

And the part that matters most is not in that table. There is no ambiguity about what those checks covered: "These checks assess eligibility, not the accuracy of the seven design fields." What the re-reading confirmed was whether a paper belonged in the corpus, not whether the label in its scoring field was right. The 40.3% and the 51.3% and the 2.4% all stand on those labels, and the accuracy of those labels was never measured.

Why it cannot be checked after the fact becomes clear from the coding prompt. Its last lines instruct the model: "Do not return a decision summary, benchmark name, evidence, citations, quotations, reasons, confidence, markdown, or extra keys." Why a given paper was classified as LLM-judged, which sentence in the full text the model took as its basis — none of that was kept anywhere. The data dictionary says the same. The released output files contain neither PDF text nor model reasoning.

5.3The arithmetic reproduces and the labels do not

This is where the boundary of reproducibility falls. The repository the author released rebuilds every table and figure from one command. In the documentation's own words, after installing dependencies "No LLM API key, paid call, PDF download or network connection is needed." So a value like the 359 is something anyone can confirm on their own machine.

What that reproduction covers is narrower than it sounds: "Deterministic reproducibility concerns analysis of archived outputs." The analysis run over archived outputs is deterministic. Producing the labels again is not part of it and could not be. The same record gives the reason. An API identifier does not freeze a provider's weights or guarantee identical responses in future. There was normally one request per paper, and the record states that papers were not re-run to pick a preferred answer. Each verdict landed once, and there is now no way to take that one pass back.

Meanwhile the validation file in the same repository carries a key reading "no_independent_semantic_accuracy_claim": true. The status value is calculation checks passed, and the caveats say that "Nonrandom screening and model coding errors are not corrected by arithmetic QA." Whether the numbers were added correctly is reproducible by anyone. Whether the labels are right is reproducible by nobody. That one sentence is the conclusion of this section.

The standing of the analysis is something the author declares up front as well. Every comparison in the study was designed after looking at the data, and none of it is preregistered confirmatory hypothesis testing. He writes this three times, in the paper's methods section, in the appendix, and in the repository documentation, adding that results were not selected on statistical significance. So the asymmetry traced above is not a hypothesis that passed a test set in advance. It is a shape that stood out in the data and survived being shaken across several windows and several assumptions. Those two things carry different weight, and since the author wrote the difference down himself, carrying it across means writing it down too.

Errors he found himself are not hidden either. A diagnostics folder holds the record of five targeted re-checks aimed at the interaction label and twenty language checks made during development. Two of those confirmed an omission in the 2026 cohort, and the paper states that adding those labels moves that cohort's share by about 0.035 percentage points. The author's own framing for the record is that these are "exposed, selected, AI-assisted source checks, not a new independent human sample or an estimate of population accuracy."

Since the author notes that cohorts are grouped by the date a paper was first posted while the labels describe the version that was read, we counted in the raw data how many records have those two dates falling in different years. Of 14,767 papers, 2,527, or 17.1%. That share varies by cohort: 36.2% in 2022, 41.1% in 2023, 32.4% in 2024, 23.3% in 2025, and 0% in 2026. The observation window closes in August 2026, so a 2026 paper cannot have a later version to be read from.

That skew points against this article's conclusion rather than for it. In the January–August 2024 cohort used for comparison, 366 papers, or 21.7%, were read from a later year's version. If those papers brought in model scoring during revision, the 2024 value is inflated and the 2024-to-2026 change is measured smaller than it is. A check the author ran matches that direction. Keep only papers whose first submission and read version fall in the same year and the change grows from 14.51 to 14.64 percentage points. Both values reproduce from the raw data, and the 0.13-point difference comes from 366 papers leaving the 2024 cohort, which drops that year's share from 25.76% to 25.63%. The version-mismatch problem remains, and at least it does not work in the direction of inflating that asymmetry.

The claim ledger, which section 4 pulled a single line from, runs to seven rows. Each carries a claim, the evidence file, the selector that reproduces the value, and a caveat. The grades come in four kinds.

All seven rows of claim_ledger.csv from the author's repository. Every figure in the body sits inside this table, and the interpretation outside the table belongs to this article.

That is what the study measured. Now it moves to practice. The prescriptions here were not invented by this report, though. They are closer to a transcription of what the author wrote at the end of his paper, and saying so is better than not.

6.1Three anchors the author recommends

The author starts by drawing a line under his own work: "Our mapping does not establish these feedback processes." In its place he writes what to do instead. Check, in three ways, whether a model-mediated test remains tied to the requirement it was meant to measure. Validated execution outcomes, external task results, and domain expertise.

All three share a character. They tie the score to something that sits outside the model's judgment. Did the code actually run. Does a different task built with no connection to this test produce the same ordering. Does a person who knows the field look at this item and nod. A score with none of the three attached to it may be a restatement of what the model prefers.

6.2The seven fields turned into questions

The form in section 1 was built by a researcher to classify papers, and flipped around it becomes a set of questions available to whoever receives the score. When a vendor hands over a score sheet, mark which fields get filled and which stay blank.

The paper's seven fields flipped into questions. The two rows in bold are the material and scoring fields, and they are also the ones most often left blank in practice.

Five of the seven can usually be answered one way or another. Either the vendor documentation says so, or asking gets an answer. The trouble is the remaining two, the ones in bold. Who wrote those items and who assigned the score almost never appear anywhere. And as this article has shown, those two happen to be the fields that moved most over the period the study measured.

One field does get answered, and the answer is the problem. If the language you deploy in is not English, the last row is that field. Papers coded as English-only run to 82.5% in January–August 2026, against 79.4% in 2024. The author files this as a secondary result and attaches a caveat: English-only means only English was confirmed in the coded language list, not that other languages were confirmed unsupported. Even allowing for that, the question stands. Were the items on the test we want to cite written in the language we are going to use.

If a line gets added to the checklist, make it the quality of the items themselves. A test paper with wrong answers makes the score meaningless whoever the grader is, and a public score that does not reproduce in our environment makes the ranking somebody else's story. This blog took those two questions up in Is the AI failing physics, or is the question wrong? and Would that score hold up on your own codebase? respectively. The seven fields are the question that comes before those two.

6.3Regulation is not asking for those fields yet

Why this checklist falls to the reader right now becomes clear from opening the rules. The EU AI Act places an evaluation obligation on providers of general-purpose AI models with systemic risk. Article 55(1)(a) requires two things: "perform model evaluation in accordance with standardised protocols and tools reflecting the state of the art, including conducting and documenting adversarial testing of the model with a view to identifying and mitigating systemic risks."

Run an evaluation, and put it in writing. There is no requirement in this provision to disclose who or what produced the score. Whether a model wrote the items, whether a model did the grading, whether that model shares a family with the system being evaluated — the provision does not ask. This is not written as a criticism of the regulation. It is written to establish that, as of now, the only parties in a position to ask for those fields are the ones procuring and the ones adopting. Article 55(2) of the same regulation leaves room for codes of practice or harmonised standards to stand in for the obligation, so whether those documents go further went unchecked here. What we did not check, we do not record as absent.

Pebblous has long said that a model's trustworthiness comes from the quality history of its training data. But the ruler that measures a model is also a dataset. It is made of items and answers and grading criteria, it came from somewhere, and somebody handled it. The provenance questions we put to training data without fail we have almost never put to evaluation sets. What this study did was turn that question into a seven-field form and actually apply it across four years and eight months. What DataClinic does for a dataset, this paper did for evaluation sets.

7.1The labels survived and the reasons did not

The line between what this study could release and what it could not falls at exactly the point Pebblous is pointing to when it talks about quality history. Whether the numbers were added correctly is reproducible by anyone. One command, no API key and no network. Whether the labels are right is reproducible by nobody, because the coding prompt instructed the model to return no reasons, no citations and no confidence.

Computational integrity and semantic accuracy are two things, and however solid the first is, it does not stand in for the second. This is also not the kind of omission that can be filled in later. Unless the basis for a verdict is written down at the moment the verdict is made, the only route back to checking a label is reading from the start again. Re-reading 14,767 papers costs what reading them the first time cost. As a textbook case for why keeping the reasoning alongside the grade is a decision with no recovery path, this is hard to beat.

Brought over to our side it comes out like this. When data quality diagnosis gets automated, grades are not the only thing to keep. Why this column was judged missing, why this label was read as inconsistent, the sentence and the row that carry the basis for that judgment — those have to survive with it. Keep only the grade and there is nothing to do when someone disputes it later except run the whole thing again.

7.2The question now has a name

The scorecard a vendor holds out in answer to "is this model any good?" can now be answered back with seven fields. A question with a name is easier to ask for, and a question that gets asked for turns into a field in the next document. We did not open vendor documents in this piece to check whether those fields are in fact filled, so what is left here is the checklist, with no verdict on current practice. Writing something up as checked when it was not is not an option.

The direction is clear enough regardless. What the study showed is that large changes happened in two fields, and what section 6 showed is that nobody formally asks for those two fields yet. The change happened and there is no box to write it in. On the data quality side we have watched that shape play out several times.

7.3Where this article stands

This article does not close on a product. What it leaves behind is the shape of a gap. The grading desk changed hands, item writing stopped at half, fully closed designs are at most 2.4% and even that share has not hardened. That is as far as counting goes. What counting cannot reach is how close the model that made the material and the model that produced the score are to each other in the remaining designs where a model entered both roles, and the label that would answer that is recorded nowhere at present. The paper went up eleven days before this article, single-authored and not peer reviewed, so anyone reading it this early should carry that with them.

The figures in the body come from three streams. Values from the body and appendices of reference 1 were taken directly from the arXiv full text, and the aggregate files and calculation code in repository 2 were downloaded and the values reproduced. References 3 through 7 are prior work that reference 1 builds on, and the quotations carried in the body were checked in each original. References 8 through 10 are works that reference 1 cites as examples and whose originals this report did not open, so they are grouped separately. Reference 11 is a regulation this report opened itself.

The backbone of this report (checked against the primary text)

• 1.Chao Wang. "What Do We Expect from LLMs? Mapping the Design of LLM Benchmarks." arXiv:2609.19182v1, submitted 15 September 2026. arXiv: 2609.19182 — a single-authored paper by an independent researcher with no institutional affiliation, and a preprint that has not been peer reviewed. The selection flow is §2.3, the seven field definitions are Table 1, the scoring and material figures are §3.3 and §3.4, co-occurrence and association strength are §3.5 and Appendix B.6, the error scenarios are §3.6 and Appendix Table B3, and the sample re-review is §2.5 and Appendix Table C4.
• 2.Public repository github.com/xxcg322/LLM-Bench-Map (code MIT, data CC BY 4.0) — the counts 359, 587 and 1,252 are absent from the paper's text and live only in this repository's data/roles.json and the calculation code data/inputs/roles_code.py. The year-by-year recomputation in section 4.4 was tallied by this report directly from the data/papers.csv raw data, confirming 14,763 valid records, 359 closed designs and 3,272 co-occurring records against the author's own aggregates. The claim ledger claim_ledger.csv, the validation record validation.json and the verbatim text of the coding prompt methods/fulltext_prompt.md also come from this repository. The classifier's identity, the basis for the top-15% cut and the mismatched training target in section 5.1, and the unexecuted audit procedure in section 5.2, are in methods/m0_freeze.json. The combination tallies of scoring and material in section 4.1 are recomputations from the raw data against the combination list in data/roles.json, and the version-year tallies in section 5.3 (17.1%, 366 papers in the 2024 cohort) and the 14.51-against-14.64 comparison were calculated by this report from data/papers.csv and set against the author's published values.

Mechanism and prior work (checked verbatim)

• 3.Dawei Li, Renliang Sun, Yue Huang, Ming Zhong, Bohan Jiang, Jiawei Han, Xiangliang Zhang, Wei Wang, Huan Liu. "Preference Leakage: A Contamination Problem in LLM-as-a-judge." arXiv:2502.01534v3, accepted at ICLR 2026. arXiv: 2502.01534 — the table in section 4.5 is the win-rate comparison from Appendix C.2, and the preference leakage scores of 23.6% and 27.9% along with the −0.1% pair where no leakage was observed come from Table 1 of the body. Unlike reference 1, this one went through peer review.
• 4.Lianmin Zheng et al. "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." NeurIPS 2023 Datasets & Benchmarks (arXiv:2306.05685v4). arXiv: 2306.05685 — the 80% agreement figure in section 4.5 comes from here, along with the naming of position, verbosity and self-preference bias.
• 5.LMArena (formerly Chatbot Arena) official documentation and Wei-Lin Chiang et al., "Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference" (arXiv:2403.04132, March 2024) — cited in section 2.1 as the representative case of people scoring directly. The more than 240,000 pairwise comparisons the original paper reports are a count of votes, while the 4.7% in reference 1 is a share of papers, so the two cannot be placed on the same axis.
• 6.Evelyn M. Kitagawa. "Components of a difference between two rates." Journal of the American Statistical Association 50(272), 1168–1194 (1955). doi:10.1080/01621459.1955.10501299 — the original source for the decomposition in section 2.2 splitting 14.51 points into a composition share of 0.56 and a within-group share of 13.95.
• 7.Andrew M. Bean et al. "Measuring what matters: Construct validity in large language model benchmarks." NeurIPS 2025 Datasets & Benchmarks. doi:10.52202/085713-0590 — the work on the receiving end of the handoff reference 1 makes in its limitations §5.2, where it writes that "The seven fields describe broad design choices rather than fine-grained abilities, adoption, or construct validity." This blog covered that paper on 16 September 2026.

Works reference 1 cites as examples (originals not checked)

The three below are works that reference 1 cites as illustrations or grounds for its own argument. This report did not open the originals separately, and what the body carries goes as far as how reference 1 described them. The verbatim text of those descriptions was checked in reference 1.

• 8.X. Wang et al. "Asuka-Bench: Benchmarking Code Agents on Underspecified User Intent and Multi-Round Refinement." arXiv:2606.05920 (2026). arXiv: 2606.05920 — cited in section 3.2 as the worked example of a fixed task collection and interaction coexisting in one test. This is the description in reference 1, §4.1.
• 9.S. Zhou et al. "WebArena: A Realistic Web Environment for Building Autonomous Agents." arXiv:2307.13854v4 (2023, revised 2024). arXiv: 2307.13854 — cited in section 2 as the example showing what several scoring labels on one paper actually means. This is the description in reference 1, §4.3.
• 10.L. Messeri, M. J. Crockett. "Artificial intelligence and illusions of understanding in scientific research." Nature 627, 49–58 (2024). doi:10.1038/s41586-024-07146-0 — the conditional risk in section 4.5 is the passage where reference 1, §4.3 cites this work and carries it over to evaluation. This report did not check the Nature original, so what is carried goes as far as reference 1's summary.

Regulation

• 11.Regulation (EU) 2024/1689 (the AI Act), Article 55(1)(a) — the verbatim text in section 6.3 comes from this provision. It requires providers of general-purpose AI models with systemic risk to perform model evaluation in accordance with standardised protocols and tools, and to conduct and document adversarial testing. There is no disclosure requirement in this provision covering who or what produced the score. Article 55(2) leaves room for codes of practice and harmonised standards to stand in, and whether those documents go further is something this report did not check.

Neighboring pieces on the Pebblous blog

• 12.The bias check AI labs report ranks models like a reasoning test — referenced in section 1.2. Where this article asks about the design, that one asks whether the name attached to the design fits.
• 13.Measuring the Quality of the Synthetic Data That Grades AI Agents — referenced in section 3.2. It takes up the separate job of measuring the quality of a model-written test paper.
• 14.An AI Wrote the Answer Key That Grades Other AIs, The LLM Judge Bias a Rating Scale Floor Can Manufacture — both referenced in section 4.5. The first covers the generous marks that appear when the model behind the answer key and the grading model overlap, and the second covers a minor choice in judge design manufacturing a score.
• 15.Is the AI failing physics, or is the question wrong?, Would that score hold up on your own codebase? — both referenced in section 6.2. They are the two questions to append behind the seven-field checklist.