Executive Summary

If you want to know how good a robot model is, you open its model card. The exam laid out there differs from company to company. The benchmarks Google grades on and the benchmarks Alibaba grades on barely overlap, and where they do not overlap the cell is simply empty. Put models on the rows, benchmarks on the columns, fill in the published scores, and the table is mostly holes. A preprint posted to arXiv on 26 August takes that hole-ridden table itself as its data.

The authors set out two consequences of those holes. First, there is no shared axis on which models can be compared. Physical AI, the paper notes at this point, has no MTEB-style leaderboard of the kind text embedding has. Second, the numbers that do get reported can mislead. A headline score is usually an average over the chosen benchmarks, and when those benchmarks measure overlapping abilities the average quietly counts the shared signal twice.

The paper's example is pointing. That single capability, outputting the coordinates of a place to grasp or an empty patch of table, gets graded several times inside one report. Gemini Robotics-ER 1.5 reports Point-Bench, RefSpatial, RoboSpatial-Pointing and Where2Place; RoboBrain 2.0 reports RoboSpatial, RefSpatial-Bench and Where2Place; Qwen3-VL reports RefSpatial and RoboSpatial-Home. The same repetition shows up in 3D configuration reasoning and relational question answering, the paper adds.

1.1We Opened the Three Reports Ourselves

This is one of the few claims in the paper that can be checked from outside it, so we pulled the three model reports and read them against it. All three said what the paper says they say. And the real situation is worse than the paper's description of it.

Google DeepMind's Gemini Robotics 1.5 report states how it computes its embodied reasoning score: "The embodied reasoning score is a weighted average of 50% spatial reasoning benchmarks and 50% question answering benchmarks (both image and video)." The report puts four benchmarks in that spatial reasoning bucket, and all four are pointing benchmarks.

The table also shows that the weighting rule is not just prose but the arithmetic that produced the total. Mix the spatial bucket mean of 52.6 and the question answering mean of 66.5 in equal parts and you get 59.55; the embodied reasoning score printed in the same table is 59.6. They agree to one decimal place.

Convert that into per-benchmark weights and a single pointing benchmark carries about 12.5% of the score while a single question answering benchmark carries about 4.5%. One pointing benchmark weighs more than two and a half question answering benchmarks. And two of those four, RefSpatial and Where2Place, are exactly the pair the audit paper flags as substitutes for each other.

BAAI's RoboBrain 2.0 report says it evaluated on nine spatial reasoning benchmarks and lists BLINK, CV-Bench, EmbSpatial, RoboSpatial, RefSpatial-Bench, SAT, VSI-Bench, Where2Place and ShareRobot-Bench. Eight of the nine are among the audit paper's 12, and those eight contain both substitute pairs the paper identifies: EmbSpatial with CV-Bench, and Where2Place with RefSpatial-Bench. The paper used only the pointing pair as its example and never notes that both pairs sit inside a single vendor's suite. Alibaba's Qwen3-VL report uses five benchmarks in its embodied bucket, ERQA, VSI-Bench, EmbSpatial, RefSpatial and RoboSpatial-Home, with RealWorldQA from its general question answering bucket and BLINK from its multi-image bucket on top.

1.2How the Holes in the Table Were Filled

The authors began by indexing 51 physical AI benchmarks from model cards, papers and official blogs. This is where the number 51 first appears, and here it counts benchmarks. Indexed the same way, the models numbered 152. Three criteria narrowed the 51 benchmarks to 12: at least five models must have reported a score on it (density), it must recur in recent model reports (recency), and together the set must cover the main tasks physical AI is asked to do (diversity). The last criterion is defensive. It keeps the authors from picking twelve variants of one task and then announcing that they found redundancy.

Those 12 benchmarks were then used to filter the models. Keeping only models with scores on at least two thirds of them, 8 of 12, left 51 of the 152. That is the second appearance of 51. This one counts models and has nothing to do with the first. The surviving models were released between 2024 and 2026, come from 15 providers, mix open-weight and closed, and range from 1B to 241B parameters.

Fifty-one models by twelve benchmarks makes 612 cells. Published scores alone did not fill enough of them to compute covariances, so the authors ran the empty cells themselves. They used each benchmark's official evaluation code and prompts as published, with greedy decoding and that benchmark's own answer-parsing rules. That produced 159 cells from their own runs. In the end 564 of the 612 cells were filled and 48 remained empty.

The appendix breaks down where the 564 filled cells came from. 405 came from published values, and those split three ways again. 379 were copied straight from a model card or paper, 25 were medians taken where published values conflicted with each other, and one was borrowed from a twin model. The remaining 159 are the authors' own runs. That figure of 25 comes back later.

How the third criterion, diversity, was actually met shows up in the grouping the authors attach to the set. The twelve fall into five groups. The two pointing benchmarks (RefSpatial-Bench, Where2Place) answer with coordinates and come closest to the format a robot policy actually consumes. Four single-image relational benchmarks (EmbSpatial, CV-Bench, OmniSpatial, RealWorldQA) show one scene and ask about relative position, depth, count or appearance from another viewpoint. Three multi-view and video benchmarks (MindCube, SAT, VSI-Bench) cannot be answered from a single frame: MindCube supplies two to four views of one scene, VSI-Bench a walkthrough video. Two embodied benchmarks (ERQA, RoboSpatial) frame the question from a robot's point of view, asking what can be grasped, where an object may be placed and whether a configuration is feasible. BLINK stands apart on its own as a general visual perception benchmark.

The grouping comes with a caveat. The authors are explicit that they use it only as the taxonomy a reader would expect and not in the analysis. They did not pre-group the benchmarks and then measure correlations within the groups. The redundancy diagnosis that follows is computed from the scores alone, blind to these five groups. That is why the results land in places the group boundaries do not predict. In section 5, both the bond that survives to the end and the one unexpected negative correlation cut across groups.

Below is the profile of the final twelve. The spread in item counts stands out: a 100-item benchmark and a 21,000-item benchmark sit in the same table. The Gini coefficient g measures how widely a benchmark spreads the models apart, and it returns in the selection procedure of section 4.

1.3One Name Points to Several Exams

What those 25 cells look like, the ones where published values conflicted and a median was taken, is visible on a single page of Google's report. In one table for one model, RoboSpatial-Pointing is listed at 31.1 and RoboSpatial-VQA at 79.3. That is a 48-point gap. Qwen3-VL uses yet another variant, RoboSpatial-Home. But in the audit paper's matrix RoboSpatial is one column, and that column's scores run from 29.4 to 72.6. The 79.3 falls outside that range.

Whichever value went in, the other one cannot also be in that column. This is not to say the authors got it wrong. Taking only a particular split may well be what the official protocol requires. The accurate statement is this: one benchmark name refers to different splits in different reports, a single matrix column has to pick one of them, and that pick is not recorded in the matrix. It is a different kind of problem from the missing values and outliers you meet when treating benchmark scores as data. What wobbles here is not the value but what the value refers to.

Nor is it only names that wobble. Setting two of Google's tables side by side, we found a place where the same benchmark score for the same model is printed twice in the same report with different values. The table that gathers all 15 benchmarks onto one page puts Gemini 2.5 Pro's Where2Place at 37.0; the later table that breaks pointing down by sub-task puts the same cell at 22.0. A 15-point gap. The other six cells in that row are identical to the decimal across both tables, and the RefSpatial row just above differs only as 49.3 against 49.2, which is rounding. Only that one row disagrees.

So the 25 cells where the audit paper found published values in conflict are not only something that happens between reports. It happens inside one report as well. From the point of view of whoever is building the matrix, cells like this are hard to even notice. Both tables live in the same document, so there appears to be only one source to check against in the first place.

The skeleton of the matrix, though, does reproduce from outside. We counted, in each of the three model reports, only the benchmarks that belong to the audit paper's twelve, and matched our counts against the published-cell counts in the paper's appendix.

All three match. The authors make no claim to have verified the published scores themselves, but the skeleton of which benchmark came from which report does yield the same counts when recounted from outside. Other defects that make benchmark scores hard to trust, such as test items leaking into pretraining data, are out of this audit's range. What this paper asks is not whether the scores are right, but whether, granting that they are, the twelve are measuring twelve different things.

This kind of audit is not new. Text LLMs have precedents. Metabench applied item response theory to the 28,632 items of six LLM benchmarks and showed that the scores of more than 5,000 models could be reconstructed from under 3% of the items. Burnell and colleagues factor-analysed 29 LLMs across 27 tasks and extracted three capability factors, which the Metric paper cites as explaining 82% of the variance.

Both precedents need item-level response data: which model got which item right. Physical AI has no such data. Aggregate scores are all that gets published. That is where this paper makes its trade. It gives up resolution, meaning it cannot say which items are redundant, and in exchange answers the same question at the benchmark level. What it buys is reach. With aggregate scores and enough overlap, the procedure works in any field. The authors state in their conclusion that the audit uses nothing specific to physical AI.

2.1All 66 Pairs Came Out Positive

The first computation is pairwise rank correlation. Benchmarks differ in difficulty, so comparing raw scores mixes difficulty in with information. In the paper's table, mean scores range from the low thirties to 82. If two benchmarks measure the same ability and differ only in difficulty, the level and spread of the scores will differ but the order they put models in will be the same. So the authors take Spearman correlation, computed on ranks, as their primary measure, and z-standardise each column before entering any multivariate analysis.

Twelve benchmarks yield 66 pairs. The mean correlation was 0.487, and all 66 were positive. Not one negative correlation means that a model doing well on any one benchmark generally does well on any other. Two pairs came in above 0.8.

• EmbSpatial and CV-Bench, ρ = 0.876 (95% CI 0.78–0.93, n = 49)
• Where2Place and RefSpatial-Bench, ρ = 0.860 (0.73–0.93, n = 50)

Next highest are ERQA with RealWorldQA at 0.758 and RealWorldQA with OmniSpatial at 0.748. At the other end sits BLINK. It is the most distinctive benchmark by correlation, and it forms its own branch in the hierarchical clustering dendrogram.

2.2Can the Other Eleven Rebuild the Twelfth?

Comparing two at a time is not enough. A benchmark may have no single twin and still be reproducible from a combination of the other eleven. The authors ran a ridge regression predicting each benchmark from the remaining eleven and computed a leave-one-model-out cross-validated coefficient of determination. A high value means the benchmark can be rebuilt from the rest; a low one means it gives something the others do not. The table below shows the three at each end.

The median benchmark has about half of its variance recoverable from the other eleven. The three worst offenders are Where2Place, RefSpatial-Bench and ERQA, and those three keep turning up as each other's predictors. Read the right-hand column downwards and MindCube appears often: it is the top predictor in seven of the eleven regressions, acting as the hub of shared behaviour in this suite. ERQA and RefSpatial-Bench follow at five each.

The authors attach a caveat here. A low R² may mean the benchmark carries distinctive information, or it may mean the benchmark is noisy and therefore unpredictable from anything. Separating the two requires repeated evaluation under resampled prompts, decoding seeds and parsing rules, and current reporting practice does not supply that data. They also note that redundancy is not a one-directional axis. A heavily overlapping benchmark is useful as a representative of the shared structure, and a distinctive one is useful as complementary evidence. That distinction is what the selection procedure in section 4 rests on.

2.3Where Did These Exams Come From?

The redundant three appear to have something in common. Working from the introducing citations in section 2 of the paper, we classified each benchmark by whether it came from an independent paper written to evaluate something or was released alongside a model. This classification is ours, not the paper's, so we give the evidence with it.

Held up against this classification, the two substitute pairs behave differently. Where2Place and RefSpatial-Bench were both released alongside a robot pointing model. That two exams from the same lineage, RoboPoint and RoboRefer, should substitute for each other makes sense. EmbSpatial and CV-Bench, by contrast, have different lineages. One is an evaluation-only paper, the other a by-product of the Cambrian-1 model paper, and they still sit together at 0.876.

That the three most redundant benchmarks are all model-release by-products is worth noticing. Exams released to show off a model's strengths do appear to resemble one another. But there is a counterexample. RealWorldQA, the most distinctive benchmark of the twelve, is also a model-release by-product. With only twelve observations no statistical test is available, so we cannot go so far as to say provenance determines redundancy.

Redundancy on its own is not yet a practical problem. The problem arrives when a total is laid on top of it. The authors took the arithmetic mean of each model's 12 scores and lined the models up from high to low, which is the same computation a model card performs. But give all twelve equal weight and the weight on any given ability becomes proportional to how many times the suite measures it. In this matrix, pointing takes two of the twelve columns and spatial reasoning over video takes one.

So the authors replaced each pair the previous section flagged as substitutes with the mean of its two columns. RefSpatial-Bench and Where2Place become one column, EmbSpatial and CV-Bench become another, and with the eight untouched benchmarks that leaves ten columns. The arithmetic mean and the ranking were recomputed over those ten. Nothing was removed; what had been counted twice is now counted once.

Of the 51 models, 22 moved by three places or more. That is over 40%. The direction split as you would expect.

The weight the authors perturbed is two of twelve columns, 16.7%. The actual weight in Google's report, as established in section 1, is half. The two situations differ in suite composition and in how the total is defined, so the number 22 cannot simply be carried across. What carries across is the direction. The more weight rides on a duplicated ability, the further the ranking tilts toward it. The condition the paper worked with was the milder of the two.

What to take from this section, then, is not a new leaderboard. No sentence in this paper says that some model is really eight places lower. What it offers is a sensitivity measurement of how much a ranking depends on how the suite was assembled, and the fact that touching two columns moved more than 40% of the models.

If the twelve overlap, the same evidence should be obtainable from fewer of them. Which ones to keep, and how far the set can be cut, is not something the previous section answers. The authors build the selection procedure on a single premise. Informativeness is not a property a benchmark holds on its own; it depends on what has already been picked. A benchmark that is indispensable in isolation becomes worthless next to its substitute. So the selection proceeds one at a time.

To earn a slot, a benchmark has to satisfy two things. First, it has to spread the models apart. A benchmark that gives every model roughly the same score is ordering them by noise, however unusual the ability it measures, and contributes nothing to a leaderboard. Second, it has to carry information the already-selected benchmarks do not. A benchmark reproducible from the current set adds no new evidence.

Discrimination alone is computed on raw scores. Every column is converted to z-scores before the multivariate stage, but the Gini coefficient is taken from the raw distribution before standardisation. Difficulty is not discarded so much as put back to work as a selection criterion. A benchmark saturated at the ceiling, where every model scores 90, is filtered out here whatever it measures.

4.1Four Steps Reach 78.5%

Starting from the empty set, R² is zero and utility reduces to discrimination alone. So the first pick is the benchmark that spreads the models widest, RefSpatial-Bench. From there each step recomputes the utility of every remaining candidate and adds the largest. The authors did not stop at four; they ran all twelve steps, so that where to stop could be read off the curve rather than fixed in advance.

The 78.5% is 78.5% of what? Not accuracy. It does not mean the twelve-benchmark ranking is reproduced to 78.5%. It is the cumulative share the four take when the utility, the product of discrimination and marginal information, is summed over all twelve and set to 100. The eight benchmarks from fifth to twelfth divide the remaining 21.5% between them, and the final four contribute 4.4% together.

The most instructive line in the table is the fifth. Where2Place has a discrimination of 0.265, second highest of the twelve. Judged alone it should be the second pick, yet the procedure passes it over three times. The reason, the authors explain, is that the moment RefSpatial-Bench entered as the first pick, much of this benchmark's variance became reproducible. When it is finally taken in fifth place, the R² of 0.783 printed in the table is against all four benchmarks selected up to that point, and the authors read most of it as coming from its substitute, RefSpatial-Bench. However high the discrimination, once marginal information is shaved down to 0.217 the product collapses to 0.0576. This is the substitute relationship acting exactly as the utility intends, and it is the clearest illustration that informativeness cannot be judged benchmark by benchmark in isolation.

The four that emerge also cover complementary skills: precise localisation (RefSpatial-Bench), consistency of a spatial model across limited views (MindCube), spatial reasoning over video (VSI-Bench), and multi-image perceptual primitives (BLINK). The authors did not impose that complementarity and the utility has no way to encode it. The four also balance the two kinds of value the previous section distinguished. MindCube is the hub, carrying the information of the discarded columns on their behalf, while BLINK is the most isolated and supplies evidence no other benchmark gives.

4.2Change the Opening Pick and the Four Hold

Greedy selection commits to its first move, so the authors forced other benchmarks into the opening slot and let the procedure run from there. Force any core member in and the order changes but the same four come back. Seating Where2Place, RefSpatial-Bench's own substitute, first does not displace RefSpatial-Bench. Even though 75% of its variance is already reproducible by the third pick, it buys the fourth slot back with 0.343 discrimination times 0.248 remaining marginal information, or 0.085. BLINK is pushed to fifth instead.

The ordering is informative in itself. Start with a core member and almost nothing is lost; start with a core member's substitute and about four points go; start with a benchmark the free run rejects outright and seven to nine points go. MindCube and VSI-Bench made the first four in every run, and RefSpatial-Bench was not displaced even by its own substitute. The authors add one line here: physical specificity alone does not earn a slot. RoboSpatial is their example, and that thread is picked up again later.

4.3Cut the Count by Two Thirds and the Bill Falls a Quarter

If the suite goes from twelve benchmarks to four, does the evaluation cost fall to a third? The paper does not take up the question. We summed the item-count column of the paper's own table.

The count falls to a third but the item total falls only by a quarter, leaving 76% of the work. Almost all of that asymmetry comes from MindCube alone. Its 21,154 items are 53% of all twelve benchmarks' items and 70% of the core's. Take MindCube out and the other three come to 9,137 items, 23% of the total.

That bill, in absolute terms, is smaller than it sounds. Running the four core benchmarks fresh on all 51 models means about 1.54 million inferences as an ideal upper bound, or around 1.32 million once the matrix's actual density is accounted for. Pricing input and output tokens per item at public API rates as of August 2026 puts that in the hundreds of dollars with a commodity model and the low thousands even with a frontier one. The range is wide, and it counts API calls only, leaving out self-hosted GPU serving and the human hours that go into validating parsing rules. Still, the order of magnitude holds. It is also why this paper could fill 159 cells by running them itself. Now that the infrastructure for running evaluations exists, what is expensive in suite design is not the money but not knowing what you are measuring.

So far the story has been that the twelve share something. What Appendix B asks is what that something is. What the benchmarks hold in common might be physical capability, or it might just be the model's general ability. If it is the latter, this suite is a vision-language leaderboard with a physical label on it. The two cannot be told apart from inside the suite, because any axis estimated from the twelve simply inherits whatever they share. So the authors bring in evidence the suite does not contain.

What they bring in is nine general benchmarks that measure nothing spatial or 3D: MMMU and MMStar for multi-discipline multimodal reasoning, MMLU-Pro and GPQA-Diamond for textual knowledge and graduate-level reasoning, DocVQA and OCRBench for reading text in documents and images, Video-MME for general video understanding, and the text and vision Elo ratings from LM-Arena, which are set by human preference. Two criteria governed the choice: no intent to test physical or spatial understanding, and wide reporting on the same models.

Run a principal component analysis on the physical suite alone and the first component explains 55.2% of the variance, with all twelve benchmarks loading positively on it. The lowest loading is RealWorldQA's 0.48 and the highest CV-Bench's 0.88. The design point comes next. The nine general anchors are excluded from the PCA entirely. The first principal component is estimated from the twelve physical benchmarks only, and the resulting axis is compared against general performance afterwards. That is a guard against circularity: the anchors cannot influence the axis they will be compared to.

Kept apart that way and then compared, the alignment was nearly complete. The physical first component tracks the anchors' first component at a rank correlation of 0.952. Individual anchors give much the same picture, MMStar at 0.950 and Video-MME at 0.942. A score on a document-reading benchmark predicts a model's position on the physical suite's main axis about as well as the physical benchmarks predict each other.

That alone does not say which pairs hang together because of general ability and which for some other reason. So the authors regressed each of the twelve on this external axis, kept the residuals, and recomputed the correlations among the residuals. Removing that single axis halved the suite's internal structure. The mean pairwise correlation fell from 0.487 to 0.250, and the number of pairs above 0.5 dropped from 34 of 66 to six.

What the three surviving bonds say is clear enough. Both substitute pairs hang together not because they share general ability but because they measure the same specific ability. The merge experiment of section 3 was aimed at the right place. Bonds like ERQA with EmbSpatial, above 0.7 in raw scores, collapse once the conditioning is applied. That relationship existed not because the two exams measure the same thing but because a smart model does well on both.

In the fourth row the sign flips. VSI-Bench and BLINK correlate at 0.002 in raw scores, effectively unrelated, but drop to −0.363 in the residuals. Restrict to the 32 models with ample anchor observations and it reaches −0.425. Among models of comparable general ability, in other words, being strong at spatial reasoning over video tends to come with being weaker at multi-image perception. The two benchmarks sit side by side in the core from section 4. This negative correlation adds one more piece of support to the claim that the four are complementary.

The same phenomenon has already been reported in another field. Metabench, mentioned earlier, found a single common factor behind six LLM benchmarks correlating with the total score at 0.94. The domain differs but the structure is the same. A large share of what several exams share is not the field's specific ability but the model's general strength.

5.1The Remaining Four Redraw the Ranking

The authors rank the models again on the four core benchmarks. The method is a Bradley–Terry fit modelled on preference arenas, with benchmarks playing the role of judges. For every benchmark on which two models both have scores, one binary observation is created, discarding the size of the gap and recording only which side was higher. That yields 4,231 observations. The win-rate model is fitted by maximum likelihood and the strengths are then mapped onto the familiar Elo scale.

The top five are HY-Embodied-0.5 MoE-407B-A32B at 2251, Qwen3.5-397B-A17B at 2032, Qwen3-VL-235B-A22B-Instruct at 1828, Seed 2.0 at 1790 and Kimi K2.5 at 1750. Open-weight and closed API models are mixed together. The authors draw two readings from this.

One. Three of the top ten are models post-trained on embodied or spatial tasks: the two sizes of HY-Embodied-0.5 and RoboBrain-32B-2.0. If physical AI scores were driven purely by general capability, this ranking would reproduce a general-purpose leaderboard and domain post-training would provide no gain at all. The 0.952 alignment from the previous section and this observation have to be held together for the picture to make sense. Saying that roughly half of the shared signal is general ability is also saying that the other half is not.

Two. Scale works only within a recipe. Inside the Qwen-VL family the ranking is monotone in size, but the rule breaks across families. HY-Embodied-0.5 MoT-4B-A2B, with only two billion active parameters, places sixth ahead of far larger models. What determines position is not parameter count but what a model was trained on and how.

There is one more column to read alongside the ranking. The paper records in the same table how many of the four core benchmarks each model has a score on, and the top four models all have three rather than four. The first model with all four filled in is Kimi K2.5 in fifth place. Even a ranking built on a de-duplicated suite is computed over a matrix in which that suite is not fully populated. The empty-cell problem that opened section 1 follows the paper all the way to its last table.

A paper that audits should be ready to be audited. The authors set out their limitations in five items. Alongside each, we note how far it cuts back a claim made earlier in this report.

The last item weighs the most. All twelve benchmarks in this audit are exams in which a model solves a problem or outputs a coordinate, and nobody measures whether a robot arm actually picked the object up. The core of four that section 4 leaves behind is therefore not a selection of the signals that transfer to physical tasks; it is a selection of the least overlapping exams among those the field currently grades on. The problem of measuring the gap between simulator and reality, and the problem of what to measure that gap with, both remain outside this paper.

6.1Two More Inconsistencies We Found

Reading the primary source end to end, we found two places where the paper does not line up with itself. Neither overturns a conclusion, but both matter to anyone copying the numbers out.

First, Appendix C states that RoboSpatial carries the least general-capability signal of any benchmark in the suite and points to Appendix B as its basis. What Appendix B actually reports is a range of first-component loadings with RealWorldQA's 0.48 as the minimum and CV-Bench's 0.88 as the maximum. Appendix B contains no separate table of per-benchmark alignment with the general axis. The statement that RoboSpatial is the least general is therefore best attributed to the authors as their own assertion, and anyone repeating it should also note that the lowest loading belongs to RealWorldQA.

Second, the equation defining the utility is typeset as a(b) in the body, while the definition in the very next paragraph and the column headings of the selection path table all read g(b), the Gini coefficient. It is a simple typo, and citing it uniformly as g is enough. That is what this report has done.

6.2Who Wrote It, and What Kind of Document Is It?

The document discussed here is a v1 preprint posted to arXiv on 26 August 2026 and has not been peer reviewed. The authors are Zaruhi Navasardyan and Hrant Davtyan. The arXiv HTML body misspells the second author's surname as Davyan; the abstract page metadata gives Davtyan, which is correct.

Their affiliation, Metric AI Lab, is an applied AI research lab that advertises more than 100 enterprise projects, over eight years in operation and a publication at EACL 2026. Its current research agenda says something about where this paper comes from. The lab lists world models, robot failure detection and benchmark science side by side under physical AI, and its site carries the project corresponding to this preprint described as "one comparable matrix across physical AI benchmarks." This is the output of a research line aimed at a product, not a one-off analysis.

None of that amounts to a conflict of interest, but it is worth knowing as context. An organisation that has made benchmark science one of its own research pillars published a benchmark audit, and the audit's findings point in a direction that justifies the need for that research line. At the same time, the tools the paper uses are rank correlation, ridge regression, principal component analysis, forward selection and Bradley–Terry, all standard statistics, so the results can be followed and recomputed. The distance between whoever builds an evaluation metric and whoever has to trust it has already come up with agent benchmarks. That the method is public and recomputable is, for now, the most practical line of defence.

What Pebblous does to training data, this paper does to evaluation data. Finding substitutes by correlation, measuring reconstructability by regression, and keeping a minimal set by greedy selection is the same skeleton as a dataset deduplication pipeline. Only the target has moved, from the data a model eats to the data that measures the model.

Once an evaluation suite is treated as a dataset, the paper's findings translate into a familiar list of quality defects. There are duplicates (two substitute pairs), skewed weights (the load carried by pointing), inconsistent provenance (the 25 cells where published values conflicted and a median was taken), missing values (48 of 612 cells), and biased representativeness (half the shared variance is general vision-language ability, not physics). When that list turns up in training data, we write a quality report. There is no reason evaluation data should be different.

Anyone designing a robot data quality report runs into the same question every time: how many metrics should it contain? The answer so far has largely been the more the better. This paper shows that answer can be wrong. Add more correlated metrics and the total tilts in proportion to how many times an ability was measured, and that tilt is invisible to whoever reads the report. Translated into practice, it comes down to four steps.

The light data requirement is the virtue of this procedure. No item-level responses are needed, only aggregate scores per metric and enough overlap between metrics. An organisation already issuing quality reports has the ingredients in hand.

No quantitative standard for how many metrics to include exists anywhere yet. As far as we could establish, the robotics scopes accredited under KOLAS, Korea's laboratory accreditation scheme, are hardware performance tests, and no accreditation criterion addresses the number of training-data quality metrics. Annex IV of the EU AI Act likewise requires justification for the choice of performance metrics without specifying how many to use. The same gap appeared in our earlier report on the evidence chain behind robot field-trial data. The demand for justification is there; the procedure for deciding what counts as sufficient justification is not.

Into that gap this paper has placed one procedure. It is not an answer key. The authors themselves call their core one defensible choice rather than the unique optimum. What has changed is that the reason for picking a set of metrics can now be written down as numbers. From the point of view of anyone who diagnoses data, the next question follows naturally. The data a quality report diagnoses gets measured, but what measures the report itself?

Primary sources

• 1.Navasardyan, Z., & Davtyan, H. (2026). A Statistical Audit of Physical-AI Benchmark Redundancy. arXiv:2608.25940v1 [cs.RO], 2026-08-26. (The primary source for this report: sections 3 to 5 and Appendices A, B and C in full.)
• 2.Metric AI Lab. metric.am. Accessed 2026-08-28. (Research agenda; description of the MetaBench Physical project.)

Model reports we checked directly

• 3.Gemini Robotics Team, Google DeepMind (2025). Gemini Robotics 1.5: Pushing the Frontier of Generalist Robots. arXiv:2510.03342. (The 50/50 weighting rule for the embodied reasoning score; the Gemini Robotics-ER 1.5 score table.)
• 4.BAAI RoboBrain Team (2025). RoboBrain 2.0 Technical Report. arXiv:2507.02029. (The list of nine spatial reasoning benchmarks.)
• 5.Qwen Team (2025). Qwen3-VL Technical Report. arXiv:2511.21631. (The five benchmarks in the embodied bucket.)
• 6.Gemini Robotics Team (2025). Gemini Robotics: Bringing AI into the Physical World. arXiv:2503.20020. (Introducing source for ERQA.)

Academic — earlier audits

• 7.Kipnis, A., Voudouris, K., Schulze Buschoff, L. M., & Schulz, E. (2024). metabench — A Sparse Benchmark of Reasoning and Knowledge in Large Language Models. arXiv:2407.12844. (28,632 items, more than 5,000 models, total score reconstructed from under 3% of items, single common factor at r = 0.94.)
• 8.Burnell, R., Hao, H., Conway, A. R. A., & Hernández-Orallo, J. (2023). Revealing the Structure of Language Model Capabilities. arXiv:2306.10062. (29 LLMs, 27 tasks, three capability factors. The 82% of variance figure follows the Metric paper's citation and is not confirmed in the abstract.)
• 9.Liu, W., Xue, Q., Wang, H., Yin, X., Yang, B., & Gao, W. (2025). Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methods. arXiv:2511.15722. (The skew toward static relational items is confirmed in the text; the count of more than 45 follows the Metric paper's citation.)
• 10.Chiang, W.-L., et al. (2024). Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. arXiv:2403.04132. (Source of the Bradley–Terry preference arena method.)

Academic — introducing sources for the 12 benchmarks

• 11.Yang, J., et al. (2024). Thinking in Space (VSI-Bench). arXiv:2412.14171.
• 12.Du, M., et al. (2024). EmbSpatial-Bench. arXiv:2406.05756.
• 13.Zhou, E., et al. (2025). RoboRefer (RefSpatial-Bench). arXiv:2506.04308.
• 14.Yuan, W., et al. (2024). RoboPoint (Where2Place). arXiv:2406.10721.
• 15.Tong, S., et al. (2024). Cambrian-1 (CV-Bench). arXiv:2406.16860.
• 16.Ray, A., et al. (2024). SAT: Dynamic Spatial Aptitude Training. arXiv:2412.07755.
• 17.Song, C. H., et al. (2024). RoboSpatial. arXiv:2411.16537.
• 18.Jia, M., et al. (2025). OmniSpatial. arXiv:2506.03135.
• 19.Wang, Q., et al. (2025). MindCube: Spatial Mental Modeling from Limited Views. arXiv:2506.21458.
• 20.Fu, X., et al. (2024). BLINK: Multimodal Large Language Models Can See but Not Perceive. arXiv:2404.12390.
• 21.xAI (2024). Grok-1.5 Vision Preview (RealWorldQA).

Earlier Pebblous reports

• 22.Pebblous (2026). 'Perfect Score Without Solving Anything' — How 8 AI Agent Benchmarks Were Broken. 2026-04-12.
• 23.Pebblous (2026). The Test Was Already in the Training Data. 2026-06-20.
• 24.Pebblous (2026). Graded Against Motion Capture, No Physics Engine Was Accurate Everywhere. 2026-08-10.
• 25.Pebblous (2026). Korean Data-Quality Test Reports Count Only as Evidence in Europe. 2026-08-25.
• 26.Pebblous (2026). A $400M Valuation for the Company That Grades AI on Real Work. 2026-08-17.