Executive Summary

Key Figures

Source: Podstawski, Auditing a Model-Hub Benchmark, arXiv:2611.03727 (2026)

Hugging Face hosts hundreds of thousands of models, datasets and Spaces, and users press like on the ones they want to keep. In December 2024 a team at Nanjing University crawled the hub into HuggingKG, a graph of 2.6 million nodes and 6.2 million edges, and built three tasks on top of it: recommend the next resource to a user, classify which task a model is built for, and trace which base model a derivative came from. That bundle is HuggingBench, published at SIGIR 2025.

This audit concentrates on the first task. It splits 1.06 million likes from 38,624 users across 25,080 items into training, validation and test, then compares eleven recommenders by Recall@10 — how much of what a user actually pressed like on lands in the top ten suggestions. The items break down into 16,200 models, 3,634 datasets and 5,246 Spaces. Three of the eleven methods read more than the like history; they also read a graph of how items relate to one another, so the benchmark ships four separate external graphs.

The four differ in kind. One is a homogeneous graph joining items to items. One lifts the neighborhood a single step out from each item in the full hub graph, another takes two steps, and the fourth groups resources by who published them. The best score the original paper reported was 0.1578, from KGCL on the homogeneous graph. One method, KGIN, scored close to zero in every configuration, and the paper printed that number as a result.

The audit reproduced those numbers before touching anything. Running all eleven methods on the published configurations and grids unchanged, LightGCN and DCCF matched all eight reported values to four decimal places, and the rest landed within 0.0010 Recall@10, apart from one KGCL cell on the 2-hop graph. KGIN's 0.0004 came back too. The reported numbers are an accurate record of what the published code computes on the published data. What stays open is the question of what that code was measuring.

A graph-aware recommender carries the item ID straight out of the like file and into the graph. To explain item 17166 it pulls whatever facts entity 17166 holds. Same number, same thing — except that promise is written down in none of the files. It is a long-standing implementation convention in this field, resting on the assumption that items take the first block of entity IDs.

The homogeneous graph keeps the promise. The other three break it. On the 1-hop and 2-hop graphs, not one of the 25,080 items carries the ID it has in the like file; on the publisher graph, four do. Each of those two graphs holds 767,927 endpoints that attach a fact to an item, and none of them reach the item they belong to. No item receives even one correct fact. On the 1-hop graph 117,260 endpoints land on a different item that does not describe them, 99,312 do the same on the 2-hop graph, and the rest fall on entities that are not items at all. Among what falls away sit 278,358 endpoints recording lineage between models — fine-tunes, quantizations, adapters, merges. Those are precisely the facts that explain why the hub holds so many near-identical copies of one model.

And yet each file was sound. All 34 single-file checks the author ran pass: keys unique, values in range, training and test kept apart. The item name list and the entity name list are each flawless on their own terms. The error appears only at the moment the two lists are matched by name. The author notes that adding that one comparison to the validation code catches it in seconds.

How it happened is legible in the shape of the release. The same like records appear in six separate file bundles, and every bundle numbers them its own way. Pick any two of the graph bundles and at least 25,077 of the 25,080 items carry different IDs, with no mapping between bundles shipped. Matched by name, the splits in all four bundles turn out identical. The same data went out with fresh numbering per graph, and the reading side trusted that equal numbers meant equal things.

One model makes the consequence concrete. SmolLM2-1.7B-Instruct is item 17166 in the like file. In the 1-hop graph it is entity 366541, carrying 153 facts, one of which is a quantization relation to SmolVLM-Instruct. The recommender never received that fact. It followed the number to entity 17166 and found a Space named dpbos/postvalidation with a single fact attached.

The repair itself is plain. Every item is renumbered so that the ID it uses in the like file is the ID it uses in the graph, and the remaining entities move behind them. Not a single fact was added or removed. The author published the corrected files, the mappings, the validators and the run logs on Zenodo in full. Rerunning the same methods there across three seeds produced the table below.

Test-split Recall@10, averaged over three seeds, 100-epoch budget. Source: Table V of the paper, in full. The homogeneous graph was not part of the repair; for reference it gives KGCL 0.1585, KGRec 0.0987 and KGIN 0.1351. KGIN figures use the original authors' implementation. The change column is computed from the rounded values printed here, so its last digits can differ from the 4-to-73-percent range the paper reports from unrounded numbers.

No cell went down. The corrected version won on every graph, every seed and every method, with gains between 4 and 73 percent — smallest for KGCL, largest for KGRec. Of the 27 pairwise comparisons between configurations, 10 reversed across all three seeds. In the original paper KGIN led KGRec on the released 2-hop graph; on the corrected graph it trails. Which method is better turned out to hinge on one data repair.

Whether the repair delivered real facts or merely fact-shaped filler got its own test. On the corrected 1-hop graph, the author rewired item-to-entity pairs at random while holding item type and degree fixed. Correct alignment beat every random rewiring. But those random rewirings still beat the released version in all nine runs. Plausible type and degree alone buy part of the gain; collecting the rest takes the right facts on the right item.

KGIN's near-zero score is not explained by the join error alone. Looking into it, the author found that the framework's port of KGIN does not train its embeddings on the GPU, and that "this defect is unrelated to the data." KGIN was therefore rerun with the original authors' implementation. That swap on its own yields 0.1169 on the released 1-hop graph, and the repair takes it to 0.1445. The reported 0.0004 sat at the level of a random ranking, and the author's reproduction returned the same 0.0004. A score at random level, the paper argues, belongs in the error column rather than the findings column.

Why nobody noticed becomes clear once items are split by popularity. On the 1-hop graph with KGRec trained for 500 epochs, items that collected between one and four likes during training recovered 0.004 of their test likes in the top ten under the misaligned graph. After the repair, 0.020 — five times as much. The 20-to-99-like band rose from 0.011 to 0.040. Items above 100 likes went from 0.148 to 0.155. On the 2-hop graph even popular items gained in absolute terms, 0.133 to 0.157, but in relative terms the rarest band again improved far more. Mean Recall is a number the popular items carry, so what happened out in the rare bands barely registered in the aggregate.

The rare side is no small corner either. Items with fewer than 100 training likes receive 54 percent of the test likes. What the average hid was not the margin but half the data.

The audit names a second defect in the same breath. A budget that cuts training off at 100 epochs was reached before the early-stopping criterion in seven or eight of 29 configurations, depending on the seed. Of the 23 truncated configuration-seed pairs, 21 improved when the budget was raised to 500 epochs, by as much as 31 percent. The two defects interact. Models trained on a misaligned graph converge more slowly, so a longer budget shrinks the measured benefit of the repair. Part of that 73 percent is undertraining. For KGRec on the 1-hop graph, the gap between the two versions narrows from an average of 0.0613 to 0.0374. Even so, in all six cases — two model depths times three seeds — the corrected version leads under both budgets.

HuggingBench publishes card text for every item: model and dataset card bodies, tags, the publishing organization and name, library, language and description, up to 3,000 characters. Not one of the eleven methods in the original paper reads it. The author aimed at that gap with the simplest recommender available, a linear model called EASE that finishes in one matrix computation with no iterative training.

The results read best in order. Plain EASE, with neither text nor graph, scored 0.1684 and already passed the 0.1578 best reported in the original paper. A variant with per-item adaptive normalization reached 0.1746, and adding the item graph on top gave 0.1776. Swapping the graph for card text in a term-frequency representation did better still at 0.1846, and the best configuration, carrying both, reached 0.1860. That is 18 percent above the previous best.

The cost side is the more disconcerting part. Plain EASE trains in 253 seconds on eight CPU cores, and the best configuration, text and graph included, finishes in 358. KGRec, by comparison, needs more than a day on a GPU to sweep its grid on the 2-hop graph.

Against expectation, the same card text vectorized with two recent embedding models lost to the term-frequency representation in all four comparisons, by 0.0046 to 0.0077, with confidence intervals that exclude zero. The author points to how formulaic hub text is. Two models published by one organization have a median term-frequency similarity of 0.40, while two items drawn at random sit at 0.03. Where names and tags recur as near-identical strings, matching characters outright works better than an embedding that generalizes meaning.

The author also pulled the fields apart one by one. Item type plus organization and name alone reaches 0.1850, and the description field by itself lifts plain EASE by 0.0134. Masking URLs, repository identifiers and an item's own organization and name, which rewrites 18,251 of the 19,191 descriptions, still leaves 0.0125. The same signal is written into several fields at once.

And the place where text contributes most overlaps with the place where the repair mattered most in section 3. For items with one to four likes, plain EASE scored 0.0057, and the text-plus-graph configuration 0.0339. For items above 100 likes, 0.1976 slipped a little to 0.1949. Users who had pressed like rarely show the same shape. Splitting by item type tells the same story: models, which have cards, gained most at 0.012, and Spaces, which have no cards at all, gained least at 0.005. The text is doing work. A broken join and an unread description are separate defects, yet both were cutting the benchmark's scores in the same place and in the same direction.

The other two tasks gave text different roles. In lineage tracing, the best figure in the original paper was MRR 0.559 for a structure-only method, and the author's reproduction gave 0.549; ranking the model names mentioned in a derivative's card body, in order of appearance, with no training whatsoever, reached 0.542. For 54.3 percent of the test triples the answer was written by name inside the card. Putting the card-derived candidates first and ranking the rest with the existing method climbs to 0.831.

Task classification, by contrast, did not yield to card reading. Returning the task names that appear in the card gives micro-F1 0.404, barely above the 0.384 you get by guessing the single most common task, and far short of the reported 0.885. Something else turned up instead. A model's task tag and a dataset's task field spell out a correct label for 99.99 percent of the test items. No classifier code was published, so there is no confirming that those fields were read, but the benchmark should have declared the channel. The audit also records that the count of test items disagrees between the release, at 27,345, and the paper, at 28,682.

The paper gives its closing section to recommendations for the people who build benchmarks and model hubs. There are six, and set side by side they ask for one thing: whoever bundles data for release should write down, outside the files, what the bundle promises.

• Validate joins, not files. The error lived between files rather than inside them, and the validator should ship with the data.
• Prefer explicit links to implicit convention. Frameworks that require a separate item-to-entity link file make this failure structurally impossible.
• Declare the provenance of every field. Record when and how it was collected, and drop or recompute on the training split any field that can carry held-out information.
• Let the protocol check itself. Report the epoch at which each configuration peaked, and flag runs that stopped at the budget ceiling.
• Include a baseline that actually uses what you published. A benchmark that releases side information should report at least one simple model that reads it.
• Separate provenance tasks from the text that states the provenance. For tasks like lineage and classification, where the answer sits in the card, mask the text or first report how much of the task the text alone solves.

The third recommendation has its origin inside this benchmark too. The like and download counters stored in the graph are crawl-time values, so they can include likes that went into the test split. In 25,057 of the 25,080 items the counter is at least as large as the likes summed across all three splits, and its rank correlation with test likes is 0.80 against 0.71 for training likes. The measured effect came out small, but a small effect and a sound design are two different claims. One thing in the same section came back clean: across the relation types of all four graphs, no user-item like appears as a relation, so test likes never leaked through the graph.

Evidence that this is not one benchmark's accident also sits in the paper. The file format that allowed the misalignment descends from KGAT, and the author ran other datasets released in that format through the same check. Releases that number items first passed. But where another team renumbered the same dataset and released it again, reading it with the earlier release's item list left only 0.7 and 2.4 percent of training pairs on their own items. Each release is sound on its own. The misalignment appears the moment files from different releases are mixed.

From here on, this is outside the paper. The structure the audit exposes is not peculiar to benchmarks. Anywhere training data is assembled by joining tables from different systems on numeric IDs, the same thing can happen. Each table passes its schema check, the loading pipeline turns green, and the model scores come out tolerable. The misalignment is visible only when two tables are matched on a name a person can read. Fewer teams than you would guess have that comparison in their automated checks.

So three questions are worth asking.

• What guarantees the join keys in our pipeline — the schema, or a convention written down nowhere?
• Which band does the average metric hide? Unless rare segments get measured on their own, integrity errors stay behind the mean.
• Which fields have we collected and never once fed to a model? This paper has already demonstrated once that such a field can be the strongest signal in the pile.

Thanks for reading this far. The paper is at arXiv:2611.03727, and the corrected files and validators are on Zenodo. If your team already runs a join comparison as a standing data check, we would like to hear how you wired it up.

• 1.Podstawski, M. (2026). "Auditing a Model-Hub Benchmark: Data Quality, Training Budgets, and Side Information in HuggingBench." arXiv:2611.03727 [cs.LG].
• 2.Chen, Q., Huang, K., Zhou, X., Luo, W., Cui, Y., & Cheng, G. (2025). "Benchmarking Recommendation, Classification, and Tracing Based on Hugging Face Knowledge Graph." SIGIR 2025.
• 3.Podstawski, M. (2026). "Corrected HuggingBench knowledge graphs, mappings, validators, and logs." Zenodo.