Executive Summary

Key Figures

The first two numbers come from the 680-column BIRD study, the last two from a clinical warehouse and a count of contradicting statements. Accuracy here means the share of content words that overlap with documentation written by other people, and the paper scores three facets separately: name expansion, description, and value domain. The 0.223 and 0.475 below are the name-expansion facet.

Source: arXiv:2608.07946, Sections 5.1 to 5.4

The paper takes aim first at an academic premise. Text-to-SQL benchmarks such as Spider and BIRD ship databases whose tables and columns already carry human-meaningful names and whose foreign keys are declared. Under that premise the remaining problem is translation, and contemporary models translate rather well. Real warehouses sit on the other side of that premise. Decades of accretion leave cryptic identifiers sitting beside clean marts, and the documentation is partial, stale, or absent. One column name from the paper is enough to make the point: pms_legacy.t_resv.amt_minor.

So the author makes the prior problem his experiment. The task is to recover what columns and their values mean from the data itself. The measurement is simple and unforgiving. Table and column identifiers across eleven BIRD databases are destroyed at maximum strength, leaving the system to look at names like t3.c0, and the recovery is then scored against the documentation the people who built those databases wrote. The answer key was written by someone else and the author did not touch it.

Three arms are compared. Statistical semantic-type detection that uses no language model, a language model shown the column and its values, and Rosetta, which puts a language model inside a verification harness. All three receive identical inputs and are scored on 680 paired columns. The score splits into three facets: name expansion, description sentence, and value domain.

Rosetta is not built to ask a language model what a column means. A deterministic profiler extracts structural evidence first, the model proposes semantics conditioned on that evidence, and every resulting fact carries provenance and a confidence capped by the class of evidence behind it. The profiler exhausts the cheap and certain evidence before it calls anything expensive and uncertain.

• Structural fingerprints. For each column it samples up to 5,000 values and computes length distribution, character-class ratios, separator and case shape, shared prefixes and suffixes, per-character entropy, numeric summaries, Benford first-digit distribution, monotonicity, and cardinality. No language model is involved at this stage.
• Pattern library. The pattern library holds 26 specifications once variants are counted, and it runs on top of the fingerprints. A value that passes a checksum lifts confidence above 0.95, a regex match alone stops at 0.85, and a fingerprint heuristic alone stops at 0.70. The checksum validators implement Luhn, ISBN-10 and 13, EAN-13, UPC-A, IBAN mod-97, and mod-11.
• Rule-verified namer. Only on genuinely cryptic columns, where no pattern fires, does the language model appear. It proposes a pattern name, a semantic type, and a validating regular expression as one package, and the proposal is accepted only if the regex compiles and fully matches at least 90% of the sampled values. The model has no authority to declare a match. It can only offer a hypothesis.

The evidence collected this way is translated into an eight-tier authority ladder. Semantics inferred by the model from a profile alone stop at a confidence cap of 0.50, a match to a known shape reaches 0.60, checksum backing reaches 0.85, and explicit human confirmation reaches 1.00. The cap holds regardless of how confident the model claims to be. The interesting design choice is that the checksum tier sits deliberately below the tier for a documentation citation. A checksum proves what kind of value this is, not what the column is for.

The diagram below puts that order on one page. Value samples enter from the left, evidence and confidence caps are fixed before the language model appears, and speaking and abstaining part company at the bottom. Only the commit gate drawn with a dashed border was missing from the system as originally shipped. Why the author had to add it later is what the next section explains.

Refusing to leave an abstention as a blank is part of the same design. When the evidence is too thin to speak, the system writes a standardized abstention record, so that whatever consumes the catalog cannot mistake an empty cell for a claim. The reported cost of the run is roughly 105M tokens and 55,000 calls for 281 dollars, which works out to about 0.0027 dollars per recovered column.

The table that places the three arms side by side is where the paper starts. Statistical detection with no language model answers on every column and scores close to nothing, and both language-model arms clear that baseline by a wide margin. Between the model alone and Rosetta, though, the result does not separate cleanly.

That the score ultimately counts word overlap is a weakness the author names himself. So he validates the metric rather than asserting it. A blinded judge rating 180 items separately tracked the token metric at Spearman 0.642, and a second rater from a different model family agreed with that judge at 0.737. The suspicion that longer writing simply scores higher was checked as well. Rosetta writes 17.4 content tokens per claimed prediction against 19.6 for the model alone, so it says less and recovers more.

The author does not stop there and goes looking for what the gap actually is. Restricting the comparison to the 283 columns where both arms speak makes the question identical, so prose is all that differs. Under that condition Rosetta did not write better metadata than the model alone, and on two of the three facets it came out slightly behind. The blinded judge agreed on the direction.

The check that follows is the center of the paper. Recomputing the deterministic evidence for all 680 columns without any language model shows 342 columns carrying structural evidence and 338 carrying none. On columns with evidence, Rosetta's coverage is 0.550, and without it 0.293, a gap of 0.257 whose confidence interval excludes zero. Ask instead whether evidence lifted the quality of the prose and no effect can be found. The model alone, which never sees the evidence at all, barely moves, going from 0.980 to 0.902. This is where the author's sentence comes from: the deterministic layer is a competence detector, not a competence amplifier.

The backbone swap draws the boundary around that claim. Holding the scorer, the prompts, and the profiler fixed and changing only the model to Claude Sonnet 4.6, the finding about prose reproduced almost exactly, but the selection behavior did not transfer. Coverage rose from 0.422 to 0.823 and the gap that had tracked evidence scattered. The cause is structural. The shipped system requested abstention in a prompt rather than enforcing it in code, so a model that likes to guess simply guessed.

The last study closes that hole. Putting a commit gate that checks the grounding tier in code on every path that writes a record drove coverage on no-evidence columns to 0.000 on all three backbones. The cost is reported alongside. Between 99 and 295 claims were withdrawn depending on the backbone, and some of them were correct. Recall on what remained did not degrade, and on the backbone that guessed most, the withdrawn claims scored lower than the surviving ones, which suggests the gate removed mostly ungrounded guesses.

What abstention actually protects shows up most sharply in clinical data. The author reconstructs 75 columns of an i2b2 clinical warehouse with the decoder withheld, then scores the code column of the fact table against a key decoded externally. From values alone the system works out that the column carries both diagnostic ICD-9 codes and drug NDC prefixes. Its first description did not carry the meaning of individual codes, and the author notes that describing them was never what the description task asked for, then asks properly and measures again. Requested to produce a decode table, the same machinery resolved 95.5% of 134 real ICD-9 codes from values alone, and abstained on all 44 NDC drug codes while the prefix was attached.

Where the line of abstention falls is what matters. A published compositional code system like ICD-9 earns an attempt at decoding from the structure of its values, and a key assigned arbitrarily by a registrar like NDC does not. The system decides, per value, which code systems it is in a position to decode. Foreign-key recovery shows the same posture. With names left in place and only the foreign-key declarations removed, it proposed 49 of the 103 ground-truth edges, and the precision on those 49 was 1.000. With identifiers destroyed as well it proposed none at all. Not a single false join was produced. Raise the unit of recovery from columns to tables and the posture holds. Across 67 tables with every identifier stripped it named three in four correctly, and on the 55 tables where it chose to speak, it identified what the table holds in 50 of them.

What the reconstructed catalog does at query time was measured too. On a Spider study with the schema fully obscured, a naive translator degrades from 0.92 to 0.42 execution accuracy, which means more than half of the answers come back quietly wrong. Under the same conditions Rosetta's gate holds 86% accuracy over 59% coverage, and the questions it declines are the genuinely hard ones.

The most operational number is the count of contradicting statements. Per 100 columns examined, the model alone emits 15.0 statements that contradict the reference documentation and Rosetta emits 8.3, because it declines 55% of them. The author attaches the number that cuts the other way as well. In the sample, restricted to what each arm chose to claim, Rosetta contradicted the reference slightly more often.

Rather than leave that unfavorable number sitting in a sample, the author re-judged it across the whole corpus. Applying the same rubric to all 574 shared columns and 776 claims across the eleven databases, the conditional contradiction rate came in at 9.6% for Rosetta against 8.0% for the model alone, with a confidence interval straddling zero that leaves the two arms statistically indistinguishable. The per-100-column total, on the other hand, split at 4.01 against 7.49, and that interval excludes zero. The apparent disadvantage in the sample did not survive power, and the difference in total volume did. Abstention does not make the system more careful when it speaks. It makes it emit measurably fewer falsehoods overall.

The honesty of the paper shows where it contradicts its own design. On the strength of three analyses, the author writes that the authority ladder he built is not the mechanism producing this result. What did the work was execution grounding and deterministic thresholds, and what the ladder leaves behind is auditability rather than accuracy. He also records that across all 680 BIRD columns and 75 clinical columns not one checksum-backed match occurred, so the upper tiers of the ladder remain untested.

There is more in the unfavorable column. On toy-simple Spider schemas, one cheap baseline matches the gate. A selector trained on general-purpose signals ranked and calibrated better than the author's own authority score, which added nothing to the decision. His single caveat is that the selector feeds on execution labels, and an undocumented warehouse has none. The paper also records that he found a bug in the scoring harness of an earlier Spider study himself, fixed it, and re-ran at larger scale before making the stronger claim. The design hypothesis that recovering columns first would make table recovery easier was not supported either.

Read from the data quality side, this is a question about contracts. Automated documentation that fills 100% of a catalog wins on the completeness metric, but leaves no way to tell during an audit which entries were grounded. A catalog that fills 42% and leaves the rest as abstention records can answer, for every filled cell, why it says what it says. The readiness argument Pebblous made in data comes before the model and the false-alarm problem raised by PrismaDV, which reads downstream code to write data validation rules, meet on the same axis here. Writing more rules and deciding where to stop are different jobs.

Three things are worth checking today.

• In the catalog you run now, can you tell an automatically generated description from one a person wrote? If you cannot, both lose credibility together in an audit.
• Are empty cells and abstained cells recorded differently? An empty cell is work not yet done and an abstention is a judgment that evidence is lacking, and the judgment vanishes if whatever consumes the catalog cannot tell them apart.
• Is the instruction not to answer without knowing written in a prompt or in code? If it is a safeguard that disappears the day you change models, it is worth counting now.

• 1.Helwig, M. (2026). "Metadata Reconstruction from Values Alone: Recovering Column Semantics in Undocumented Warehouses." arXiv:2608.07946 [cs.DB, cs.AI].
• 2.Helwig, M. (2026). "Deterministic by Default: Design Principles for Language Models inside Verification Harnesses." Zenodo, DOI: 10.5281/zenodo.21892964. Companion methodology paper.
• 3.Helwig, M. "mike-m6online/Rosetta" GitHub repository (paper artifact, MIT License).