Executive Summary

First, let's be precise about that 27%. The number is not a "share of organizations that own an enterprise semantic layer." It is an integration-perception measure — organizations that self-reported their data, processes, and applications as well-connected in the HBR × Hyland survey. It reflects self-assessment, not technical architecture. So the 27% doesn't mean a semantic layer got built. What actually made that minority feel well-connected is the real question — and the answer is not a search index; it's a layer of meaning, relationships, and standards.

Vector indexes and semantic layers are often lumped together, but they do different jobs. A vector index is a retrieval layer: it turns text fragments into embeddings and finds nearby fragments by similarity. It is good at "sentences similar to this one," but it knows nothing about relationships like "after Company A acquired Company B, which subsidiary does clause 3 of that contract apply to?" A semantic layer is a meaning layer: it merges entities scattered across documents (companies, people, clauses) into one, standardizes the relationships and schema among them, and makes "what connects to what, and how" consumable by a machine.

That difference shows up in performance. Ask GPT-4 forty-three enterprise questions with only the raw SQL schema, and accuracy sits at 16.7%. Give it a knowledge-graph representation of the same questions and it climbs past 54.2% — more than triple (data.world benchmark). Evidence runs the other way too. On an easy academic text-to-SQL benchmark (the original Spider), top models clear 86%; on Spider 2.0, which reflects real enterprise schemas, models of the same class land at just 31% execution accuracy. The gap between "easy benchmark" and "real schema" is exactly the space a semantic layer fills.

Vendor whitepapers claim even larger effects for this layer. dbt reports that adopting a semantic layer lifted metric-definition consistency to 83%; some BI tools claim dashboard errors dropped by two-thirds. The direction is right, but these figures are vendor-published with no independent verification, so we treat them as background rather than evidence. What matters isn't the size of the number but the level it operates on: retrieving well and standardizing meaning are different jobs, and without the latter, the performance of the former hits a ceiling.

For a single unstructured document to become a machine-consumable semantic layer, it passes through several stages. It reads the layout (parsing), splits a long document into meaningful units (chunking), pulls out fields (extraction), merges references to the same thing into one (entity resolution), maps them onto a shared schema (ontology alignment), and attaches when and where each fact came from (provenance). Each stage breaks in its own way, and a failure at one stage propagates quietly into the next.

2.1Parsing: the illusion of a solved stage

Layout parsing looks like the most mature stage. On clean, formatted academic documents, table-structure recognition (TEDS) reaches 82.5. But on the same benchmark (OmniDocBench), layout detection for non-academic documents — multi-column, scanned, handwritten — collapses to mAP 48.7. Change the domain and it's starker: train a table-recognition model on one domain (PubTables-1M) and apply it to another (FinTabNet), and exact-match accuracy falls from 65% to 42%. A parser's benchmark score does not guarantee its performance on your documents.

The bigger illusion is confusing field accuracy with document completeness. On ExtractBench (2,076 pages, 12,867 fields), field-level accuracy averages 72.9% when valid JSON is produced. Yet the share of documents whose schema passes cleanly from start to finish is just 4.6%. Even if each field is mostly right, one wrong field out of ten marks that document's structuring as a failure. The moment you look at "72.9% field accuracy" and conclude parsing works, the reality is that only one document in twenty comes out whole.

2.2Chunking: invisible, yet it governs everything

How you split a long document seems trivial, but it moves the performance of the whole pipeline. In one knowledge-graph construction study, switching only the chunking strategy to hierarchical raised extraction F1 from 70.2% to 87.8%. Same parser, same extraction model, same rest of the pipeline — only the way of cutting changed. Tables flattened into text, cross-references in a contract severed at a chunk boundary, a speaker and timestamp split apart in meeting notes: most of these losses happen at this stage.

2.3Extraction: the split between easy fields and hard ones

Extraction performance splits sharply by field. In a personal-information extraction study, GPT-4 pulled email addresses at 100% and phone numbers at 98%. But for fields that require reasoning across context — affiliation, work history — accuracy dropped off a cliff. Values written plainly on the surface are easy; values that require weaving together clues from across a document are hard. The problem is that exactly the information a semantic layer needs — who signed this contract, on behalf of which organization — sits on the hard side. This is where the parsing problem turns into a resolution problem.

The two points where the pipeline breaks most often, and most quietly, are entity resolution and provenance. Neither shows up in parser benchmarks. And both are exactly where a model begins to invent the context it was never given: it can reason all it wants, but it cannot conjure context the data never handed it.

3.1Entity resolution: not a tool, a design

Entity resolution is the work of merging references to the same thing — one company, one person, one contract clause — when they appear across documents under different spellings, aliases, and cross-references. You have to know that "Pebblous Inc.," "Pebblous," and "㈜페블러스" are the same party before a relationship graph can hold together. On clean, curated benchmarks this task clears F1 above 99%, but on real unstructured documents riddled with aliases and spelling variants, it collapses.

The decisive fact is that this is not a "buy a better parser" problem. On the same Abt-Buy benchmark, embeddings plus approximate nearest-neighbor search alone reach only 35% precision (F1 44.5%). Add an LLM as a judge — leaving the data untouched and changing only the method — and precision jumps to 95.4% (F1 66.3%), at four cents per case. Nothing was swapped out; the pipeline was simply designed differently, and precision nearly tripled. That this merging is genuinely hard also shows up in where the time goes. In one knowledge-graph build, 22.5 of 38 total hours (about 59%) went not to reading characters off the page but to merging entities and relations into one. In other words, most of the effort lands not on reading text but on resolving who and what.

3.2Temporality and provenance: retrieval succeeds, context gets invented

The second hurdle is attaching when, and from which version, a fact came. Contracts get amended, policies get versioned, meeting notes capture a decision at a specific moment. When that time axis is lost, retrieval still pulls back "plausibly relevant" fragments — but whether they belong to the valid version, no one knows. On a benchmark of version-controlled documents, standard RAG accuracy sits at 58% (VersionRAG). In regulated domains, attaching a temporal and version schema has been reported to improve on standard vector RAG by 70% — a measure of how much headroom was there.

There's a more counterintuitive fact. We usually blame "the model invents context it doesn't have" on retrieval failing to find the right passage. But recent reclassification work shows a different picture. A type where retrieval succeeds — the correct evidence is in context — and the generation step ignores it anyway, so-called evidence override, was several times more common than retrieval failure in specific domains (28.4% in medical, 42.3% in HotpotQA). The weaker the temporal and provenance signal, the worse this gets. When evidence conflicts across versions or the timing is ambiguous, the model picks the most plausible option or fills the gap itself.

So far we've watched benchmarks show how badly each stage breaks. But the question worth asking is this: can you measure, ahead of time, how broken your organization's semantic layer is? Here the pipeline's real gap surfaces. There is still no agreed industry-standard metric that directly measures the intrinsic quality of a knowledge graph or semantic layer — entity-duplication rate, relation precision, temporal consistency.

So teams work around it. They infer layer quality indirectly from downstream accuracy (did the Q&A come out right?), or they hand-annotate samples to check. Neither scales. Downstream metrics don't tell you which layer was the problem, and manual annotation buckles as document counts grow. This gap is confirmed academically too: the fact that papers proposing new metrics to quantify knowledge-graph quality (KGCQual and others) only appeared in 2026 is itself evidence that the standard was absent. So is the finding that automatically extracted knowledge graphs have an average degree of ~1 — nodes barely connected to one another, a fragmented graph. It points to the same void.

What happens when you can't measure quality is visible in commercial products. Even a commercial legal RAG marketed as "hallucination-free" hallucinated 17–33% of the time in a preregistered independent audit (Stanford RegLab: Lexis+ AI 17%, Westlaw 33%). Same RAG architecture family — the main driver of the gap was not the retrieval method but the quality of the curated data. There's a sharper contrast still. Retrieval grounded in a domain knowledge base hallucinated at 6%, versus 35% for general web search (JMIR Cancer). The resolution and governance of what you retrieve from is what splits the outcome.

This is the actual break-even that separates a pilot from production. In a demo, a handful of well-chosen documents make it run. In production, hundreds of thousands of documents arrive in wildly different states of resolution and provenance. If you can't measure that state, you also can't know where to invest to fix what gets you past break-even. Buying one more parser will never make this layer visible.

For a team that can't get its pilot to production, "buy one more parser" is usually the wrong answer. Parsing already mostly works; the bottleneck is downstream of it. Everything above resolves into three practical priorities.

These three share a common thread: each is a problem of measurement — of gauging what was fixed and by how much. If you can't quantify which layer of the pipeline broke, and how badly, you can't decide which of the three priorities to fund first.

This diagnostic layer is exactly where Pebblous DataClinic stands. Its role is to quantify what broke in the "unstructured → semantic layer" conversion: whether entities merged into one, whether temporality stayed consistent, whether the schema passed cleanly. AI-Ready Data is the name for the output state of this pipeline, and making that state measurable is the work of this layer. Measuring a layer that has no industry standard yet — that is the space between pilot and production that no one has quantified.

Models keep getting better at reasoning. But the habit of inventing context the data never gave them will not disappear with a better model. It disappears only when that context is filled in from the data side — through resolution, through temporality, through provenance. It's not parsing. It's resolution.

References

The benchmark and survey figures in this report draw on the sources below. Vendor-published figures lack independent verification and are cited as background only.

Academic

• 1.Ouyang et al., "OmniDocBench: Benchmarking Diverse PDF Document Parsing," CVPR 2025. Table-structure recognition TEDS 82.5 / layout detection mAP 48.7.
• 2.Magesh, Surani, Dahl, Suzgun, Manning, Ho, "Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools," Journal of Empirical Legal Studies 2025 / arXiv:2405.20362. Legal RAG hallucination 17–33%.
• 3."ExtractBench: Field-level vs Document-level Extraction," arXiv:2602.12247. Field 72.9% vs. whole-document clean pass 4.6%.
• 4."VersionRAG: Version-aware Retrieval for Evolving Documents," arXiv:2510.08109. Standard RAG without temporality at 58% accuracy.
• 5."Diagnosing Retrieval vs. Utilization Bottlenecks in LLM Agent Memory," arXiv:2603.02473. Evidence-override reclassification.
• 6."KGCQual: Quality Metrics for Knowledge Graph Construction," arXiv:2607.10212. Academic grounding for the semantic-layer quality-metric gap.
• 7.Li et al., "Ditto: Deep Entity Matching with Pre-Trained Language Models," VLDB 2020 / arXiv:2004.00584. Entity-matching F1 96.5% (clean benchmark).
• 8.Smock et al., "PubTables-1M / GriTS," arXiv:2110.00061. Table-structure recognition, TATR domain transfer 65% → 42%.
• 9.Huang et al., "LayoutLMv3," 2022. FUNSD F1 92.08 / CORD F1 97.46.
• 10.Barnett et al., "Seven Failure Points When Engineering a RAG System," arXiv:2401.05856. RAG failure typology (no quantitative failure rate in the original — cited for the typology only).
• 11."Personal Information Extraction," arXiv:2408.07291. GPT-4 email 100% / phone 98%, affiliation and work history collapse.
• 12.Towards AI, "GPT-4o-mini as an Entity Resolution Judge: 95% Precision for $0.04," 2026. Abt-Buy 35% → 95.4%.

Policy & Statistics (surveys — intro hook)

• 13.Harvard Business Review Analytic Services × Hyland, "AI Readiness Survey," Dec 2025 (n=325). Structured 65% / unstructured 39% / connected 27% / 81%.
• 14.Cloudera × HBR Analytic Services, 2026 (n=230+). Fully ready 7% / difficulty preparing 73%.
• 15.Teradata × Wakefield Research, 2026 (n=1,000, 6 countries). Development stalled 68% / pilot failure 40%+ / metadata gaps 43%.

Vendor & Other (no independent verification — background citation)

• 16.data.world knowledge-graph benchmark, 2026. SQL 16.7% → 54.2%.
• 17.Spider 2.0 leaderboard. Enterprise-schema execution accuracy 31% (vs. ~86% on original Spider).
• 18.JMIR Cancer study. Domain knowledge base vs. web search hallucination 6% vs. 35%.
• 19.ChemRxiv, "KG Construction from Materials Science Literature," Jan 2026. Chunking-strategy change alone F1 70.2% → 87.8% (domain-specific).