Executive Summary

Key figures

The first three numbers are what the system gains by extracting values before questions arrive. The last one is the size of the sample that decides whether the schema doing that extraction is sound.

Source: arXiv:2608.13384, Table I, §VI-B and §VI-D

The example question the paper opens with is an ordinary one. What is the average age of the active players on NBA teams with more than five championships? To a person that is a single sentence. To a system it is a conditional filter, an aggregation, and a join across documents, all at once. On questions like this, the authors argue, existing approaches break in two places.

The first is retrieval. Turning documents and questions into vectors and returning the nearest K measures whether the topic is close, not whether a condition holds. Ask for a birth year earlier than 1985 and you get documents that are merely on topic with the wrong year, while documents that satisfy the condition without making it prominent drop out. What drops out here is not recovered later. Real systems therefore push the retrieved documents back through an LLM to check the conditions one by one, and the bill grows in proportion to how many documents came back.

The second is computation. Averages, comparisons, and joins are operations that assume values already sit in columns. What vector retrieval returns is a bag of text fragments with no order and no schema, so there is nowhere to attach those operations. Knowledge graphs do pre-extract entities and relations, but they behave like path traversal engines, so averaging over a filtered set or connecting two sets with no edge between them still ends with a subgraph being handed to an LLM.

What the paper identifies underneath both failures is not accuracy but cost structure. Neither approach separates the cost of extracting attributes from the cost of running a query. Ask the same corpus ten questions and the attributes are extracted ten times; ask a hundred and they are extracted a hundred times. As long as cost scales with the number of queries, large-scale analytical work is hard to put on top of it. That is where this paper starts.

AnnoIndex answers by moving the work earlier. Instead of searching documents once a question lands, it sweeps the corpus beforehand, decides which fields to extract, and fills them in. The paper calls the resulting table an annotation index. A condition such as a birth year earlier than 1985 becomes a single numeric comparison on that index, and the LLM is never called.

The module that builds the schema is SchemaLoop, and its structure is borrowed straight from relational databases. The dataset layer splits the corpus into logical databases, the table layer groups documents into entity types, and the document layer decides the fine-grained fields to extract from each document. Each layer is designed to narrow the search space for the layer below, so by the time the document-level filters apply, the candidate set has already shrunk considerably.

Who sets the schema differs by layer. The dataset layer is defined by people in advance. The table and document layers are drafted by GPT-4o, called once per document group. Humans draw the outer boundary of the corpus and the model fills in the fields inside it. Neither fully automatic nor fully manual, and that division is what defines the character of the system.

Once values are extracted, the question of representation follows. Dates are converted to a single format such as ISO 8601, numbers are given consistent units and decimal notation, and categorical strings are trimmed and normalized. The target formats are derived from the value shapes that recur in the corpus, and the normalization rules themselves cycle through the same feedback loop as the schema. When candidate schemas contain fields that mean the same thing, synonym matching and embedding similarity collapse them into one. The normalization and deduplication a data team used to do by hand have moved inside the index build.

The paper also counts what this preparation costs. The dataset layer calls no model at all, and the table and document layers call GPT-4o exactly once per document group. Verification, which actually extracts values to test a schema, runs on a small model, with a full-size model reserved for the final document-level check when necessary. The index that comes out of this typically shrinks the document scanning scope by two to three orders of magnitude, and on a corpus of 1,600 documents the entire schema induction and index build cost the equivalent of one or two online queries. The claim that preparing before the question is cheaper rests on that arithmetic.

An auto-generated schema is not adopted as is. Every time SchemaLoop produces a candidate, a verification phase follows. Five documents are drawn at random from the corresponding document group, Mistral-7B extracts values for every field, and two metrics score the result. One is the extraction success rate, the share of attribute-document pairs that came back non-empty. The other is filtering efficiency, which measures how evenly documents spread across the distinct value combinations of the schema. The thresholds are 0.6 and 0.3.

If either metric falls short, the failure is handed back to the model, which merges, splits, or redefines fields. Iteration continues until both metrics stabilize, capped at five rounds, and the authors report that the document layer usually converges within three to five.

Those two numbers, 0.6 and 0.3, are applied identically to all three datasets. The same bar governs legal provisions averaging more than 6,000 tokens per document and templated web pages that repeat the same layout. There is no experiment in the paper showing what happens when the thresholds are tuned to the character of a corpus.

The extraction success rate asks whether a value was filled in. Filtering efficiency asks whether documents divide evenly. Neither asks whether the value is right. Comparison against ground truth appears only in the experimental evaluation, where the authors had GPT-4o draft candidate attributes and then had eight graduate students verify and correct every attribute-value pair to produce a gold standard. That is an asset built to score performance, not a mechanism the running system consults.

So if five documents yield well-populated values and a clean split, a field that is extracting the wrong thing still passes. The schema that passes is applied to the entire corpus, and every subsequent query draws its answers from it. Errors in an auto-generated schema are written into the index and quietly reused.

Once the index stands, a query compiles into a SQL-like execution plan. Conditions expressible over indexed fields are filtered first, with no model call involved. Conditions absent from the index are handled by an operator called EXTRACT, which attaches tools in ascending order of cost. Whatever a regular expression can settle goes to a regular expression, what it cannot goes to the small model Mistral-7B, and only what remains after that goes to GPT-4o.

The example plan the paper walks through end to end looks for baseball players born before 1985 who are still active. The plan first keeps only baseball player documents using a table-level field, then applies a less-than-1985 condition on the birth year field. Up to here everything is a comparison over the index and the model is never called. Whether a player is still active is not in the index, so it falls to EXTRACT at the end, by which point only a handful of documents remain and GPT-4o has correspondingly little to read. The condition vector search could not enforce becomes one line in a WHERE clause.

A query over the legal corpus that hunts for provisions conflicting with superior law shows the ordering well. A pre-extracted conflict-type field narrows 1,600 documents to 120, a lightweight model extracting environmental keywords narrows that to 35, and GPT-4o only has to pull the conflict explanations out of those 35. On that query class F1 is 0.79, against 0.61 for QUEST with its LLM calls restricted. Let QUEST use them freely and accuracy rises, but the token count passes 200K.

The executor has a few more tricks. The order in which conditions are checked is decided per document, and the moment one fails the rest are skipped and the document is dropped. Joins first collect the values satisfying a condition on one side and rewrite them into an IN filter, cutting calls on the other side. Each query carries a ceiling on how many LLM calls it may issue, so a plan that goes wrong cannot run up an unbounded bill. On queries spanning three tables the plan is laid out as a directed acyclic graph that extracts only the attributes each step needs, and this is exactly where it parts ways with QUEST and its fixed left-deep join order.

Values produced by EXTRACT are not thrown away. They are first stored as virtual fields in an extension area, and once a field has been referenced more than ten times it is promoted to a formal schema field and fed back to SchemaLoop. In the authors' framing, it moves extraction cost out of a per-query charge and into a one-time charge levied when the value is first needed. Note that the promotion criterion is again not whether the value is right but how often it was called. The mechanism rests on the assumption that a frequently used field is a good field.

The payoff shows up in repeated querying. Running five consecutive batches of 100 queries, the number of extractions that trigger an LLM call fell from 48 to 13 in economical mode and from 72 to 15 in performance mode. Summed across the five batches that comes to 128 and 177 calls respectively, against roughly 300 expected with no reuse at all. The more the same questions recur, the lower the marginal cost.

The three corpora tested differ in character: 1,600 legal provisions averaging more than 6,000 tokens each, 219 Wikipedia pages spanning ten domains, and 1,050 templated web pages. Each dataset got 500 queries, stratified from simple filters up to three-way joins and multi-step reasoning. An answer counts as correct only when every attribute value in the tuple matches exactly.

The widest gap in the table sits in the legal column. A vector database with RAG stays at 0.34 while AnnoIndex in performance mode reaches 0.81. On a corpus where a single document runs past 6,000 tokens and the conditions are buried inside the provisions, the difference between fetching topically similar documents and extracting values into columns beforehand is on full display. Isolating three-way join queries, AnnoIndex scores 0.86 and QUEST 0.74, while feeding whole documents to GPT-4o collapses to 0.35. That complex-query evaluation was run on a separate set: 30 two-way joins, 20 three-way joins and progressive reasoning queries, and 25 legal conflict analyses.

The way the baseline ranking flips between datasets is another clue for reading the table. Lotus, which feeds whole documents to the model, hits 0.95 on web pages and falls to 0.46 on legal provisions. Push long, dense provisions in all at once and the model gets buried in them. Web pages repeat the same template, so values sit in nearly fixed positions and the top three methods all clear 0.94. To see the difference between methods, then, look at the legal column.

The paper also runs ablations, stripping components out one at a time. Swap in a schema a domain expert wrote by hand after reading 20 documents and Wikipedia F1 drops from 0.87 to 0.72 while tokens climb from 26.8K to 51.8K, because manual schemas miss important attributes or carry useless ones. Remove the verification step instead and accuracy holds steady while tokens rise to 38.7K, because the redundant fields left in an unpruned schema trigger extractions nobody needed. Automatic induction and its verification are attached to both accuracy and cost.

The third variant keeps the schema and removes only the structured query engine, replacing it with pure vector retrieval, and F1 falls to 0.64. Tokens in that variant come to 32.5K, not far off performance mode. Even with an index full of pre-extracted values, accuracy collapses without something above it to apply filters, join tables, and execute in order, and the loss is not a side effect of saving tokens. A schema-first design is not complete with the index alone. It holds only when paired with an executor that can read it.

Tokens per query, as the paper defines them, sum offline indexing tokens and online query tokens and divide by the number of queries. Each dataset carries 500 queries and the build costs the equivalent of one or two of them, so the more queries there are the closer this figure slides to the online marginal cost. Economical mode, attaching the cheapest extractor first, runs at 15.7K tokens per query for F1 0.83. Put the other way around, the arithmetic does not hold for a corpus you only query a handful of times.

Some conditions belong next to the numbers. All three datasets were chosen by the authors and the gold standard was built by the authors. The top schema layer was defined by people in advance, so this is not the score of an index assembled fully automatically. Above all, there is still no mechanism for surfacing the fact that a schema is wrong. Build an index wrong once and every query above it tilts the same way. Set against the expensive approach of extracting per query, which at least judges anew each time, what this design relocates is not only cost but also how long an error survives.

So rather than reading this paper as a tool to bolt onto a pipeline today, it is more useful as a prompt for teams working with document assets to check three things.

• What share of the questions you ask your documents re-extracts the same fields every time? Recurring fields are candidates to move out of the query and into an index.
• Can you say who decided each of those fields? Where boundaries set by people and fields filled by a model are mixed together, it is worth recording where human decisions stop.
• Is there any channel that would tell you an auto-generated schema is wrong? A check that only asks whether extraction happened will pass values that are wrong.

Earlier pieces along the same axis include Databricks Genie on moving document evidence into governed data, Rosetta on recovering column meaning from values alone, and the conditions for AI-Ready Data. The full paper is available at arXiv:2608.13384.

• 1.Lin, T., Luo, Y., Tang, N. (2026). "Structure then Query: Enabling Precise Analytical Queries over Unstructured Documents." arXiv:2608.13384.
• 2.Sun, Z. et al. (2025). "QUEST: Query Optimization in Unstructured Document Analysis." Proc. VLDB Endowment, 18(11).
• 3.Lin, T., Luo, Y., Tang, N. (2026). "AnnoRetrieve: Efficient Structured Retrieval for Unstructured Document Analysis." arXiv:2604.02690.
• 4.Edge, D. et al. (2024). "From Local to Global: A Graph RAG Approach to Query-Focused Summarization." arXiv:2404.16130.