Executive Summary

Key figures

Source: Fu et al., OmniTable: A Unified Wide-Table System for Petabyte-Scale LLM Data Curation and Exploration, arXiv:2609.11148 (September 10, 2026), sections 5 and 6

The introduction lists three pain points where industrial LLM data preparation collapses into a maze. The first is data silos. Corpora from dozens of sources sit scattered across hundreds of physical tables, so finding what lives where across datasets is already hard. The second is costly feature engineering. Adding a single feature means coordinating tasks by hand across every dataset it touches, and the authors attach an in-house case to that point, quotation marks and all. For one feature, an engineer had to "drag and drop 106 tables onto the task canvas."

The third is broken lineage. UDF logic is spread across codebases with no centralized version control, so the same feature ends up defined differently in different places. Lineage and feature definitions are not systematically captured, so teams cannot reliably trace how each data and feature iteration affected the training runs and the model quality that followed. The paper carries this item into its design requirements, writing that stale lineage brings definition drift and irreproducible results.

The first two are problems that scale created. They appeared because sources multiplied and data grew, so a faster engine and a bigger cluster pay some of the debt back. The third does not work that way. Double the cluster and there is still nothing written down anywhere about which filter trimmed which corpus by how much. Two problems of scale sit beside one problem of quality, and the one this article follows is the third.

Curation you cannot retrace turns into trouble after training finishes. When model quality comes out worse than last time, the question to ask is which filter changed when, and how much of which corpus that filter trimmed. Without the record, two options remain. Rerun everything, or let it go. Either way the same question comes back next time.

When the lineage record survives, what has to be redone narrows too. In a rule-revision relabeling case we covered in August, tracing lineage to pick out only the items a revision touched meant reevaluating 14.7 percent of the historical data, and accuracy landed within 0.5 points of relabeling the whole collection. Without the record there is no way to point at that 14.7 percent, which leaves the two options above.

OmniTable forces the record by raising features to first-class metadata. Computation logic, input and output dependencies, execution environment preference and version history all register atomically in the Catalog. When a computation finishes, the system writes the feature ID, version, timestamp and execution engine into lineage and updates the column status. Every row carries one global primary key, so ordering across sources, processing stages and derived features, single-record lookups, and lineage audits all turn on the same key.

Reduced to one line, the structure is logical unification and physical separation. To the user, every source, every processing stage and every derived feature shows up as a column in a single table. Adding rows is ingestion, and adding columns is a feature declaration. Underneath, storage is split across several physical tables, and the Catalog holds the map between logical and physical along with the feature dependencies. When a user writes SQL, the Catalog turns it into a physical execution plan.

Production numbers make the separation visible. Across four domains, web, code, PDF and supervised fine-tuning, 35 petabytes and 305 billion records sit in four logical tables, with sixteen physical tables beneath them. The web table alone holds 25 petabytes and 300 billion records, with more than 800 logical columns of which more than 200 are registered features. Its storage spreads over six physical tables, and column splitting moves low-frequency columns into auxiliary tables according to access frequency.

Keeping the physical side in repair is not optional. The authors look back on having deferred background compaction and splitting as a nice-to-have, and within three months accumulated small files had degraded query latency by 3 to 5 times and column growth had hit the engine limit. Their conclusion to that section runs as one sentence: without continuous physical layout evolution, a PB-scale continuously-written system becomes unusable within weeks.

The authors also confirmed the same conclusion by experiment. A separate run disabled background compaction and row and column splitting. Below 1 petabyte its throughput is comparable, but at 25 petabytes it drops to 5 terabytes an hour, and in the experiment that grew logical columns from 200 to 2,500, past about 1,200 columns some queries failed outright and the table had to be redesigned by hand. The version that keeps repairing the layout held its throughput across both ranges, and its response time stayed within 1.53 times while the column count grew more than twelvefold. When the authors say repair is not optional, this gap is what they point at.

The comparison ran on one real scenario, preparing data for supervised fine-tuning. Instruction data comes in from 8 sources, 12 features covering quality scoring, safety compliance and domain classification get computed, and a selected high-quality subset goes out for ablation experiments. Of the 12 features, 9 are CPU UDFs and 3 are GPU inference. Input batches and filter thresholds were fixed identically on both sides.

Where the time went in the legacy workflow is broken out by stage. About 2 days to locate 8 source tables, write conversion scripts and copy data. About 9.5 days to backfill features. At that stage engineers placed roughly 96 nodes, 12 feature nodes across 8 tables, on the orchestration canvas and kept cleaning up by hand the failures that came from wrong parameters and bad records. Then about 2.5 days to tie 8 result tables together with multi-way joins and export. That adds to 14 days, with about 45 manual steps, 24 pipelines and 35 physical tables involved.

The same work on OmniTable ends in 0.5 days of ingestion, 1.7 days of feature backfill and 0.3 days of filtered export. Ingestion is eight commands. Backfill means registering 12 feature definitions and running a single plan, after which dependency resolution, engine routing, parameter tuning and failure isolation follow on their own. Export is one SQL query against the logical table. Manual steps come to 12, commands to 10, and the target is one logical wide table.

How the 7.8 days came off the backfill is broken out too. Operator fusion took about 1.5 days out of I/O overhead, UDF-level failure isolation about 2 to 3 days out of investigate-and-retry cycles, and adaptive tuning about 1 to 2 days out of parameter-related failures. Those savings partly overlap, totaling 5 to 6.5 days, and the rest came from scheduling improvements. Measured on its own, fusion cuts CPU hours by 55.9 percent, takes scans from eight down to one, and brings end-to-end time from 38 hours to 14.

One experiment pulls out failure handling by itself. The job backfilled a fastText feature that picks out mathematical content over a 500 gigabyte batch of about 600 million records, and about 31,000 of those records blew past memory because their text ran extremely long or their encoding was broken. As a share, that is 0.005 percent. In the legacy workflow that 0.005 percent killed the entire terabyte-scale job, and three rounds of finding the cause, removing records and resubmitting took about 52 hours, 18 of them human effort. Isolating at the record level finishes the same job in one pass in about 6.2 hours with zero human intervention, and the 31,247 offending records land in an error table. In a structure where 0.005 percent costs 52 hours, the size of a failure and its cost are not proportional.

The place people actually wait is usually the lookup. With the global ID index on, a single-record lookup has a P50 latency of 8.3 seconds and a P99 of 14.7. Without the index, scanning every physical table that might hold the ID puts the same two figures at 184 seconds and 612 seconds or more. Aggregation queries offload to OLAP and come back 94 to 154 times faster, all of them inside 10 seconds. Throughput on filtered exports holds at 18 to 23 terabytes an hour from 1 terabyte up to 25 petabytes. The single thread of spotting one odd record, checking the distribution and pulling a subset stops being a batch submission and becomes an interactive query.

Two things have to travel with these numbers. The legacy workflow being compared is not an outside system but a baseline the team reconstructed from historical task records and operation logs of their own pre-OmniTable workflow. And all 19 authors work at Ant Group, which makes this a before-and-after account of the company's own system written by the company. Being picked as VLDB's best industry paper does not change that structure.

From here on this is reading the paper rather than reporting it. The first number that comes up in any discussion of data readiness is usually volume. How many terabytes did we collect? How many items did we label? This paper dug into something else: the time between a person making one judgment and being able to run it again. At 14 days, a proposal to change a filter threshold becomes a two-week decision. At 2.5 days, the same proposal becomes an experiment. Somewhere between the two sits the point where trying it and rolling it back turns into an available option.

So the questions to hand back to our own organizations do not ask about volume. They ask how many days a change takes, and what the record keeps.

• How many days does it take to change one feature or one filter? If the answer comes in weeks, the team is already choosing not to change it.
• Is there a queryable record of which filter trimmed which corpus by how much? If it lives only in people's memory and in documents, that is not far from not having it.
• What share of engineering hours goes into finding the cause of failures? At Ant Group, before the isolation machinery went in, that item ran 30 to 40 percent.

The speed came from spending more resources. Record-level failure isolation costs 3 to 5 percent more in execution, and materializing frequently read columns in advance costs 8 to 15 percent more in storage. The compaction work that merges small files takes a standing 5 to 8 percent of cluster resources. That is roughly the price of staying in a state you can retrace. That the paper prints the price at all counts in its favor.

The last lesson is about people. Algorithm engineers lived and breathed SQL, the authors write, and their instinct was to spin up yet another table and write yet another query. Instead of asking them to drop that habit first, the team made the logical wide table speak plain SQL, let a feature template library turn registration into a matter of minutes, and kept lineage as queryable tables. The section closes on a sentence: a system that demands a workflow change before delivering value will be worked around, not adopted. An organization trying to raise its data readiness may need to ask about that order before it asks about tools.

Thank you for reading this far. The full text is at arXiv:2609.11148, and the proceedings version is PVLDB volume 19, issue 12, pages 4276 to 4289. Every figure in this article was checked against the abstract and sections 5 and 6. If you have ever measured how many days your team's curation cycle takes, we would like to hear which unit you measured it in.

Pebblous Data Communication Team
September 12, 2026

• 1.Fu, Y. et al. (2026). "OmniTable: A Unified Wide-Table System for Petabyte-Scale LLM Data Curation and Exploration." arXiv:2609.11148. VLDB 2026 Best Industry Paper, PVLDB 19(12), 4276-4289.