Executive Summary

An API now exists that takes a single paragraph describing a task and returns tens of thousands of rows of training data. It is datasets.invent, from Adaption Labs, the company Sara Hooker and Sudip Roy founded. The sentence on the company blog reads: "You do not need to arrive with a seed corpus, predefined schema, or detailed labeling guide." Synthetic data generators have been around since the 2010s, but the step that dropped out this time is not generation. It is the structure definition that used to come before it.

Every performance figure is one the company measured itself. That was true of the September claim of a 35 percent average improvement over configurations its own researchers had set, and it is true of the arXiv technical report revised on October 18, whose five authors all work at Adaption Labs. That report is a step up from the September announcement in one way: it put five outside models through the same test. Even so, on the score for relevance to the task, the report places its own system short of the lead.

Sections 1 through 4 follow what Adaption Labs has published and the conditions those figures stand on. Section 5, which carries the question over to data quality practice, is this article's reading.

Key Figures

Sources: arXiv:2610.01674v2 and the Adaption Labs blog

All four numbers trace back to Adaption Labs itself. No outside party has run the same conditions again.

0.0%

Exact duplicates at 20,000 rows

The five model APIs compared in the same test ran from 6.5 to 18.8 percent

7.1–8.9

Score band for matching the task as given

All six generators fell inside this band, with Invent in the middle of them

48→64%

Win rate against the company's own researchers

An evaluation Adaption Labs ran itself across eight verticals at 5,000 to 100,000 rows

9.4–13.2%

Win-rate edge of checklist-filtered data

Measured against 10,000 rows sampled at random. What the checklist contains is not public

1

One Paragraph of Task Description Becomes a Dataset

Adaption Labs was founded by Sara Hooker and Sudip Roy at the end of 2025. Both came out of Cohere, and Hooker was at Google DeepMind before that. In February 2026 the company raised a $50 million seed round led by Emergence Capital, and it set its direction toward models that keep learning while in use rather than models that keep getting larger. AutoScientist, the fine-tuning automation product, arrived in May, and Invent a Dataset was attached in front of it in September.

Using it takes one call. You pass datasets.invent a piece of writing describing the data you want, up to 10,000 characters, along with a row count, and optionally a domain and subdomain code. The output comes in one of two shapes. An instruction_dataset pairs instructions with responses for supervised fine-tuning, and preference_pairs pairs a chosen answer with a rejected one for preference learning. Turn on the translation and localization option and the same data multiplies into other languages. The finished dataset downloads as JSON Lines, JSON, CSV or Parquet. There is nothing left to reshape, so the file that arrives can go straight into a training script.

The call runs asynchronously. The first response comes back as running, the caller polls datasets.get until succeeded appears, and then downloads. Switching on estimate returns a credit quote without generating anything, and when translation is applied the billing follows the multiplied row count. Hand the ID of the finished dataset to autoscientist.create and data generation runs into training setup in a single line.

Generation happens only on infrastructure Adaption Labs runs. The published API documentation describes no path for hosting it on your own servers. Both the process that makes the data and whatever that process filters out stay inside the company's perimeter.

2

The Automated Step Moved One Place Earlier

Stamping out synthetic data by machine is not new in itself. The tools built until now automated only the step of drawing examples against a frame: a person first fixed the shape of the table, wrote down what share each kind of task should take, and put in a labeling guide what would count as a correct answer. Those first three were human judgment, and that judgment survived as a document.

Those three are what Invent a Dataset takes over. By the company's account the feature interprets the goal, settles on its own what dataset structure will carry that goal, and produces training examples to match. The document people used to write shrinks into one call parameter. The diagram below puts the two flows on the same axis to show where a human hand lands in each.

Where does a human hand land? Until now A person writes the schema and the guide A generator draws the rows Dataset A person reviews it against that guide datasets.invent A person writes only a task description up to 10,000 characters It settles the structure itself and produces the examples Dataset ? Once the written-down step moves inside the product, the measure used in the last box leaves with it
▲ Pebblous original diagram. The order of operations Adaption Labs set out on its blog and in its API documentation, placed on the same axis as the conventional synthetic data workflow. | Source: Adaption Labs

The question mark at the right edge of the diagram is where this article starts. A labeling guide has two uses. It is the frame followed while the data is being made, and it is also the measure held up against the finished data to decide whether it is any good. Both uses live in one document, so when the first use disappears the second measure goes with it.

Making data and judging data have leaned on the same document all along. Automate the making and the judging does not get easier. It loses its grounds.

3

Who Scored the Average 35 Percent Improvement?

Two numbers came attached to the September announcement. AutoScientist beat human-set training configurations by 35 percent on average, and the win rate moved from 48 percent to 64 percent. The company put this on its blog without attaching a condition to it.

Who the human is here has to be read separately. The comparison is not against an arbitrary practitioner but against Adaption Labs' own researchers. The shape of it is that configurations set by in-house people who knew the model family, the vertical and the data scale were beaten by an automation product from the same company. The test covered eight verticals at dataset sizes between 5,000 and 100,000 rows, and the fine-tuning ran on an architecture provided by Together AI.

There is use in a setup like that. Taking your own researchers as the baseline is common during product development, and it is at least a stronger opponent than an arbitrary configuration. The trouble starts when the result travels outward. Which eight verticals they were, who judged wins and losses and by what standard, whether the same conditions reproduce on a second run, none of it appears in the public material. What a prospective buyer can check is the number.

So the question left over in September was a simple one. Had anyone outside the company confirmed these figures? Two months later, the company answered that question itself.

4

Two Months Later, a Technical Report Written by Five of Its Own

A document went up on arXiv on October 1 and picked up a revision on October 18. It is titled "Invent a Dataset: Measuring Dataset Generation Abilities with Zero Seed Data," and it calls itself a technical report rather than a paper. The authors are Shivalika Singh, Andrija Djurisic, Gbemileke Onilude, Sudip Roy and Sara Hooker, and all five are Adaption Labs people. Some secondhand summaries have carried the report as though it were an outside benchmark, but opening the author list settles that.

One thing is plainly better than the September announcement. This time the work was benchmarked against models from outside. The same prompts went to Anthropic's Claude Opus 5, Google's Gemini-3.1-pro, OpenAI's GPT-5.6 Sol, Z-ai's GLM-5.3 and DeepSeek-V4-Pro, each asked to build a dataset, and the outputs were compared. This is a different test from the September evaluation, and its range is eight task types at dataset sizes from 200 up to 20,000 samples. The abstract reports a 17 percent relative lead on quality and 19 percent on diversity.

The 19 percent on diversity is a number to read together with scale. The body of the report says the six generators were all on par at 200 samples. The gap widens as the data grows, and at 20,000 samples Invent recorded 0.268 against 0.196 for GLM-5.3, the strongest of the baselines. In relative terms that is a 37 percent difference.

The figure that stands out most is duplication. At 20,000 rows the five model APIs came back with between 6.5 and 18.8 percent of their items exactly repeated, while the report puts Invent at 0.0 percent. Anyone who has scaled a dataset up by hand knows what that number means. Run the same prompt through the same model a few thousand times and the same sentence starts coming back.

Share of exactly repeated items at 20,000 rows Invent API 0.0% Gemini-3.1-pro 6.5% Claude Opus 5 8.4% GLM-5.3 8.9% GPT-5.6 Sol 12.4% DeepSeek-V4-Pro 18.8% All six values come from a test Adaption Labs designed and scored itself
▲ Pebblous original diagram. The duplication rates the technical report published for six generators at 20,000 rows, placed on one axis. | Source: arXiv:2610.01674v2

Results from actually training on the data are in there too. Models post-trained on what Invent produced were picked first on 54 percent of the prompts with Llama-3.3-70B and 41 percent with Gemma-4-31B-it.

The same report writes down a weakness of its own. On the item measuring how well the data matches the task it was handed, all six generators landed between 7.1 and 8.9, the ones ahead in that band were Claude Opus 5 and GLM-5.3, and the Invent API sits in the middle rather than at the top. At 5,000 samples the values the report prints put Claude Opus 5 at 8.86 and GLM-5.3 at 8.66 above, with Invent at 7.47, effectively level with Gemini-3.1-pro at 7.51, and DeepSeek-V4-Pro at 7.44 and GPT-5.6 Sol at 7.11 below. The strength of this product lies in making data that spreads wide without repeating itself, and not in making the sample that fits the order most tightly.

The report lists its limits too. The system does not yet extend to tool-call traces or multi-step, agentic trajectories, and it is limited to the text modality at present. A question is left open in the same place. Whether data like this is best internalized in parametric weight updates is a valid question, the report says of itself.

September's question was whether anyone outside had confirmed the figures. As of November the closest thing to an answer is a report by the same five people at the same company, and even inside that report, relevance to the task is not a measure the system leads on.

5

Why Pebblous Is Watching This Product

Adaption Labs, introducing its data platform, writes that its agentic checklist method removes training examples that miss quality requirements. It posts a figure alongside: a model trained on 10,000 rows filtered that way reached relative win rates 9.4 to 13.2 percent higher than models trained on 10,000 randomly sampled examples (Llama3.3-70B, October 8, 2026).

Holding a list of quality requirements and striking out examples one at a time. Reviewing data against a labeling guide is, in other words, exactly that. The document sent out the front door has come back in the back, except that what came back is not a document a person can read. It is a procedure running inside the product. What the checklist says, who decided its items, and why a discarded example was discarded are not public.

Why this arrangement is uncomfortable from the data quality side is clear enough. Quality is not a property of the data but a judgment about its use, and a judgment needs a standard. Once the document holding that standard moves from our side to the supplier's side, what we can say about data we receive shrinks to "looks good" and "looks off." When a metric worsens there is no way to separate the data from the training setup from the task definition.

None of which leads to a conclusion that products like this should go unused. A 0.0 percent duplication rate is hard to reach by hand, and for a team that has been spending weeks writing guide documents, those weeks genuinely come back. But if you have decided not to write the guide, the guide's second job still has to happen somewhere. Three things can be settled in advance on the receiving side.

  • Do we hold, separately, the sentence that divides a right answer from a wrong one in our task? The description fed to the generator is an order form, not a standard for judging.
  • Can a person read 100 rows sampled at random from what arrives and grade each one pass or fail? If nobody can, the quality of that data has never been measured.
  • Can we measure again with the same standard after switching suppliers? If the criterion lives only inside one supplier's product, no comparison is possible at all.

All three are questions about writing a standard down, not about picking a model. It is also where Pebblous most often lands when the work is getting data into a usable state. The data has piled up, and review cannot start because nobody wrote down what counts as a good answer. The faster generation gets, the faster that gap shows.

Editor's Note

One more company has a name close enough to note here. The Adaption Labs in this article is the one Sara Hooker and Sudip Roy founded, and Adaptive ML, acquired by Datadog in June 2026, is an entirely different company. The spelling is Adaption, not Adaptation. It is an easy place for sources to get mixed when you search.

R

References

Academic Papers

Official Sources

Industry and Press