Executive Summary
In scientific AI, the preprocessing that shapes raw data into something a model can train on almost always happens inside one researcher's notebook. The script never makes it into the paper, and the next person rebuilds the same work from scratch. REDI, from the U.S. Oak Ridge National Laboratory, standardizes that preprocessing into five stages — ingest, preprocess, transform, structure, output — and builds provenance instrumentation into the pipeline so that how the data changed at each stage is recorded automatically. This article looks at what "data that carries its own path" makes possible, rather than merely "clean data."
The instrumentation yields an unexpected byproduct too. When every stage's runtime was measured, what dominated pipeline cost turned out to be not the algorithms but file I/O. Simply switching the storage format from NPZ to Zarr raised parallel-access throughput about threefold. The habit of keeping a record drew a map for optimization. That said, REDI does not automate deeply domain-specific transformations either.
REDI exposes those five stages not as scripts a person rewrites each time, but as five operating modes an agent invokes through a standard interface. That is exactly the point where preprocessing moves from one-off manual labor to shared infrastructure that keeps a lineage.
~3×
Zarr parallel throughput
vs NPZ. The first format lever the instrumentation revealed
86%
Ingest share of fusion runtime
The bottleneck was file I/O, not computation
η ≥ 0.95
Parallel efficiency, 20–800 workers
Near-ideal scaling at supercomputer scale
r = 1.000
Transform validation correlation
Protein data. Provenance guarantees reproducibility numerically
Why preprocessing was always trapped in private scripts
Scientific AI papers fill up with model architectures and performance metrics, yet the work that actually consumes the most time upstream is the preprocessing that turns data into a form a model can train on. The problem is that this preprocessing usually ran only inside one researcher's local setup. In what order the data was downloaded, how the grids were aligned, what was normalized, which format it was saved in: none of that makes it into the paper. Only the final numbers remain; the path to those numbers disappears. The next person picks up the same data and rewrites the script from the beginning.
The paper "Automated Data Readiness for Scientific AI," from researchers affiliated with the U.S. Oak Ridge National Laboratory, takes aim at exactly this attrition. Its diagnosis is simple: until now, no single framework had tied together automated transformation, readiness assessment, provenance tracking, and agent integration. Separate tools existed for each, but there was no pipeline that let the full journey, from raw source to trainable state, flow through while being instrumented reproducibly.
What tries to fill that gap is REDI. It is a five-stage pipeline that automatically converts massive scientific datasets for AI training, beginning with ingest and passing through preprocess, transform, and structure before ending at output. Unpacked, the flow is: download the data (ingest), align the grids (preprocess), extract features and normalize them (transform), lay out the structure as tensors or graphs (structure), and save into a training-ready format (output). Each stage is encapsulated as a single PipelineStep class, and every time that step is initialized, provenance instrumentation automatically comes along with it. That is where this article's question arises: what, exactly, does that instrumentation leave behind, and what does it make possible?
There is a companion piece that approached the same paper from a different angle: how the yardstick differs across domains. It is The AI Readiness of Scientific Data, Measured Differently by Field. Where that article looked at how the criteria of completeness, consistency, and fitness diverge from one domain to the next, this one looks at how the pipeline records the path it traveled and hands it over to an agent. Same paper, a different vein.
What each preprocessing stage leaves behind
REDI's provenance instrumentation is handled by a framework called Flowcept. When each PipelineStep in the pipeline is initialized, a FlowceptTask is created, and that task observes what happens before and after the step runs. It captures both the state of the data going in and the state coming out, recording how statistics like the mean and standard deviation shifted and how long the transformation took. The key point is that a person does not plant the logging by hand. Build a stage, and the instrumentation follows on its own.
The records that accumulate this way are automatically compiled into a human-readable document called PROVENANCE_CARD.md. It is an audit-trail card that lets you retrace which stages, in which order, brought the data to its current state. It is the move from a world where only the raw numbers survived and the process vanished, to a world where the process itself remains as an artifact. Even if someone reopens the same dataset six months later, they can read off the card where and with what coefficient the normalization was applied.
What the instrumentation left behind was not only reproducibility. Laid side by side, the runtimes of all stages showed that what dominated pipeline cost was not computation like normalization or encoding, but the I/O of reading and writing files. Fusion simulation data spent 86% of its processing time in the ingest stage, reading the data in. So the paper argues that, before touching any algorithm, the choice of storage format is the first lever of optimization. In practice, Zarr, a chunked array format, delivered about three times the parallel-access throughput of NPZ, which stores individual arrays whole. The habit of keeping a record drew, as a byproduct, a map of where the fixes should go.
Five modes an agent calls directly
Keeping a record, on its own, still does not escape the private script. The decisive step is that REDI exposes this pipeline not as code a person rewrites each time, but as skills an agent can call. In the paper's own words, it "exposes prebuilt, domain-aware transformation logic as agent-invocable skills." Concretely, there are five operating modes, each a unit an AI agent can invoke programmatically.
| Mode | What it does |
|---|---|
| redi run | Runs the full pipeline end to end, from ingest to output |
| redi discover | Proposes a preprocessing plan — a draft a human can review |
| redi inspect | Observes a pipeline that is already running |
| redi assess | Quantitatively evaluates the data's readiness |
| redi validate | Validates against specified reference values |
Why these five modes matter becomes clear by picking apart just one, discover. In the paper's example, discover takes 32.94GB of CMIP6 data from the TaiESM1 climate model and returns a plan recommending xESMF regridding to align the grids. At the same time, it attaches a warning that shuffling the time axis would cause data leakage. A domain pitfall a person had to remember every time is flagged, at the planning stage, by a standard mode the agent calls. That running (run), planning (discover), and assessment (assess) all sit on the same interface is what decisively separates this from a world of rewriting scripts by hand. A companion tool, SetGo, adds FAIR-compliance validation and catalog publishing on top.
3.1How it differs from tools that only detect
It is not that tools for preprocessing and quality did not exist. It is that each held only one piece. Below is where REDI sits, laid alongside the tools the paper cites in its related work.
- • Great Expectations detects data-quality problems, but does not automatically transform the data to fix them.
- • AIDRIN scores readiness across fifteen dimensions, but stops there: it grades, it does not transform.
- • Nextflow and Snakemake weave workflows well, but are weak at handling scientific data formats like NetCDF, HDF5, and ADIOS2.
- • MLflow tracks model hyperparameters and metrics, but does not record the transformation itself: from what state the data changed to which.
REDI's place is the gap between these: it ties transformation, readiness assessment, provenance tracking, and agent integration into a single pipeline. It puts onto one flow the pieces that tools which only detect, only score, or only keep model parameters each let slip.
When a one-off script becomes shared infrastructure
Put the pieces together and the outline of the shift comes into focus. The preprocessing is instrumented and left as provenance (reproduce), that provenance is compiled into a card a person reads (audit), and the whole pipeline is exposed as standard modes an agent calls (reuse). When those three hold together, the know-how of preparing data leaves the individual's notebook and becomes an asset that is shared and verified like a paper or code. Not an attrition rebuilt each time, but infrastructure with a continuing lineage.
This shift did not collapse at supercomputer scale either. Even scaling from 20 to 800 workers, REDI held parallel efficiency above 0.95, and its transformation accuracy reached a correlation coefficient of 1.000 on protein data. It showed by measurement that growing the scale does not scramble provenance and reproducibility. There is, of course, a limit worth leaving honestly on the table. REDI itself states that it cannot yet automatically handle deeply domain-specific transformation sequences. The domain knowledge of what to transform and in what order remains a human's part, and what automation does well is to record and reproduce, without omission, the process that knowledge leaves behind.
This concern meets Pebblous's way of working with data head-on. The direction DataClinic has pursued in diagnosing datasets was also to leave not only the result of cleaning, but where and how that cleaning happened, as an auditable provenance. REDI's provenance instrumentation reads as one case of implementing that direction on top of national-supercomputer-grade scientific data. The adjacent question of who grades the data was covered in an earlier piece, Now AI Grades AI Readiness. Where that article asked who does the grading, this one asked how the preparation process itself leaves a trace. Before you hold the scorecard, the path the data traveled has to already be there. That is the thread tying the two together.
References
- 1.Wilkinson, S. R. et al. (2026). "Automated Data Readiness for Scientific AI." arXiv:2607.02771.
- 2.Brewer, W. et al. (2026). "Data Readiness Pipeline Patterns for Scientific AI at Scale." AI Magazine, Wiley. DOI 10.1002/aaai.70056.
- 3.Hiniduma, K. et al. (2025). "Data Readiness for AI: A 360-Degree Survey." ACM Computing Surveys.