Executive Summary

Key Figures

Sources: Donaghy et al., arXiv:2609.17380 (2026-09-15) · audit harness repository

The report sorts today's model releases into three tiers. Closed models reachable only through an API, open weight models that publish the final weights alone, and open source models that hand over the data and the recipe as well. Then it proposes a fourth tier, in which every operation on every data sample during training can be checked again by anyone on their own machine with bitwise certainty. The authors call that tier fully auditable.

A new tier needs two words separated first. Determinism means a computation returns the same result every time it runs on the same machine in the same environment. That is the property most deep learning frameworks offer under the name of a deterministic mode. Reproducibility asks for more. The result has to come back identical to the last bit across different hardware, a processor and a GPU included. The first can hold while the second fails, and that is exactly where open source models have been sitting.

The abstract says why the distinction bites in practice. Without the ability to produce the same result on another machine, a user cannot verify that the checkpoint they downloaded was actually produced by the declared recipe. That empty space leaves room for undisclosed data, injected biases, or backdoors. Existing techniques such as proof-of-learning or proof-of-training-data give probabilistic guarantees only, and a backdoor can be planted with a handful of poisoned examples, which slips straight through the net of sampled inspection.

The starting point is that the associative law you learned at school does not hold inside a computer. Floating point stores only a subset of the real numbers exactly and rounds the rest to the nearest value it can hold. Add three numbers and the rounding lands in a different place depending on whether you add the first two or the last two first, so the final bit changes. A GPU that splits a long sum across thousands of lanes and a processor that accumulates end to end are adding in different orders to begin with.

The report collects the places where bit-for-bit equality breaks into five items, and pairs each one with the way RepOps holds it.

The fourth row, subnormals, shows the character of hardware particularly well. Apple GPUs discard values very close to zero by default, while processors and NVIDIA GPUs keep them. The same value survives on one side and vanishes on the other, so the training trajectory splits. The fix is to match the least permissive backend. Subnormals get thrown away deliberately, even on hardware that would not otherwise need to. This is the general shape of every reproducibility decision, the authors write. To be reproducible across a set of machines, you adopt the behavior of the stingiest one on purpose.

The random number row also runs against intuition. With the same seed pinned, a processor, an NVIDIA GPU and an Apple GPU still draw three different sequences. Inside a single vendor the story repeats: on an NVIDIA GPU the generator spreads its work across thousands of parallel lanes, so which lane draws which number depends on how the computation was parallelized.

One place did not make it into the table. The appendix shows that even the division used to take an average was rewritten. Dividing a floating-point tensor by an integer does not travel across backends, because the processor performs a correctly rounded division while CUDA lowers the same expression to a multiply by a host-computed reciprocal. Unless the divisor is a power of two, the two disagree in the last place. So both the training side and the audit side avoid the division sign and multiply by a precomputed reciprocal instead. A small discrepancy earns no leniency here. Take the gradients reduced across nodes, add them up sequentially on one machine, and you land within a relative error of about a hundred billionth of what the cluster produced, yet the bits differ. In an audit that value gets the same verdict as a badly wrong one.

Nondeterminism comes from three sources: reductions inside GPU kernels, the ordering of batches spread over many machines, and collective communication within and between nodes. Each one gets a device of its own.

3.1Operations That Add in the Same Order Everywhere

RepOps is a library of operations that return the same bits on processors, NVIDIA GPUs and Apple GPUs. Everywhere a sum appears, inside matrix multiplication, inside normalization, in the loss and in the gradient reductions, it imposes a fixed order, and it gives up speed in exchange for the order the hardware would have preferred. On the random number side the character of the thing changed outright. In a counter-based generator the value at index i is the value of a pure function of the seed and a logical position. How the work was split, how many lanes ran, how large the tensor was, none of it enters.

3.2A Data Stream That Does Not Depend on Machine Count

Matching operations are not enough. The step an auditor replays has to receive the tokens the cluster fed into that step, in the same order. An ordinary data loader cannot hold that condition, because the batch ordering changes when the number of workers changes. So the training data is defined as a single canonical stream. The stream is a function of the training seed, the corpus manifests and the sequence length, and it leans on neither node count nor rank. The seed of the generator that orders documents takes a SHA-256 hash of the source name, because Python's per-process hash would give a different permutation whenever the machine changed.

Parallelism arrives after the stream is settled. A different number of GPUs changes only which rank reads which window; the contents of a window and the order of the whole stay put. A machine can die mid-training and the run can be re-sharded onto a different number of devices without shaking the data order, and an auditor can rebuild that step's microbatches on a single machine.

A separate passage records what went into that stream. The report calls the training corpus the Open-1B Mix 0626 and counts 450.5 billion tokens across 332.2 million documents. Four permissively licensed sources are mixed, with token shares of 66.7% for DCLM-Baseline, 13.3% for FineWeb-Edu, 12.0% for The Stack v2 and 8.0% for Proof-Pile-2. The first two come from the web, so about 80% of the whole is web documents. Source weights were set proportional to the on-disk size of each source, so all four were consumed at the same rate, and within the 400 billion token budget each source completed 0.89 epochs. No document was seen twice. One number sits at the head of every ordering here. The global seed of the published run is 42, and the generators that pick sources, order shards and order documents all branch from it.

The authors stop short of claiming data quality as their own work. They lean on the filtering done by the upstream releases and record only three additions of their own. FineWeb-Edu was filtered to an educational score of 3 or higher, which keeps about a quarter of it; The Stack v2 was limited to permissively licensed files of 1MB or less; empty documents were dropped. They ran no deduplication of their own, and they state that near-duplicates therefore remain between DCLM-Baseline and FineWeb-Edu. Being able to count everything that went in, and being sure that what went in is clean, are two different claims. This release establishes the first one.

3.3Collective Communication a Single Machine Can Retrace

The communication that combines gradients across machines rides on order as well. The widely used ring all-reduce is fast, but the order in which it combines depends on the shape of the cluster, so a single-machine replay with no topology has nothing to retrace that order from. Instead the gradients are summed in ascending shard order inside a node, and folded across nodes by a binary tree whose pairing order is fixed. The audit harness runs that same order sequentially on one machine. The repository configuration does not hide the choice. Switch the reduction mode to ordinary NCCL and throughput goes up while the run becomes impossible to replay, and the audit tool refuses such a run rather than reproducing it wrongly.

The run used 48 H100s and spent 27.8 days actually training. In GPU time that is more than 1,300 days. No single machine can go around that again. So the authors split the verification instead. Much as open source software gets read one module at a time, many people each certify a different step, and together they cover the whole run.

A split like that needs a target to compare against at every step. Each step carries one state hash. A single blake2b digest holds the weights, the optimizer moments, that step's post-clip gradients and the per-rank batch digests together, and each step's hash folds in the hash of the step before it. The chain means that tampering with one place in the middle throws off everything after it. Open-1B recorded this hash for every step without exception, and the 810 intermediate checkpoints at 100-step intervals are public. However short the interval you pick, it has something to compare against.

An audit needs no account and no credential. Install two wheels, run one command, read pass or fail. Checking that the files you received match the published sha256 comes before installing anything, and the documentation is blunt about a mismatch: stop there, because verifying other bytes would tell you about some other artifact rather than about the claim. Apple silicon Macs, Linux with an NVIDIA GPU, and a processor with no GPU at all are all first class. The processor is in fact the reference the other two are compared against. Apple silicon and NVIDIA being equally valid is the claim rather than a convenience, the runbook stresses, and a verification on an NVIDIA box is worth exactly as much as one on a Mac.

Why the work of 48 cards fits in the memory of one machine is worked out in the appendix. The harness replays what each GPU in the cluster did during that step, one after another, which leaves partial sums to combine stacked up as high as the node count. The authors chose a fold that works like binary carry. The moment two partials of the same size exist they are combined and carried up a level, so the number of full-size accumulators resident at once drops from the node count to its logarithm. With eight nodes it stays at four rather than eight. The implementation spills the idle entries to disk, leaving only the two terms currently being added in RAM. The premise is that the fold's pairing order matches the cluster's, and a change in that order changes the value.

The cost varies sharply by unit. A unit that only checks the initial state finishes in 29.7 seconds on an M4 Max and wants 5.8GB of memory. A unit that replays one full training step takes something on the order of half a day on a machine of the same class. That is why an audit unit is cut to a single step rather than several, and why one volunteer takes on one step rather than a stretch. The arithmetic on those figures makes the case for splitting plain. One person on one machine, going through all 80,957 steps in sequence, would be counting in centuries.

The repository documentation narrows down what a match proves. A match establishes that the selected replay, with the pinned artifacts and inputs, produced the expected state hash on the tested backend. A unit that checked initialization only does not verify a training interval. The hash in the current version covers weights, optimizer state, gradients and the batch digest, and it does not cover the random state, the data-stream cursor or the spike state. A warning comes attached to the checkpoint directories as well: treat them as code rather than data, because the metadata index gets unpickled during loading before any of the project's own code runs. Replay only checkpoints whose provenance you trust, the documentation advises, and sandbox anything else.

A structure on paper and a verification in progress are two different things. Gensyn publishes the audit record on the web and offers a tally with it. Here is what came back on September 17, 2026. Steps with an accepted check: 19. Contributors: 10. Replay hours reported by volunteers: 67.9 in total. Nowhere among the 810 segments is there one whose hundred steps are filled without a gap. Nine segments have been claimed and are in progress, seven hold at least one provisional record, and the remaining 794 are untouched. These are the numbers of the third day after release, so a small count is not something to fault. The count does show where the threshold of an audit sits.

The slowness has a reason. An audit rides on order. A replay of one step needs the state immediately before it, and the published checkpoints sit 100 steps apart. So an auditor either starts straight from a public checkpoint or picks up an intermediate checkpoint that an earlier volunteer uploaded. A handed-over checkpoint has to have passed the loss values and file digest checks at submission, and the receiving auditor's tool re-checks the digest and the training state hash before the replay begins. The step numbers in the record show that relay plainly. Steps 100, 101 and 102 run under three different names, and steps 12100 through 12103 sit as four under one name. Confirming a single segment takes 100 steps in an unbroken line, and the furthest along today reaches four.

Which machine ran it and how long it took stay in the same record. Volunteers enter those values themselves, so conditions are uneven, yet the size of the gap between machines is unmistakable.

The requirements the app states before anyone claims a step belong next to that table. Memory starts at 24GB, and on Apple silicon the app asks for 40GB or more. Below 40GB the replay runs in swap, which stretches the time, and two runs on swapping machines returned a hash that matched nothing, a different one each time, the app reports. So it advises running a mismatch again on a machine with more memory before reading it as a finding. On an NVIDIA card 24GB is enough, and an audit has in fact run on a 24GB RTX 4090. For free disk space the advice is to plan on 74GiB on a cold setup, and the checkpoint to upload after a match runs about 25.7GB. The time estimate shown against each step is an Apple reference figure measured on a 48GB M5, and machines that swap below 40GB are budgeted at three times that. Step 0 alone stands apart: it cannot offload the optimizer, so 48GiB of host memory is posted as a provisional threshold, and no machine has timed it yet. The claim that it runs on consumer hardware holds, but the consumer hardware in question is fairly well equipped.

There is no reward. The notice attached to the record states first that the list of names is alphabetical and not a ranking, then that there is no payment, no points and no future reward. A name joins the list when the record accepts an audit, while the steps themselves stay provisional until their segment confirms. Ethereum addresses appear only for those who entered one, and five of the ten did. The one thing left is a name on the record of the first fully audited training run, which is how the company put it in its launch post as well.

The report spends a whole section on the cost. The hash comes first. The state hash attached to each step is fixed work per rank, so it stays between 5.77 and 5.96 seconds per step regardless of scale. With more nodes the step itself gets shorter, so the share taken by that fixed cost grows. What was 4.9% on one node becomes 9.3% on two, 16.7% on four, and 21.4% of step time on six. The strong-scaling efficiency of 71% measured with hashing on, against 86% with it off, comes from the same place. The forward and backward passes taken alone reach 99.7%, so most of what is lost sits in the hash and in the communication between nodes.

That figure should not be counted together with the cost of reproducibility. The appendix draws a line: the hash is an audit cost and not a reproducibility cost. Turn hashing off entirely and the deterministic kernels and collectives still return the same bits. The cost shows up at the intermediate steps, which then have no target to compare against. How much the order-pinned communication demands as nodes are added was measured separately. A deterministic all-reduce fixes the order at the price of sending the whole buffer on every hop, so against ordinary NCCL measured on the same subgroups it costs 1.13x at two nodes, 1.59x at four, and 3.54x at six. The jump at six nodes comes from the node count not being a power of two, which splits the fold into two blocks, rather than from bandwidth. That is why the roughly five times summary is a figure with conditions attached. Part of why the gap looks narrow at six nodes lies on the baseline side. This cluster uses a socket fabric without GPUDirect, where PyTorch's strong-scaling efficiency sinks to 66% while the reproducible arms hold 86% to 91%.

The larger cost is on the kernel side. On a single node, the GPU utilization of the bitwise reproducible bf16 kernels came to 3.3%, while the same model and the same recipe in stock PyTorch reached 40.5%. That is a 12.2x gap. int8 quantization-aware training cut it to 6.8x, and going up to six nodes narrows it to about 5x. The authors summarize the burden of reproducibility as roughly five times the cost of optimized PyTorch. The biggest room for improvement they name is the hash, which is computed on the host today, so a memory transfer between device and host cuts into every step.

That burden stays visible in the benchmark scores. Training tokens stopped at 400 billion, and OLMo 2 1B, the comparison model, saw 4 trillion. The OLMES macro score is 50.1 against 61.5, and the average over the ten tasks is 25.4 against 31.9. Open-1B leads on two of the ten, ARC Challenge and MMLU Pro. The comparison figures were measured by the authors themselves. Since the OLMo 2 report does not publish five of the ten tasks, they pinned an evaluation harness, ran it on the public OLMo 2 1B checkpoint, and recomputed the macro. In the conclusion the authors name the causes of the difference directly: reproducibility overhead, resource constraints, and a token budget ten times smaller. This is closer to losing for want of the same conditions than to losing under them.

The published run did not arrive in one go. The report carries two failures they walked through on the way to it. One was a gradient that crept upward from around 116 billion tokens. Digging in, they found that the backward flash attention implementation, quantized at int8, was producing zero gradients along the query path, and the gain parameter of the QK normalization had been growing monotonically to make up for it. The prescription was a scale factor inside the gradient kernel that does not break reproducibility, plus the removal of the gain from QK normalization. The other was the gradient norm of the first block passing 1,000 after about 2,000 steps. They swapped out candidates one by one, from the integer attention kernel to the learning rate, until the weight quantizer was confirmed as the culprit, and moved from learning the step size to recomputing it every step. The cost is about 2% of step time. Both prescriptions are in the published recipe.

The second prescription also shows reproducibility narrowing the options. The usual prescription for instability in low-precision training is stochastic rounding, which pushes a value to the lattice above or below it by chance, and the randomness in it cannot travel with bitwise reproducibility. Not unless a counter-based generator is pulled inside the optimizer as well. The step size recomputed each time is a deterministic function of the weights, so it needed no such machinery, and the other road under consideration at the time, disabling quantization, would have demanded about 70% of step time. The skipped steps point at the data side in a similar way. Two batches in the entire run had a pre-clip gradient norm above 5, at steps 1,782 and 43,737, and a protocol that skips five steps per event took out ten in total. Clipping engaged on 0.16% of steady-state steps, which the authors read as a data anomaly such as a batch with a low unique-token ratio rather than as a trace of unstable optimization. Because data order is a function of the seed, exactly which documents that batch held can be retraced from the step number alone.

That the whole run finished in seven segments because of engineering problems did not remain purely a blemish either. Each segment restarted from the most recent checkpoint, and the hashes of the overlapping steps were compared against those of the preceding segment to confirm that the restart was correct. Seven restarts means the test of rerunning the same step and seeing the same hash had already happened seven times before release.

From here the article leaves the report and rereads the news from the side of people who work with data. The question Pebblous has held onto for a long time is where a value came from and what it passed through to look the way it does now. When a model provider says it learned from this data, whether you can confirm that yourself is the last gate of that provenance problem. Hardly any provider today can answer yes to that question. This case puts up one existence proof that a way to answer yes could exist.

One attribution should be settled first. Gensyn wrote this question into its own launch post on the same day. The post names proving what data was trained into a model as the first of the applications that auditability opens up. Alongside the model, the post reports releasing a search tool that goes through the verifiably complete training data. Labs could use these techniques, it adds, to let consumers know what parts of their own data, their likeness or their content for instance, are being used to train a model, similar to how websites provide consumers with their data on request today. The conclusions the passages below draw for data pipelines are what this article adds, but the framing of the problem itself belongs to the company.

The lesson worth carrying over lies in the order of design. Auditability is not a property you can attach once training is over. Fixing the order of operations, turning randomness into a function of position, and prying data order loose from the number of machines all have to be nailed in before the first step runs. Moved to the data side it reads like this. To be able to dig up provenance later, the pipeline has to be built from the start in a form that can reconstruct which document went into which window. That is a different task from restoring the record by digging through logs after the fact, and starting late usually means it cannot be restored at all. The launch post nails the same point down in one line. The verification tool released today cannot be used to audit Llama or GPT.

The boundaries are worth stating clearly too. The scale is 1.6 billion parameters, the speed penalty is about five times, and exactly one model has been verified this way so far. Under the guarantee there is also a layer of trust left. The report's conclusion calls RepOps, which every operation passes through, a proprietary library, the audit wheels carry compiled kernels only, and the source tree is not public. The runbook does not conceal the point. Shipping machine code means shipping something that can be disassembled, it writes, and reading an implementation back out takes real work without being impossible; then it says the kit asks you to accept arithmetic you can re-derive rather than code you cannot look at. Whether to accept that trade is the reader's call, and that where trust remains and how much of it is written into the documents says something about the character of this release.

Why this particular company did this particular work is part of the picture as well. Gensyn builds decentralized compute networks. The launch post says the process is functionally identical to decentralized training, except that model verification rather than model creation has been shifted to those nodes. Since the training itself was finished in a centralized way, the explanation goes, it avoids the huge upfront costs and the communication problems that decentralized training carries. The same post claims the approach can be scaled to larger models today. What the report actually measured points the other way, with the cost of order-pinned communication rising as nodes are added, and the distance between the two has not yet been filled in with published figures.

Teams that train their own models, or bring in models from outside, may find these five questions worth asking.

• Can we point, document by document, at what our model learned from? If we cannot point at it, we also cannot say what did not go in.
• Does the same recipe give the same result when we run it again? We should be able to answer that separately for the same machine and for a different one.
• Is our data order tied to the machine configuration? A pipeline that produces a different model when the GPU count changes proves nothing later on.
• How often do we record intermediate state? With nothing recorded there is no target to compare against, and no way to narrow down which interval a problem came from.
• When we look at a model brought in from outside, are we distinguishing between a published result and a published process? Releasing weights and releasing the process are different promises.

Thank you for reading this far. The figures and sentences this article cites can be read in the original by anyone, in the report on arXiv and the audit harness repository, and the audit app lets you page through the step-by-step record yourself. We would be glad to hear how your own organization confirms what a model learned from.

Academic Paper

• 1.Donaghy, J. et al. (2026). "OPEN-1B: A Fully Auditable Training Run." arXiv:2609.17380.

Primary Documents & Audit Tooling

• 2.Gensyn. "gensyn-ai/open-transformers." GitHub, audit harness repository.
• 3.Gensyn. "Audit Volunteer Runbook." GitHub — gensyn-ai/open-transformers.
• 4.Gensyn (2026). "Introducing open-1b: the first model you don't have to trust." Gensyn News, Sep 15, 2026.
• 5.Gensyn. "Open-1B Audit." Live coverage and ledger, as retrieved Sep 17, 2026.