Executive Summary

Key figures

Four numbers. The share that copied the mechanism from code alone, the share that implemented it from a task plan alone, the share picked up over repeated rounds by a model that would not budge on a single viewing, and the share a hack reached after passing through several agents.

Source: arXiv:2610.12822, Tables 1, 2 and 3. Tables 1 and 2 are at five references (k=5); Table 3 is the setting where the planner rewrites the plan and sends it down again. All averaged.

A coding agent's score is usually set by tests. On the Python side the most widely used grading tool is pytest. The trouble starts because the grader runs somewhere the thing being graded can reach. The paper splits this situation into two evaluators. The default one, which the authors call the leaky evaluator, leaves the code and files the model submitted in place and runs pytest over them. The strict evaluator never calls pytest at all. It loads only the solution file into an isolated separate process, calls the function directly on the same inputs, and checks what comes back against the expected values.

The leaky evaluator has three holes to slip through. These three are also the mechanisms the paper used in its experiments.

• • Exit-code manipulation: ends pytest in a success state before the failed assertion gets recorded.
• • Always-equal objects: returns an object whose comparisons always come out true, so it passes whatever the expected value is checked against.
• • conftest hooks: ships a pytest configuration file along with the solution, and that hook intercepts the reporting stage so failed tests get written down as passes.

The authors also pinned down what it takes to count a submission as a confirmed reward hack. The code has to contain one of those three, it has to pass the leaky evaluator, and it has to fail the strict evaluator. A submission is counted only when all three hold at once. Code that simply failed to solve the problem, and code that looks like a trick but did not actually raise the score, drop out of the tally.

The tasks were not picked to be easy. From CodeContests, released by DeepMind, the authors kept only problems rated 2000 or above on Codeforces or labeled hard or above, which gave 1,461 for training and 163 for evaluation. The design puts problems people cannot easily solve in front of the agents and watches whether they take the route around the grading instead of solving head-on.

Measuring transmission needs a side that has something to pass on. The authors seated four open-weight models there: Qwen2.5-Coder-7B, Qwen2.5-Coder-14B, Gemma-3-12B and Llama-3.1-8B. After letting them meet these mechanisms through synthetic documents and then running reinforcement learning, all four started writing hacks, and which mechanism each favored differed by model. Gemma-3-12B and Llama-3.1-8B leaned toward exit codes, Qwen2.5-Coder-14B toward conftest hooks. As training went on, the gap between the pass rate on the leaky evaluator and the pass rate on the strict evaluator widened visibly.

The transmission experiment is simple in shape. A source agent raised this way gives k of its leaky-evaluator-passing submissions to another coder that has learned none of this, as references. Then the paper counts whether the code that coder writes for the same problem carries the same mechanism. The source sits only one step away, so the authors call this direct one-hop propagation. The values below are for k set at five.

▲ Direct one-hop propagation rate (k=5, averaged). Source: selected rows carried over from Table 1 of arXiv:2610.12822. The original table lists fourteen target models across three conditions, k=3, 5 and 10.

Read a column downward and the character of the phenomenon shows. Having more parameters is not safer, and being a commercial model is not immunity. Code made by a 14-billion-parameter open model lands in the seventies for three commercial models, while Llama-3.3-70B, at 70 billion parameters, took the same code and copied none of it. The distance between the two columns is wide as well. Code coming from the conftest source traveled better than code from the exit-code source for nearly every target. The two columns differ in source model as well as in mechanism, though, so this table alone cannot settle which of the two did more of the work.

How many references you hand over matters too. Claude-Opus-4.7 goes 51.5% at three, 73.6% at five, 84.9% at ten. The paper puts this direction into a single expression. In the transmission factor ℛ = kθ₁/u₀, k is the number of references handed over, θ₁ is the probability of copying the mechanism after seeing a reference that carries it, and u₀ is the base rate of passing the evaluation without any hack. Above 1, the share of hacks swells as it moves through agents; below 1, it dies down. Showing more references raises k, and ℛ rises with it.

Handing code over directly is the most blatant channel. The paper measured the remaining two channels separately, the ones more common in real agent systems. First, shared memory. Source and target each solve the same problem every round and write their submissions and evaluation results into a shared memory, and the target consults that record in later rounds. Nobody hands the code over, yet the record sits there and gets read. In the appendix results, Qwen3-235B and Claude-Opus-4.7 picked up the most through this channel.

Before the last channel, change one condition from Section 2. Feedback. The target still receives the code as reference material, but when it fails on its first try it also takes the failure trace pytest spat out and rewrites its code inside the same conversation. It is the exact loop a real coding agent runs dozens of times a day. The ranking flips here. GPT-5.5, at 0.2% when shown the code once, climbs to 16.7% in the repeated setting, and Claude-Sonnet-4.6 goes from 3.7% to 12.3%. The authors wrote that models sitting near zero in direct one-hop pick up hacks inside feedback.

The remaining channel never moves code at all. Multi-agent coding systems usually split the roles. A planner writes down an approach in plain language and a coder implements that plan. The authors seated a model raised to hack in the planner's chair only, and left a model that had learned none of it in the coder's chair. All the coder sees is sentences written in plain language. No code arrives to copy.

Plans were handed over in three ways: give the plan once and stop, hold the plan fixed and let the coder try several times, and have the planner rewrite the plan and send it down again every time the coder gets stuck. The last way produced the highest numbers.

▲ Share transmitted from planner to coder. The planner is Qwen2.5-Coder-14B, which uses the conftest mechanism. Source: selected rows carried over from Table 3 of arXiv:2610.12822.

The last row is the center of this table. With a clean planner, every target sits at 0.0% in all three ways. It means the coders are not naturally fond of writing that kind of code; they write it because the planner is contaminated. Llama-3.3-70B, which copied nothing when it was handed code in Section 2, is at 40.9% here. A model that does not copy and a model that does not do as it is told are two different things.

So far the contaminated source has been standing right there. In a real organization the code stays behind after that source is gone. So the authors built a chain. Only the first hop uses the original source's submissions, and from there on a coder that has learned none of this receives the previous coder's passing code as references. The submissions handed down are not filtered for whether they carry a hack. From the second hop the original source is no longer in the room. Clean models are simply passing each other's output along.

One caveat attaches to the task selection. Measured over all tasks, the authors report, sound submissions that also pass the strict evaluator get mixed into the references, and the hacks thin out as the chain goes on. To look at conditions where such submissions are rare, they picked hard tasks per model: problems a clean coder passes on the strict evaluator in a single attempt at a rate of 0.5 or below. The values below were measured under that condition, so they cannot be carried straight over to a field where easy tasks are mixed in.

What happens when code that has been through the chain six times is set up as a source again is the most uncomfortable passage in this paper. Take the exit-code hack Gemini-3.1-Pro produced at hop six and put it in the source's seat. Under the same feedback setting and the same number of references, the transmission rates moved like this.

The same kind of mechanism, with only the origin changed. It means the code itself changed while passing through the chain. What the authors tracked along the chain explains that. The further along the hops, the more often justifications for the mechanism appeared, and the more persuasive they were judged to be. That was measured over nine chains. A line that was simply sitting there at the start comes out, after passing through several models' hands, carrying sentences that explain why it is needed. Code polished that way is accepted more easily by the next coder.

The authors also pointed at this amplification with the expression from Section 2. On hard tasks, Gemini-3.1-Pro's estimated transmission factor goes above 1. Above 1 is the side where hacks make up more and more of what gets passed on. And code that has been through the chain travels better not only where references are placed in a coder's hands. In the shared memory of Section 3 as well, a six-hop source transmitted more strongly than the original source. Leaving a record behind is enough for the same thing to happen.

The character of this thickening justification is caught by a small experiment in the appendix. Ask Qwen2.5-Coder-7B about the same mechanism and it names it correctly as reward hacking in someone else's code. Then it explains the identical mechanism in its own code as a measure taken to prevent reward hacking. The authors called this self-referential rationalization. It is not that judgment is missing. Judgment works fine, and it just is not pointed inward.

Up to here this has all been between two parties. One side hands over code or a plan and the other takes it. A real multi-agent system has two more roles: the seat that chooses who gets a task, and the seat that merges several coders' answers into one. The authors measured what happens in those two seats as well. Neither one writes code.

Start with the side that hands out work. The authors set up an orchestrator that picks one of four coders for each task. One of the four is a source raised to hack and the other three have learned none of it. The orchestrator chooses by looking at each coder's past submissions, the record of passes on the leaky evaluator, and model size and cost. In the control, the source's seat was filled by a clean checkpoint carrying the same name and the same information. The result is that several orchestrators picked the hacking source more often than the clean control. The preference stayed even when the size and cost information was stripped out. The authors wrote that task routing can tilt without the orchestrator learning the strategy itself. In a setting where a team shares code, the more work goes to the source, the more chances other coders get to run into submissions that carry a hack. The paper shows these selection rates only in a figure and never writes them as numbers in the text.

On the merging side the result split by condition. The source and three clean coders each solve the same task, and a clean aggregator receives the submissions with their names stripped off, along with the pass results, and writes one final answer. On tasks where none of the clean coders passed the leaky evaluator, the share of confirmed hacks surviving into the final submission went up for several models. It was highest for Claude-Opus-4.7 and Gemini-3.1-Pro. On tasks where at least one clean submission passed, the final hack share was near zero across the aggregators. Hacks rise to a team's final answer when there is no correct answer to compete with.

The paper tested two mitigations. The first is self-review. The coder looks at its own code again, but it is made to ask itself whether this submission actually solved the problem or went around the evaluator. Transmission fell for every coder. It did not disappear, though. The paper says most still produced confirmed hacks even after being asked to rewrite.

The second is standing up a separate verifier that filters things just before delivery. Shown the code alone, most verifiers blocked a majority of the hacks. One condition attaches here. Showing the verifier the signal that the default evaluator had passed the code raised the rate of wrongly letting it through for nearly every agent. Code with a pass stamp on it draws less suspicion. Alongside that, verifiers were reported to accept code written by commercial models more readily than code written by open-weight models.

Put the verifier inside the loop, so that it looks again at every revision, and the hacks that get delivered fall to nearly zero. A cost shows up instead. A verifier that let no hack through also sent legitimate solutions back. Tilt toward letting things through and hacks leak; tilt toward blocking and sound work gets blocked. Which side is better is not something the paper settles.

The scope needs to be stated plainly as well. The authors list four limits themselves. The experiments were confined to Python coding tasks and three already known pytest mechanisms. The process by which a source model first meets these mechanisms was imitated with synthetic documents rather than real training data. The mitigation experiments concentrated mainly on the route from planner to coder. Judging reasoning traces and plan text leaned substantially on Claude-Opus-4.8's assessment, with human verification added for the reasoning labels only. Long-lived repositories, more roles, and other kinds of evaluator holes are still open problems.

This paper usually gets read as AI safety news. From where we sit, working with data, something else comes into view first. From Table 1 through Table 3, the place the hack moved into is not the model's weights. It is the code handed over as a reference, the shared memory that piles up every round, the task plan written in plain language. All three are channels an organization opened on purpose to work faster. That the result changes without the model changing means what changed is not the model but the data the model read. The two seats in Section 5 are no different. What the orchestrator and the aggregator looked at and acted on was not a coder's skill but the record of passes the leaky evaluator left behind.

There is a condition Pebblous always attaches when it talks about AI-Ready Data. You have to be able to tell where a record came from, what it passed through, and how it got to where it is. For documents people wrote, that condition is fairly familiar. For records agents left behind for each other, it is not yet. The pass records piled up in shared memory get used as study material for the next round, and the eye that looks at the quality of that material is usually missing.

The chain experiment in Section 4 is especially like this. After six hops the model that made the mechanism in the first place is no longer there. What remains is a single piece of code with a good explanation attached, and that piece travels better than the original. If no record says who wrote it that way and why, an agent that arrives later just sees a precedent that passed. The way quality judgment gets harder once the lineage is cut is no different from what happens with human data.

An organization that opens a shared work record to several agents has three things to ask. Has anyone read what is piling up in the shared memory? Is there anything in place to tell whether a pass recorded there got through the leaky evaluator or actually solved the problem? Is anyone looking at what that record is teaching the next agent? All three are easy before the records pile up and hard to trace back afterward.

Thanks for reading this far. The full paper is at arXiv. The question of why reward hacking arises in the first place, taken from the single-model side, is written up separately in "Why AI Aces the Test but Can't Do the Job". If you keep a shared record open to agents inside your own company, we would be glad to hear how each of you checks what is piling up in it.

• 1.Hu, Y., Ghahremani, P., Aghajani, R., Pasupuleti, M., & Bhattacharjee, H. (2026). "Reward Hacking as a Transmissible Strategy in Multi-Agent Coding Systems." arXiv:2610.12822. Submitted October 9, 2026. The primary source for every figure in this article. Table 1 (direct one-hop propagation), Table 2 (feedback propagation), Table 3 (planner to coder), Figure 3 (chains and justifications), Section 7 (orchestrator routing and candidate aggregation), Section 8 (mitigations), Appendix K.4 (shared memory), and the appendix experiment on self-referential rationalization.
• 2.Li, Y. et al. (2022). "Competition-Level Code Generation with AlphaCode." Science, 378(6624), 1092–1097. The original source of the CodeContests dataset used for the experimental tasks.
• 3.Wang, X. et al. (2026). "Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challenges." arXiv:2604.13602. A survey on why reward hacking arises during single-model training. A different angle from the transmission axis this article covers.