Executive Summary

Key figures

Source: Chen Shen, What Eviction Destroys: A Restore-Counterfactual Audit of Forgetting in Agent Memory, arXiv:2609.08279 (2026), Table 1, §5, and Appendix C

Over days and weeks with the same user, what an agent has been through outgrows its context window. The memory pipeline therefore decides in turn what to write down, what to keep and what to let go, and what to read back when a question arrives. The paper calls that middle step eviction: the moment the budget overflows and stored material gets selected for disposal. Curves that trace the accuracy lost as the budget shrinks already exist on both the theoretical and the empirical side. Those curves state the size of the loss and say nothing about its composition.

Prior work has piled up around this exact point. The optimal boundary for how much an agent may forget has been formalized. Cost and accuracy have been charted empirically on LoCoMo and LongMemEval. Other work goes further and learns what to retain inside a budget. The empirical line reports end-task accuracy without breaking down why that accuracy fell. Some papers do split failures apart. WhenLoss separates write-side mass from retrieval-side mass in aggregate, and the work the author names as closest injects paired oracle probes into an agent-written store to diagnose write, retrieval, and utilization failures. That one aims at aging mechanisms on custom scenarios rather than at eviction driven by a capacity bound. This paper differs on two counts: it splits at the level of a single question instead of an aggregate, and it attributes the error to one eviction decision.

Two identical drops in accuracy can have opposite causes. If the evidence a question needs has already been pushed out of the store, no retriever will bring it back and retention is the only lever. If the evidence survived and retrieval simply missed it, there is no reason to retain more and fixing the retriever is enough. The introduction puts the situation in a single line.

“A frontier point conflates two failure modes with opposite remedies. … An all-recoverable frontier and an all-irreversible frontier can look identical on the accuracy axis but require different interventions.”

Turning eviction off entirely shows why the distinction earns its keep. The no-evict reference still left 95 errors, and 38 of them concerned evidence sitting untouched in the store that retrieval failed to pick up. Retrieval was leaking even where destruction was zero. When accuracy falls under a tightened budget, the curve alone will not tell you how much of that fall a new retriever could fill back in.

The simpler alternative is worth ruling out first. Subtracting accuracy with eviction on from accuracy with eviction off summarizes the loss in one number, and that number can diverge from the damage eviction actually did. Recent work on KV caches found cases where a selectively evicted cache scored better than the full one. A single aggregate difference cannot count what broke.

The restore counterfactual is an intervention applied to one question at a time. Working over the store left behind by a policy, it takes a question the reader answered wrong, forces that question's gold evidence back into the read-time context, and reruns the same model at the same prompt and the same temperature. Reader, judge, and ranker all stay frozen, so availability of the evidence is the only thing that moves. The change in correctness is recorded as restore_gain(q) = acc(restored) − acc(policy), which takes one of three values: −1, 0, or 1.

The verdict comes down to two questions. Did the answer flip from wrong to correct, and had at least one piece of that question's gold evidence been evicted? A flip plus an evicted piece means the loss is irreversible. Destruction happened, and retention is the only repair. A flip with all the gold still present means the loss is recoverable. Retrieval missed it, and a better retriever fixes it. An answer that stays wrong even with every piece of gold handed back counts separately as residual: the reader held the evidence and failed to use it.

The worked case in the paper asks how long the user's daily commute takes. At an 80k budget, FIFO had pushed out the session recording it, and the reader answered “I don't know.” Reinstating the evicted session produced the correct “45 minutes each way.” One gold unit had been evicted and restoration flipped the answer, so the case is irreversible, and the only fix is keeping that session.

That “I don't know” came out exactly as instructed. The reader prompt in the appendix tells the model that if the snippets truly do not contain the answer it must reply exactly “I don't know.” The answer therefore marks the place where evidence went missing rather than a model hedging.

The other two bins come with real instances as well. On the recoverable side the question asks how many magazine subscriptions the user currently has. The gold answer is two, yet even with eviction turned off the reader replied “one (The New Yorker).” Top-60 retrieval had returned the New Yorker entry and left the other one behind, and restoring the missed entry corrected the count to “two (The New Yorker and Architectural Digest).” An error arising where nothing at all was discarded shows what this bin is made of.

The decomposition carries one more adjustment, and it narrows the denominator. Only questions the reader answers correctly when handed the clean gold evidence are counted. Under that restriction a scored error points at what the budget did to the evidence rather than at a question the model was never going to solve. Starting from the 470 evidence-labeled questions of LongMemEval-S, the author dropped the abstention questions and applied this filter, which left 336 questions as the denominator of the baseline grid. Each session history runs to roughly 102k tokens, so all three budgets of 80k, 30k, and 8k genuinely force eviction.

Before the main measurement began, the author checked whether the instrument itself was loose. Restoration can also turn a correct answer wrong, which is the −1 value, so a 40-question calibration run measured that rate first. Against a cap of 0.05 the observed rate was 0.00, and the judge's repeat self-consistency came out at 1.00 against a threshold of 0.90. Even when a −1 does occur, the irreversible value is clipped at zero, so a restore that spoils an answer never counts as destruction. Session ids holding the answer are hidden from reader and judge alike, and what gets reinstated is capped at 2,000 tokens, a ceiling no gold set approaches since the largest runs to about 1,000.

The share the paper puts in its headline gathers only the errors restoration corrected and asks what fraction of those was irreversible. Residual errors sit outside it. Bring residual back in and the three-bin share at 80k tokens drops to somewhere between 0.40 and 0.44 for FIFO, random, and redundancy-aware eviction. The same table supports both readings, so any figure carried out of it has to travel with the denominator attached.

Below are the values at 80k tokens, the loosest budget, measured under the realistic condition of top-60 retrieval. The middle four rows only read properly next to the no-eviction reference above them and the deliberately destructive control below.

With a tighter budget the composition tips one way. At 30k tokens the irreversible share sits between 0.98 and 0.99, and at 8k tokens all four policies reach 1.00. The raw counts say it more plainly than the shares do. Recoverable errors under FIFO went from 22 to 3 to 0, redundancy-aware from 25 to 3 to 1, and LLM-importance from 23 to 4 to 0. The narrower the budget, the fewer errors remain that a better retriever could rescue. The prescription in the discussion section runs the same way. Where the budget is tight, retention deserves attention before retrieval does, because a better retriever gains little until more evidence survives.

The shape of a question also sorts the composition. Questions answered inside a single session land in the irreversible bin about 95% of the time, while questions that span sessions or require temporal reasoning carry residual shares of 23% to 34%. Swapping in the stronger GPT-5.4-mini reader left the reject counts and the ordering of policies unchanged, and the residual bin was the only component that moved.

The reason for splitting the condition in two came out of earlier work. A frozen retriever misses evidence that is sitting perfectly well in the store, and that has been confirmed more than once. Eviction effects can therefore only be read under controlled retrieval conditions, and the author went one step further by promoting the difference between conditions to a result in its own right.

The paper ran the same experiment twice, under two read-time conditions. One is forced-gold injection, where any surviving gold is guaranteed to reach the reader. The other is a frozen ranker that pulls the top 60 units by lexical overlap. The first condition exists to isolate destruction, which empties the recoverable bin by construction and pins the irreversible share at 1.00. That 1.00 is an algebraic identity rather than a finding. The paper says so in as many words and insists the real numbers come only from the second condition.

The distance between the conditions is itself the result. At 80k tokens the recoverable-share gap runs +0.29 for FIFO (CI .19–.39), +0.27 for random, and +0.33 for redundancy-aware, all surviving Holm correction (p=.007). With no eviction at all the gap reaches +1.00. A small gap for random at 30k tokens (+0.019) also survives correction. Of the 12 cells involving eviction, 4 reject, while the 3 no-evict cells have an empty denominator under forced gold and drop out of that count. The gap concentrates at 80k tokens for a simple reason. Tighter than that, the evidence is destroyed outright and nothing is left for retrieval to miss.

The demand lands on the paper itself with equal force. The ranker used here is a frozen one that looks only at lexical overlap, so the share classified as recoverable measures what that ranker missed rather than the limits of retrieval in general. A better ranker would keep those errors from arising at all, which would shrink the recoverable bin and lift the irreversible share. Anyone carrying this audit into their own pipeline should therefore record the read-time setting before recording the budget. The same condition applies when last month's number gets held up against this month's.

The comparison in this section is not the test that was registered in advance. The original statistic examined the irreversible share under forced-gold injection, a condition where the value is structurally pinned at 1.00 and can detect no difference whatsoever. The author calls it a degenerate statistic, states that the test was replaced after measurement, and flags the replacement as exploratory. Two confirmatory tests remain as registered: one checking that the decomposition is well defined, and one measuring the gap between the two retrieval conditions.

Match the policies on accuracy first and the differences disappear. Across the three baseline pairs at three budgets, all 9 comparisons within an accuracy caliper of 0.05 showed no detectable difference in irreversible rate, with every confidence interval spanning zero. The LLM-importance policy, which scores general importance from 1 to 10 without ever seeing the question, behaved the same way in the 6 comparisons where accuracy matched.

LLM-importance does post the lowest irreversible share at 80k tokens, 0.60, and that cell is not accuracy-matched. At 0.69 against roughly 0.62 it falls outside the caliper and drops out of the comparison from the start. The paper keeps the number as a descriptive observation and draws a line against reading it as a policy ranking. At 30k and 8k tokens, where accuracy does match, no difference from the baselines was detected. The prompt behind that policy appears verbatim in the appendix. It asks the model to rate how generally important a single memory snippet is to remember about the user, from 1 for trivial small talk to 10 for a durable fact, preference, or commitment, and it never reveals what will be asked later.

So was the instrument simply too blunt? A control that deliberately evicts gold evidence at roughly three times the baseline rate produced a significant difference in all 9 contrasts, which confirms the instrument does register large destruction. The paper attaches the caveat itself: the contrasts were not matched on accuracy, so they did not establish power inside the caliper. Sensitivity for the matched comparisons is the paired-bootstrap confidence interval half-width of 1.2 to 6 percentage points. The conclusion is accordingly worded as no dissociation detected at this resolution rather than no difference exists. An equivalence claim, the author notes, would need its own test, and that is left to follow-up work.

The residual bin resolves into a reader problem. Errors that survive a full restore come almost entirely from counting and summing across sessions. One case answers $65 to a question whose yearly bike spending totals $185. Cases attributable to an overly strict judge account for at most 20% even on a conservative reading. The judge prompt requires dates and numbers to match in value, so there was never room to let $65 pass for $185. Under the stronger reader this bin more than halves, dropping from 59 to 28 for no-eviction at 80k tokens under forced gold, and the cases it fixed were precisely those aggregation errors.

Whether the irreversible verdict is inflated was measured separately. Of the 2,276 cases classified as irreversible, 87% had no surviving gold at all. Restoring only the gold that survived reclassifies 0.3% as recoverable, and injecting the entire retained store flips just 2.0%. For 98% of these cases, nothing left in the store produced an answer. A placebo test that puts equal-length unrelated text in the gold position flipped 8.3%, against 100% when the real gold went in. Availability of the evidence decided the answer, not its position.

The paper ends there. Anyone who works with data will recognize the decision it describes. A retention period expires or an erasure request arrives, and we delete the record. What we deleted goes into a log, and what that deletion later made impossible goes nowhere at all. The problem of identifying what to delete came up on this blog in Memory Without a Name Tag Can't Be Erased, and this paper aims at the next square over: the question that remains once deletion has already run.

The author addresses that square directly in the ethics statement, noting that privacy and data-protection requirements sometimes make forgetting necessary, and that retaining more may then be undesirable or disallowed. The passage continues.

“The instrument quantifies the task cost of a forgetting decision without prescribing whether information should be retained or deleted; the same decomposition can audit whether a required deletion incurs recoverable or irreversible task loss.”

Whether something ought to be deleted is not for this instrument to judge. It measures whether a deletion already carried out produced a loss that can be undone. Three pieces are worth carrying outside the agent pipeline.

• Keep a place, separate from the deletion log, that measures what became impossible. Counting the tasks that failed after a deletion shows the loss in a way that counting deleted items never will.
• Sort the failed tasks into two bins. Something retrievable from elsewhere and something whose original is gone call for opposite responses. Adding another index helps the first kind and does nothing for the second.
• Record the retrieval setting alongside the result. Last quarter's number and this quarter's number from the same pipeline cannot be placed side by side if the way they were read back has changed.

The limits the author drew belong in the transfer as well. The audit runs on gold evidence labels attached to every question, which suits benchmark analysis and rules out attaching it to a live system unchanged. The eviction arms tested are policy classes imitating the shape of the real thing rather than shipped memory systems. The results rest on one benchmark, two readers, and one primary judge, and that judge shares its provider with the reader. An independent judge agreed at 95.7% (Cohen's κ = 0.90), and since that judge comes from the same company, the author notes that a model from a different provider or human annotation would be a stronger check. Most of the disagreements went the way of the new judge grading more strictly. Scope narrows in one more place: abstention questions were filtered out at the start because they have no gold location to restore, so this audit says nothing about whether an agent admits it does not know.

Thank you for reading. The paper is at arXiv:2609.08279, and the evaluation code with per-question records is public at github.com/megagonlabs/restore-counterfactual. The mirror case, where deleted content keeps disturbing search results, is written up in Deleted Records Kept Reshaping Vector Search Results. If your team already runs something that tells a retrieval failure apart from a permanent loss when a deleted record turns out to be needed, we would be glad to hear what standard you use to draw that line.