Executive Summary

Frank Gilbreth, the efficiency engineer of the early twentieth century, buttoned his waistcoat from the bottom up because that took three seconds instead of seven. For a while he also shaved with two razors at once. Then he gave it up. The grumble that followed survives in the memoir his children wrote.

Three years of research into how much faster AI makes work fit inside that one scene. There is a tool that speeds up the visible part of a job, a cost that turns up somewhere else, and a small sample. And Gilbreth had one thing the recent literature often lacks. He counted the bandages. That is where the paper's title, Counting the Bandages, comes from.

What the recent work lacks is that bandage column. Which is why answers to the same question pile up and disagree. Developers using Copilot finished a bounded programming task in 56% less time. In METR's public-repository experiment, measured from February to June 2025 with Cursor Pro and Claude 3.5 and 3.7 Sonnet, experienced developers took 19% longer. Consultants were 25% faster inside the region the model handles well, and on a task deliberately placed outside it they were faster too but 19 points less likely to be right. OpenAI's GDPval announcement said frontier models are "roughly 100x faster and 100x cheaper than industry experts," and the GDPval paper itself, once an expert reviews the output and redoes what fails, puts the figure for GPT-5 at 1.12 to 1.39 times.

These numbers do not contradict each other. They were never measuring the same thing. The paper's point is that there are two ways of seeing "time saved" and that the two have never been pulled apart.

The first stands there with a clock. Run the task once each way, read off the two durations, and the gap between them is the saving; the measuring is finished when the clock is. Whatever follows belongs to a different piece of research. The second opens a ledger. The saving is what separates two complete workflows carried all the way to the point where the deliverable is actually put to use, and whatever minutes landed on somebody else's desk en route are entered there as well. Call the first the stopwatch view and the second the ledger view.

The distinction the paper draws in its first section. A clock records where time was spent; it does not record what the time bought. Some of what looks like saved time is moved time — from writing to reading, from now to later, from me to whoever finds my mistake.

The thought experiment the paper uses to separate them is short. Two analysts get the same model and the same task: a one-page brief built from figures in three companies' filings, ninety minutes by hand. The first reads the draft against the filings, finds two wrong figures, fixes them, and is done in thirty-four minutes. The second reads the draft, likes it, and is done in six. Who saved more?

On the stopwatch, the second, by a wide margin. In the paper's phrasing, the second analyst has rediscovered "the oldest productivity technology there is, which is not checking." On the ledger the second analyst has saved nothing yet. They have borrowed twenty-eight minutes against two wrong figures, at a rate that will be set by whoever finds them.

The author is Solomon Shalom Lijo. He gives his affiliation as independent researcher and declares no outside funding. The arXiv posting date is 23 October 2026, the length is eighteen pages, the licence is CC BY 4.0, and there is a mirror on Zenodo. His day job is at Rōmy, a company that builds AI software, and he states inside the paper that nothing in it evaluates that product. There is one unusual thing in his record. This paper withdraws a proposal the same author published in April 2026 and takes its place. Back then he put forward a five-metric "time-saved benchmark," and in this paper he writes out how it was wrong. The reliability-adjusted metric charged a full redo on top of completion times that already contained the redo; it treated errors that ship as though they cost redo time; and "two of its headline numbers did not survive contact with their sources." Rather than patch the old version he replaced it whole, and he left the wrong code in the repository so that the error stays reproducible.

Before the argument proper, three terms. The whole article moves on these three. The formulas come out in section 2; here only the meanings.

• Time saved — by what percentage the mean completion time of the arm with the tool falls below the mean of the arm without. It is what a manager means by "we saved 30% of our time."
• Excess escaped defects — the probability that the final deliverable carries a defect in the arm with the tool, minus the same probability without it. A defect is something the person who commissioned the work would send back: a wrong figure, a failing test, an invented citation. When this number is negative, the tool improved quality.
• Break-even defect cost — how many minutes one escaped defect has to eat downstream before the tool stops paying. It is a single number, written in minutes. A tool that saves 18 minutes per task and lets three extra defects through per 100 tasks pays only if one defect costs less than 600 minutes downstream.

The third is the design centre of the paper. It never fixes the price of a defect itself. A wrong figure in a board report and a badly colored slide cannot carry the same price, and naming either as the default would be a declaration rather than a finding. What it reports instead is the price at which the verdict flips. A reader who knows their own situation can follow the decision through without accepting the author's price list. The shape is the incremental cost-effectiveness ratio from health economics, with minutes in place of money and escaped defects in place of health outcomes. The only new thing is pointing that instrument at claims about time saved.

In its third section the paper builds a very plain model of how a person checking work behaves. The conclusion first: the rate at which a worker who checks properly lets defects through has no model performance in it. What a better model buys is not quality but checking time, and less of that than you would think. The paper itself notes in this section that it is a model and that its predictions are labeled as predictions, so what follows is best read as a result derived on top of assumptions.

2.1Hunting a defect works like hunting a lost object

Suppose a draft contains one defect. Write down, as a single number, how many minutes it takes on average to find it, and call that mean κ (kappa). With no particular clue, and looking steadily, the probability of finding the defect within V minutes flattens out as time goes on. The first few minutes return the most, and later minutes return less. This is the exponential detection law that came out of submarine search in the Second World War, and the paper takes Koopman's (1946) expression as it stands.

Two more quantities get attached. Call R the minutes it takes to repair a defect once found, and c the minutes an escaped defect eats downstream. c is usually far larger than R, and more to the point somebody else usually pays it. If the worker picks the checking time that minimizes their own minutes plus the downstream minutes, one expression drops out.

What the threshold means is simple. If the draft's error rate sits above it, checking pays; below it, not checking pays. And nowhere in that expression is there any model performance. There are three things in it and only three: how cheap checking is (κ), how cheap repair is (R), and how expensive a miss is (c).

2.2An improved model pays in checking time instead of quality

Here is the model's sharpest conclusion. For a worker who has the error rate right, the share of defects that reach the final deliverable is whichever is smaller, the error rate q or the threshold q^†. While the error rate sits above the threshold, that share is pinned at the threshold value. A better model lowers the error rate, the worker checks correspondingly less, and the amount going out stays put. Quality starts to improve only once the error rate falls below the threshold and nobody checks at all any more.

So what does the improved model buy? Checking time. Optimal checking time moves with the logarithm of the error rate. Drop the error rate by a factor of ten and about 2.3κ minutes of checking fall away. That one line changes how a leaderboard reads. Going from 90% accuracy to 99% and going from 99% to 99.9% look like entirely different events on a leaderboard, but they buy the same amount of checking time. Leaderboards are linear in accuracy; checking is linear in log error.

2.3Over-trust gains 12 minutes on the clock and loses 33.5 on the ledger

The third proposition sits closest to the paper's title. Suppose the worker rates the model's true error rate as m times better than it is. Whatever the shape of the detection curve, less belief in error means more checking and more belief means less. Over-trust always shortens the clock and raises what escapes. With numbers on it: at κ = 5 minutes, a worker who over-trusts by a factor of ten shows about 12 minutes saved per task on the clock. The same worker's ledger shows 33.5 minutes lost once the extra defects are paid for.

The loss is not symmetric. Over-trusting by a factor of ten costs 6.7κ; under-trusting by the same factor costs 1.4κ. That is nearly a fivefold difference. From this asymmetry the paper draws one line of practical advice. "If you must be wrong about your model, be wrong in the paranoid direction."

2.4How many clean audits before you may stop checking

By the model, review may stop once the error rate drops under the threshold. The trouble is that the error rate is not printed on the model. It has to be estimated from audited drafts. Once you have stacked up k audits with no failure, there is a fixed ceiling you can claim at 95% confidence, and the required number of audits falls out of it. Roughly three divided by the threshold. In the worked example above the threshold was 0.85%, so stopping review on that kind of task takes 352 audits that all came through clean.

The levers this model points at read straight off the same expression. To lower the threshold, make checking cheaper (κ), make repair cheaper (R), or price a miss higher (c). Cheaper checking lowers the threshold, so fewer defects ship and the total minutes including downstream fall as well. One caveat rides along. Clock time falls with it only while the error rate sits well above the threshold; just above the threshold, cheaper checking makes it worth checking more. Drop the caveat and the sentence becomes false.

The paper collects this section into four predictions and tests only two of them in its seventh section: that checking time tracks the logarithm of the error rate, and that over-trust produces faster and worse work. The prediction that escapes are flat in model improvement among workers above the threshold, and the prediction that easily verified output reduces escapes, are both untested. Testing them needs several models, which a small experiment cannot reach.

The real data the paper has in hand is the developer experiment METR published in 2025. Sixteen experienced open-source developers took 246 real issues from their own repositories, with AI use allowed or disallowed at random on each issue. The window was February to June 2025 and the tools were Cursor Pro with Claude 3.5 and 3.7 Sonnet. The published result was that the arm with AI took 19% longer, a figure widely cited at the time. The paper downloaded that repository, reproduced the published number exactly, and then laid other summaries over the same data.

Changing the summary means this. Average the per-issue time ratios on the log scale and issues of wildly different length count evenly; average the times in each arm and divide, and the long issues carry far more weight. The two answer different questions. On top of that sits whether or not you adjust for the fact that the forecast times of the issues in the two arms were uneven to begin with, and the cells multiply to four.

Every sign points the same way, towards "took longer." The intervals come from a jackknife that drops one developer at a time. METR's published interval is [+1.6, +39.0], and the paper reproduced it exactly. The spread across the four cells is not a matter of the sums being done differently. It comes from which summary does the grouping, and from whether the forecast imbalance between the arms gets adjusted for. The 19%, 24% and 34% on the cards at the top of this article are three of these four cells, and those same three are what the paper's abstract leads with.

Why the four cells pull apart is written in the data too. Issue length does not explain it. The interaction between treatment and log forecast is 0.06 with an interval of [−0.11, 0.23], which is effectively zero. What does pull them apart is dispersion. Completion times in the AI arm are spread wider (residual standard deviation 0.66 against 0.60 on the log scale), and that difference alone lifts 18.8% to somewhere near 23%. When there are two ways to group the same data and only one gets printed, the reader never learns that a choice was made.

One more quantity sits separately counted inside the same data: the time spent fixing a pull request after review. On the 230 issues where that time was recorded, the AI arm averaged 14.9 minutes against 9.7 without. The 5.2-minute difference has an interval of [−0.6, +12.1], so it contains zero and the direction cannot be asserted. What it does mark is exactly the column the ledger view pointed at in section 1. Time spent after the clock stops enters no summary at all unless somebody counts it separately.

On METR's data the three summaries all pointed the same way, but the paper also constructs a case where even the sign flips. There the geometric mean ratio comes out at +10.5%, the mean of ratios at +20.9%, and the ratio of means at −10.4%. That is what happens when a tool loses time on common tasks and saves it only on long ones. So the paper recommends the ratio of means, which answers the question a manager is asking, and above all insists that a headline has to name the summary it used.

The row carrying the most weight in that table, though, is not either of the top two. It is the bottom one. "What percentage of issues did AI actually help on" is not a question a mean can answer, and this design cannot recover it at all. Because each issue is performed once in one arm only, the time that same issue would have taken without AI is nowhere in the record. Without the joint distribution of the two times, "the probability that one was shorter than the other" is not pinned to a point but only to a range. These are the Makarov bounds, and on METR's data that range runs from 2.6% to 82.1%. It is not a confidence interval. Multiply the sample by a hundred and the range stays where it is. In the paper's own words, "these data are consistent with AI having helped on most issues, or on almost none."

3.1METR has already stamped that 19% out of date

For this article not to turn into a piece against METR, one fact has to go in alongside. On 24 February 2026, METR announced that it was changing the design of its developer productivity experiment, and put a warning banner on the 2025 study page. The sentence sitting on that page now reads: "These results are out of date. We have released results that are current as of early 2026."

Re-measured with newer tools, the numbers look like this. Among developers who had taken part in the original study and came back, the estimate is 18% faster; among newly recruited developers, 4% faster. The sign has flipped relative to 2025, but the sign is not the point. Both intervals cross zero ([−38, +9] and [−15, +9]). Of the three estimates, the only one whose interval clears zero is the +19% from 2025, and that is the one METR has stamped out of date itself.

The reasons METR gave for the redesign are three. Some developers avoid taking part at all because they do not wish to work without AI. In a survey, 30% to 50% of developers said they had been choosing not to submit some tasks because they did not want to do them without AI. And time-on-task measurement is unreliable for the share of developers who run several AI agents at once. That is a reflection arriving at exactly the same spot as this article's argument. Here too the problem was not the model but the measurement design.

The paper takes METR's side as well. One sentence sums up the posture: "METR chose a defensible estimator and reported the unadjusted 34% itself; the point is not that they chose wrong." It adds that at a scale of sixteen people the inference method itself moves the conclusion. Task-level bootstrap gave 87.8% interval coverage and developer-clustered bootstrap 91.0%; the one that produced 95.0% was the jackknife-t. Power was not generous either. At METR's standard error of 0.080 the minimum detectable effect is about 25%, and seeing a 10% effect would take roughly 1,350 issues.

3.2'Share of tasks handled reliably' inflates the moment you select on it

The problem of choosing a summary shows up in another shape as well. "The share of tasks this model handles reliably," a favorite in model evaluation, is one. Say you take ten drafts per task and mark a task reliable if at least 90% pass. In a world where every task has a true reliability of exactly 0.8, this procedure marks 37.6% of tasks reliable and reports their reliability as 0.93. That is the winner's curse, arising because the drafts used to select are the drafts used to score. Split the drafts in half, select on one half and score on the other, and the number comes back to 0.80. Retire one summary and another moves into the vacancy, carrying a trap of the same family.

In September 2025 OpenAI released GDPval, an evaluation of models on real occupational tasks, and wrote that frontier models are "roughly 100x faster and 100x cheaper than industry experts." That sentence has been among the most-cited productivity figures of the past year. The paper does not call it false. It takes out the caveat that was already sitting inside the same announcement and works out how far the number comes down once the caveat is carried through.

The caveat runs like this. The 100x is measured on model inference time and API pricing, and the human time spent supervising the output, fixing it and fitting it into the work is not in there. OpenAI wrote that into the same text. So the problem is not that the announcement hid anything, but that the caveat fell off somewhere between the announcement and the citation.

Put the caveat back and the room narrows quickly. In the gold-standard task set GDPval released, an expert takes 404 minutes to finish one task, of which 109 minutes go to reviewing the deliverable, 27% of the whole. A GPT-5 draft arrives in about 4.5 minutes, and is rated as good as or better than the expert's own deliverable 39% of the time. Assuming a failed draft is rebuilt from scratch, the figures the GDPval paper puts up are 1.12 times (one attempt) and 1.39 times (repeated attempts). All of the numbers below are values this paper's author transcribed from the GDPval appendix.

What the paper objects to is not the value 1.12 but the assumption that produced it. The repair-cost assumption — that every failed draft is rebuilt from scratch — is doing nearly all of the arithmetic. So it releases that assumption, allows the repair cost to fall anywhere between zero (a trivial fix) and a full rebuild, and the speedup comes out as a range rather than a point.

3.56x is an upper bound, not a measurement. Safer to nail that sentence down in the same place. It does not say GPT-5 ran this fast; it says that as long as review time stays where it is in this workflow, no model however good can beat that. Even if generation were free and finished in zero minutes, it stops at 404 over 109, about 3.7 times. What stands between the announcement's 100x and the 1.1 to 3.6 times of actual deployment is not a difference in model performance but the 109 minutes a person spends reviewing.

The ceiling itself is nothing new, and the paper says as much. It is Amdahl's law — the part of a job a tool cannot speed up bounds the speedup of the whole — out of computer architecture in 1967. In the paper's phrasing, it is "Amdahl's law wearing a lanyard." There is exactly one reason to bring it out again. Accuracy leaderboards do not measure the time spent explaining the task, waiting for the result and verifying the result, and deployed speedup is decided by that time.

And the GDPval arithmetic carries one more quiet assumption: that the review is perfect. Under the detection curve of section 2, finite review always misses something, so the escaped-defect column of GDPval's ledger is still blank. Narrowing the speed side to a range leaves the quality side unmeasured.

4.1Worth it only if a wrong answer can be undone in 48 minutes

There is one place where the answer comes out as a single value rather than a range. In a 2026 experiment with Boston Consulting Group consultants, the researchers prepared a task deliberately placed in a region the model handles badly. Because time and accuracy were both reported for that task, the break-even defect cost defined in section 1 can actually be computed.

Results on a task placed in a region the model handles badly. Time fell by 30% and 18%, and wrong answers rose by 24.5 and 13.9 percentage points. The rightmost column is time saved divided by the extra wrong answers, which is the price at which the verdict flips.

Here is what about 48 minutes means. For AI assistance to pay on this task, catching and undoing one wrong answer later has to cost less than 48 minutes of a consultant's time. But doing the entire task by hand took 37.7 minutes. Clearing one defect, in other words, would have to cost 1.2 to 1.3 times what the whole task costs. "I doubt many clients would sign that." That is the paper's entire comment on it.

Everything so far has been other people's data, recounted. The paper closes by testing two of the section 2 predictions directly. It is not an experiment on people. The setting is an AI reviewer checking AI drafts, and the paper states flatly that it "says nothing directly about people." What this setting does offer, which a human study cannot, is that the experimenter sets the error rate and then tells the reviewer what it is.

5.1A task whose ground truth a machine can settle

The task is the brief from the section 1 thought experiment. Revenue, net income and operating margin for three companies, for the quarter ended 30 June 2026, on one page. Three companies times three line items means exactly nine figures that can be checked. The sources are quarterly reports (10-Q) filed after 1 July 2026, and that timing is the heart of the design: they are documents that entered the world after the models' training data were collected, so no model can have the answers memorized. Ground truth is not set by a person but fixed by machine extraction from XBRL. For all thirty sampled companies, all twelve extracted figures were confirmed to appear on the printed statement.

Drafts come from a small, fast model (claude-haiku-5-5), and review is handled by a larger one (claude-sonnet-5-5). A draft has either no defect or one seeded wrong figure. There are five kinds of wrong figure: a stale period, a year-to-date figure, another company's figure, transposed digits, and the wrong numerator. Scoring is done not by a model but by a deterministic auditor. It counts as a catch only when the corrected value matches the true one.

The cleanest piece of design in this experiment is how it measures effort. The reviewer gets exactly one tool: a lookup that shows one line of one company's statement. Lookups are counted by the tool server. Not estimated, logged. And nine of them cover every line. A price list comes attached. The prompt tells the reviewer outright that a lookup costs 3 minutes and an escaped error costs 10 hours. From those two numbers a break-even error rate falls out: 27 divided by 600, or 4.5%. Above that rate, checking every line pays.

The scale is small. The analyzed run is 540 sessions, each a separate isolated headless session. It ran on the author's own subscription and comes to $9.27 at list price. The hypotheses and analysis code were frozen in a repository commit after a 66-trial pilot. The paper states directly that this is not a pre-registration lodged with an outside registry. One instrument fault surfaced mid-run: three company names held a no-break space, so lookups of those companies failed; it was fixed and the 23 affected trials were re-run.

5.2One lookup already catches one in three

The first experiment fixed the reviewer's lookup budget at 0, 1, 3 and 9 and watched how many seeded defects it caught. This is the place where the exponential detection law assumed in section 2 gets checked against this particular task.

The striking value is the 32% at one lookup. Opening one of nine lines at random would land on the wrong line about 15% of the time, and the reviewer caught twice that. Which means it was not picking at random. It guessed where the suspicious place was and opened that. The reviewer had the ability to check partially. And in the next experiment that ability goes almost entirely unused.

5.3The mean follows the logarithm, individual reviews are all or nothing

The second experiment released the lookup budget and instead told the reviewer the drafter's true error rate, truthfully, at five levels from 2% to 60%, across 120 reviews. On the average the model held up. Each one-unit rise in the log error rate added 2.2 lookups, with an interval of [1.8, 2.5].

Unfold the individual reviews, though, and the shape is nothing like that. 96% of reviews made either zero lookups or nine (interval [91, 99]). There is almost no middle. And the switch happens between error rates of 2% and 5% — exactly where the prompt's own break-even of 4.5% sits. At 2%, three in four opened nothing. At 5%, 54% opened every line. From 10% up, all of them opened all of it. "The mean follows ln q; no review does." That is how the paper sums the result up.

The reviewer carried out the arithmetic its prompt gave it. A lookup costs 3 minutes, all nine cost 27, so below an error rate of 4.5% it is cheaper in expectation not to look. That calculation is correct. But all-or-nothing is optimal only when partial checking is worth nothing, and the experiment just before had shown partial checking to be worth something. One lookup catches one in three. On an exploratory recomputation, one lookup beats both corners by under a minute per draft at 2% and by about four minutes at 5%. The reviewer did not use the partial checking it knew how to do.

5.4Told to be ten times more optimistic, the reviewer opened fewer than two of nine

The third experiment sits closest to the paper's title. The true error rate of the drafts was held at 30%, and only the figure told to the reviewer changed. One arm was told 30%, the truth. The other was told 3%, ten times too optimistic. The drafts were paired, so both arms saw the same ones.

Forty drafts per condition. Told 3% when the truth was 30%, lookups fell by 7.15 on average ([−8.12, −6.03]) and escapes rose by 22.5 points ([10, 35]). Of the forty told 3%, twenty-nine opened not a single one of the nine lines. Among the forty told the truth, none did that.

One distinction has to be drawn here. Did the reviewer wave things through, or did it simply never look? The paper registered a hypothesis separating the two in advance and measured it. The result landed on one side. The quantity that asks whether the reviewer missed more defects at the same level of effort came out at −0.03, with an interval of [−0.15, 0.10] that contains zero. Its judgment, in other words, was intact. Every review that actually opened the line exposing the wrong figure corrected it, twenty times out of twenty. And all sixteen escapes came from reviews that never opened that line. The verdict column of the paper's table reads: no, it ran entirely through effort.

An attempt to block the effect by prompt was built into the design too: give the 3% figure and at the same time add "treat track-record claims as unverified." It barely helped. Escapes went from 22.5% down to 17.5%. The more striking number is the next one. Twenty-six of forty reviews wrote out that the 3% figure was unverified and then checked nothing. Repeating a warning back and acting on it were not the same act. Two earlier studies report the same shape, the paper notes: redacting the misleading information outright restored detection, where telling the model to disregard it did not.

5.521 minutes saved, 135 minutes pushed downstream

Because the prompt wrote down a price list, this result converts into minutes. Over-trust saved 21.45 minutes of checking per draft — the 7.15 fewer lookups multiplied by 3 minutes, which the paper's prose rounds to 21. And it pushed 135 minutes downstream, the 0.225 rise in escapes multiplied by 600 minutes. Divide one by the other and you get the point at which this trade starts to pay.

That comparison is the experiment's conclusion. The reviewer knew the price list its own prompt had written down, and by that price list the over-trusting behavior was plainly a loss. It behaved that way anyway, because it accepted one wrong figure. The stopwatch records 21 minutes saved; the ledger records 135 minutes lost. Both are true. Only one of them gets reported.

Catch rates vary quite a bit by kind of defect. A figure lifted from another company was caught 89% of the time, because the same value then appears twice in the draft and stands out. The other four kinds ran between 44% and 67%. By line item, revenue was caught 74.1% of the time, operating margin 54.2% and net income 50.0%. Some defects are structurally better at hiding.

The limits the paper writes down for itself go in here too. Seeded defects can make detection look better than natural ones would, and the planned comparison against natural errors collapsed because no natural errors turned up to compare against. The 2% condition had no defective drafts at all and the 5% condition had one, so this experiment has nothing to say about escapes at low error rates. And it is one model, one kind of task, powered only for large effects. The full-scale design exists only as a pre-specification in the repository.

One more comparison rides along with the same experiment. It asks whether the arrangement everyone uses now — a cheap model writes the draft and an expensive one reviews it — beats the expensive model working alone. Twenty packets were run paired.

No quality difference at all, and both cost and time went up. No faster and no better, which in the vocabulary of section 1 makes this pipeline dominated.

The reason is plain. Across 140 briefs, the small model's drafts contained no naturally occurring errors at all. The reviewer, told nothing about who wrote the drafts, checked every line of all twenty and found nothing to correct. Going carefully over drafts with nothing to correct cost 2.2 times the money and 3.2 times the clock. As a side effect, the experiment meant to test the winner's curse also failed to run, because there was nothing to select on.

This is the point where an article veers easily into "so drop the review," and the paper takes one more step in precisely the opposite direction. The certification rule of section 2 puts a price on that review. Counted as independent audits — generously, since they cluster on twenty packets — 140 clean briefs bound the error rate below 2.1% at 95% confidence. That is under the 4.5% break-even the prompt wrote down. Certifying that 4.5% takes about 67 clean audits. For the comparison to exist at all, somebody had to count the 140.

Deciding that review can stop and simply not reviewing look identical from outside and are entirely different on the ledger. The first is the result of counting; the second is the result of not counting. The two scenes this paper's small experiment produced are exactly those two. The reviewer told 3% in section 5 skipped without counting and let 22.5% through, and the pipeline in this section earned the conclusion that review could be skipped by counting 140 briefs.

6.1Three of the paper's ten reporting lines

In its eighth section the paper lays out, in ten lines, what a claim about time saved ought to carry. Most of it costs nothing and amounts to bookkeeping. Three of them bear directly on this article's argument.

• Name the summary, and say what it averages over. Ratio of means or geometric mean ratio, and which tasks and which workers the average runs across.
• Do not use self-reports in place of observed times. The developers in METR's 2025 experiment misjudged their own speedup by about 40 points. Afterwards they said they had been 20% faster; the figure measured in the February–June 2025 records was 19% longer.
• Attach a count of clean audits to any "no review needed" claim. Roughly three divided by the threshold, on that kind of task. At a threshold of 4.5% that is about 67; at 0.85%, 352.

The paper holds its own work to the same standard. The list of limits runs long. "It reports no new human-subjects data." The model in its third section assumes one defect per draft, assumes no false alarms, and assumes a risk-neutral worker. The Makarov bounds laid over METR's data are plug-in values with no sampling error in them. And the most candid line is this one: "The reviewer and the drafters are Claude models, and Claude helped prepare this paper."

What remains is the blank column this article opened with. That the same 246 issues hand back a different number for every summary you pick, that 3.6x is a ceiling rather than a measurement, that an AI reviewer with intact judgment opened fewer than two of nine lines — all of it comes out of one fact. Nobody has written down the exchange rate between saving a minute and letting one wrong figure through. Without that rate, time saved is a reading on a dial rather than an amount of money. And section 2 adds one thing that weighs more. One end of that exchange rate, the volume of defects left in the deliverable, does not move when you change the model.

AI-Ready Data asks whether a dataset is in a state where it can be used right now. This paper stands another column next to that question. When it was not in that state, how many minutes does that one record eat downstream? Without that number the checking threshold of section 2 cannot be set, and without a threshold "good enough" is a feeling rather than a calculation. Data readiness scores come out as percentages with no price tag in minutes attached, and that is the gap as it stands.

7.1The lever on quality is cheaper checking, not a better model

For anyone working on data quality, the most useful part of the paper is the flat stretch in section 2. Swapping the model alone will not budge how many defects are left in the deliverable, because model performance never enters that expression. Of the three values that can move, the downstream cost and the repair cost are mostly set outside the organization. The one an organization actually gets its hands on is the third: the average time it takes to find one defect. Make checking cheaper and the threshold comes down, and fewer defects go out with it. Strip off the caveat section 2 attached, though, and the sentence turns false. The stopwatch reading comes down too only where the draft's error rate is still far above the threshold; near the threshold, the cheaper checking gets, the more of it is worth doing.

7.2Three things you can check today

None of the three needs a new tool. First, write down in one line whether the "N% faster" you are quoting is a ratio of means, a geometric mean ratio or a mean of ratios. That those three can come out with opposite signs is what section 3 showed. Second, put a single number on how many minutes one defect eats downstream. That number is what turns "we can review less now" into a calculation. Third, before reducing review, count how many audits have come through clean. At a threshold of 5% that is about 60; at 1%, about 300.

These three are not a checklist for whether you are doing well. They are a way of asking whether the question you are currently asking is one that can be answered. That distinction is what the Makarov bounds in section 3 put on display. No amount of accumulated data answers a question the design cannot answer.

7.3Where quality gets a price tag in minutes

Every place in this paper where a conclusion flipped, what flipped it was the price list and not the denominator. The market for choosing models is already crowded. The place where you write down in minutes what a wrong piece of data costs, set the checking threshold from that figure, and prove by counting when review can be reduced, is still empty. That is where Pebblous stands in working on data quality. It is the same reason DataClinic, when it diagnoses a dataset, puts the conditions a score came out under alongside the score. Without the conditions a score does not reproduce, and without a price list quality does not turn into a decision.

Sources for the figures in this article, grouped by kind. Number 1 is the paper this article is about; 2 through 8 are the primary literature it recounts or borrows from. Numbers 9 through 14 are notices the institutions issued themselves and public repositories. From 15 on are Pebblous blog articles that took up the same question somewhere else.

Academic literature

• 1.Solomon Shalom Lijo, "Counting the Bandages: On measuring the time that AI saves", arXiv:2610.35942v1, 23 October 2026. arxiv.org/abs/2610.35942 · Zenodo mirror doi:10.5281/zenodo.23557773. Every section of this article follows this paper.
• 2.Joel Becker, Nate Rush, Elizabeth Barnes, David Rein, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity", arXiv:2507.09089v2, 2025. arxiv.org/abs/2507.09089 The 246 issues, 16 developers and 19% in section 3, and the 40-point self-assessment gap in section 6, come from here.
• 3.Tejal Patwardhan et al., "GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks", arXiv:2510.04374, 2025. arxiv.org/abs/2510.04374 The 404 minutes, 109 minutes, 4.5 minutes, 39% and 0.61 in section 4 are values paper 1 transcribed from this one's appendix.
• 4.Fabrizio Dell'Acqua et al., "Navigating the Jagged Technological Frontier", Organization Science 37(2), 403–423, 2026. The accuracy and time figures in section 4.1, and the break-even defect cost of about 48 minutes, come from here.
• 5.Bernard O. Koopman, "Search and Screening", 1946/1957. The exponential detection law in section 2 is this work's expression.
• 6.G. D. Makarov (1981) · Charles F. Manski, "Partial Identification of Probability Distributions" (2003). The Makarov bounds and partial identification in section 3 come from here.
• 7.Gene M. Amdahl, "Validity of the Single Processor Approach to Achieving Large Scale Computing Capabilities", AFIPS, 1967. The ceiling proposition in section 4 is a variant of this law.
• 8.Frank B. Gilbreth Jr. & Ernestine Gilbreth Carey, Cheaper by the Dozen, 1948. The two razors and the bandage in section 1, and the paper's title.

Institutional notices and public repositories

• 9.METR, "We are Changing our Developer Productivity Experiment Design", 24 February 2026. metr.org/blog/2026-02-24-uplift-update/ The re-measured values and the three stated reasons for the redesign in section 3.1 are here.
• 10.METR, the original 2025 study blog post (10 July 2025). metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/ The "These results are out of date" banner quoted in section 3.1 sits on this page.
• 11.OpenAI, "Measuring the performance of our models on real-world tasks" (the GDPval announcement), September 2025. openai.com/index/gdpval/ The 100x of section 4, and the caveat that it is measured on model inference time and API pricing, sit in the same text.
• 12.github.com/solomonshalom/tsb-bench — the repository for paper 1. It holds the pre-specification commit, the deviation log and the code of the withdrawn April 2026 version. As checked on 30 October 2026 it has zero stars, forks and issues.
• 13.github.com/METR/Measuring-Early-2025-AI-on-Exp-OSS-Devs — METR's public data, used for the recount in section 3 (commit 7ff2d76).
• 14.Solomon Shalom Lijo, "TSB", SSRN 6682338, April 2026. The author's own proposal, which section 1 records him withdrawing.

Neighboring Pebblous articles

• 15.OpenAI's math papers carry a review field, and what it says is unchecked — another case where one source yielded three different rates. That article was about what counts as one item; this one is about a quantity that no amount of counting pins down.
• 16.Torsten Slok's Zero AI Payoff, Absent or Just Invisible? — the same question seen from the macro-indicator side.
• 17.Rippling's Employee Scorecard Puts Rework Rate Next to AI Spend — a case of quality placed next to cost. The exchange rate this article finds missing belongs there.
• 18.METR Sees Inside Anthropic, and Anthropic Decides What Gets Hidden — on METR's evaluation work, for readers meeting the name here.
• 19.AfterQuery Turns How Doctors and Lawyers Decide into Training Data — material that unsettles the "review is perfect" assumption section 4 points at.