Executive Summary

In September 2026, an evaluation startup called Specific Labs published a benchmark named Real-SWE. The way its tasks were built departs from the usual coding benchmark. Nobody scraped issues off public GitHub repositories. Instead, each task came with a private production codebase licensed from a different company. The page puts it this way: "Each task comes from a private production codebase that we licensed from a real-world company. These are problems their engineers work on, with all the context and complexity that comes with an existing product."

How many companies handed over code is not disclosed. The page describes only the kind of product: "a Luma/Partiful competitor with 200K+ users and a top 100 App Store ranking," "a consumer fintech platform processing 100K+ bank statements," and "enterprise AI sales platforms supporting complex business workflows." The task environment does not stop at a repository either. Each task exposes only the services its own workflow needs, and that list includes an AWS emulator, Docker, Kubernetes, GitHub, three different databases, Slack, Intercom and Google Drive. It looks close to what an in-house engineer opens on a Monday morning.

The selection rule is on the page as well. Codebases went through "a rigorous screening process, focusing on real companies with substantial usage, strong engineering teams, and demanding production workloads." The sentence that follows sums up the whole design: "We prioritize code written to meet an actual user or business need over code written solely to create a benchmark task." Tasks had to qualify on the same terms. Every one of them had a direct relationship to spend and had been assigned to an engineer earning a salary. The builders draw their own conclusion in the same section: "Does AI code match the bar of a real-world enterprise? Our results show us that we're far from that reality."

1.1The Table That 640 Rollouts Produced

The scale is ten tasks, eight models, eight attempts per task, which multiplies out to 640 rollouts. A rollout is one attempt at solving a single task end to end. The score is the resolution rate, which the page defines as "equivalent to pass@1, averaged over eight independent runs per task." No human grades anything. All tasks are in Harbor format and verifiers are injected at grading time, and those verifiers are "inspired by existing test suites in the codebase or use those tests verbatim." The tests a company already wrote to protect its own code become the invigilator.

One more design choice deserves attention. Every model ran inside the coding tool built by the company that made it. That execution layer, the harness, wraps the model and supplies file reads, command execution, context management and verification. The page gives the reason: "We use native harnesses to reflect how enterprise engineers work in practice, evaluating model-and-harness combinations rather than models in isolation." That single sentence becomes the premise of section 3.

The result table has eight rows. Fable 5.1 on Claude Code leads at 38.8%, and GPT-5.6 Sol on Codex CLI sits last at 16.2%. The page also publishes an estimated cost per rollout, so both columns appear side by side below.

The eight values average out to 26.9%. The page never prints that number, but a short check confirms it. Every bar sits on a multiple of 1.25%, which means 80 rollouts per model, and the per-task passes add up to 172 out of 640. That ratio is 26.875%, the same as the arithmetic mean of the eight rows. On average, then, three attempts in four failed.

1.2Weeks of Work, Handed Over in One Go

It is tempting to translate 38.8% into "coding agents are not ready for production work." Knowing what the tasks actually asked for, that translation does not survive. Janak Sunil, a co-founder, came into the Hacker News thread directly to explain the low numbers, and in the same thread identified the company as a Y Combinator Fall 2025 startup. "the reason for lower success rates is that we gave models ambitious tasks that real engineers worked on for weeks." The benchmark page does not carry that line. It exists only in a comment, and it is decisive for reading the score.

A description of difficulty is easy to nod at and hard to feel, so here is one of the actual instructions. Below is part of the tax task, which finished at a 3.1% resolution rate: two passes out of 64 attempts.

Billing reopens on Monday and every invoice this service issues is coming out untaxed. Each business on the platform settles its tax a different way: some maintain a rate themselves, some want each invoice priced against the buyer's destination by our tax authority provider, and some collect nothing at all, while a customer we hold an exemption for is charged nothing whichever way its business is configured. (…) an address the authority refuses must be reported without stopping the invoice. (…) once an invoice is settled the sale is filed back to the authority under that invoice's number so the returns reconcile. Invoices between European parties show both sides' VAT registrations. The authority and ledger are available at TAX_JAR_URL, PROD_TAX_JAR_URL and INFLUX_URL.

Six or seven conditions stack on top of each other, and dropping any one of them fails the verifier. The last line repays attention. The instruction hands over three environment variables that reach the external tax authority and the ledger. It withholds the rest: which file in the codebase these rules belong in, and where the existing invoice flow has to be touched. Specific Labs sets exactly that boundary as a design rule: "Any behavior required by the verifier must be stated or reasonably discoverable." So the failed rollouts did not lose to information nobody could have had. They lost by not looking, or by failing to carry what they found all the way through. Failure on this exam happens outside the code itself.

468 of the 640 rollouts failed. Specific Labs sorted those failures into five kinds by looking only at what the submission did, and applied the same taxonomy to all eight models. The scheme is not their invention: the page says the grouping follows DeepSWE. The definitions in the table below are the sentences printed on that page.

The line that matters most in this table runs between the top three rows and the bottom two. Regressions and wrong files are mistakes of craft, of handling code badly, and together they come to 5.1%. The other 94.9% have a different character. A missed requirement means the instruction was never read to the end. An integration error means the neighboring system's expectations were never learned. An unverified assumption means something knowable by opening the codebase went unchecked. All three hang on reading a system somebody else built.

One commenter in the Hacker News thread has already pulled that lever in production. Working on a codebase "that evolved over 15 years," the user ttul described building a "code atlas" that "provides the LLM with a semantically queryable map of how things connect and relate." The raw material was not only code. An agent read the Helm charts and the underlying repositories to see how the parts of the system talk to each other, with the internal wiki, the issue tracker and Slack history attached through MCP servers "so it could dig around to its heart's content as would a human developer trying to figure out the same problem." The agent drafted the atlas, a human corrected it, and the finished map went back out as a tool. By that account it "dramatically" cut the time models spend reading code and pulled in nuances they would otherwise have missed. No measurement accompanies the report, so it stands as a single case. The direction it points, though, lands squarely on the 94.9%.

2.1Each Model Fails Its Own Way

The per-model breakdown is more interesting than the overall split. The page prints each model's failure types as a share of that model's own failed runs, and the profiles differ sharply. Grok 4.6 put 41 of its 61 failures into missed requirements, 67.2% piled into one category. Gemini 3.8 Flash went the other way, with 27 of 55 failures classed as integration errors at 49.1% and missed requirements at 29.1%. Last-placed GPT-5.6 Sol leads all eight on unverified assumptions at 43.3%.

A user in the thread recognized that profile from daily use. "Sol failing mostly on 'unverified assumptions' and rarely hitting 'integration errors' seems about right to me," wrote didgeoridoo, who rated the model highly at architecting the right implementation "but only if you are extremely specific and provide tight guidelines and guardrails." The table agrees: Sol's integration-error share is 16.4%, the second lowest of the eight. Benchmark numbers and hands-on intuition rarely point the same way this cleanly.

For a buyer the lesson is concrete. Among models that are all "good at coding," which one holds up in your organization depends on whether what you lack is specifications or system documentation. A single average score cannot settle that.

2.2More Time on the Problem Changed Nothing

The hope that another hour would crack these tasks also meets an answer in the data. Of the 98 rollouts that finished in under ten minutes, 70 failed, a failure rate of 71.4%. Of the 542 that ran ten minutes or longer, 398 failed, or 73.4%. The gap is almost nothing. The denominators have to travel with those rates, though. The short group is only 15% of the total, and some of the runs that ended early were easy tasks to begin with. The safe sentence is the narrow one: holding on longer did not bring the failure rate down.

2.3Thin Instructions Are Not What Makes This Hard

Specific Labs measured the source of the difficulty itself. The median instruction runs 1,742 characters, shorter than FrontierCode at 2,056 and DeepSWE at 1,975, longer than Terminal-Bench 3 at 1,584. In the page's own words, the prompts end up "slightly underspecified, about par with DeepSWE and Terminal Bench, but specific enough to not omit instructions." Then comes the other measurement: the reference solution edits a median of 11 files, against 6 for the same two comparisons.

Instructions of ordinary length, twice the surface to touch. The difficulty comes from how far a change has to reach, not from how little the prompt says. That single line locks into the failure breakdown above. In a change that spans eleven places, dropping one requirement or breaking one contract with a neighboring system is close to the default outcome. One caveat rides along with the length comparison. A footnote states that the sample used for it was "Real-SWE's eight repository-backed sample tasks," a different count from the ten tasks behind the leaderboard and the analysis.

Everything above is what the benchmark measured. The short version: this data cannot hold up an ordering of eight rows. That is not a complaint about the builders, who never asked anyone to read it that way and in fact wrote the opposite on the page. The trouble starts when a table like this gets cited, because citation almost always compresses it to one line about which model came first.

3.1The Intervals Are Drawn, the Numbers Are Not Printed

The caption under the result table states that "95% confidence intervals are shown," and a thin whisker does sit beside each bar. The numbers behind those intervals appear nowhere on the page. So this report pulled the coordinates of the whisker elements out of the page's raw HTML and converted them into the same scale as the bars. All eight point estimates land within 0.05 percentage points of their bar positions, which means the intervals below are the values the builders drew and left unlabeled.

With the intervals drawn in, the table changes shape. First-placed Fable (32.1–45.4) overlaps second-placed Astra (27.1–40.4) widely, and it still overlaps fourth-placed GLM (19.0–38.5). The two models tied at fifth and seventh-placed Kimi overlap each other as well. Statistically, this data separates roughly the top group from the bottom row, while the order from first to fourth would move if a single task were swapped out.

3.2Ten Tasks Are Holding Up the Ranking

The intervals are that wide for a simple reason. All eight rows of the leaderboard rest on ten tasks. The builders introduce this analysis as "a small sample of tasks from our benchmark," yet adding up the per-task passes reproduces the leaderboard values exactly. The published ranking came out of these ten. Here is how they went.

The page's own subheads read "6 of 10 tasks have resolution rates below 15%" and "No model solves every task." The bottom row went unsolved across all 64 attempts by the eight models. Drop that one task and every score in the table rises and the gaps rearrange. This is what the wide confidence intervals of the previous section look like in raw form.

The way models and tasks interlock is more striking still. Grok 4.6, tied for fifth overall, went 8 for 8 on "Customer identity migration," alone among the eight. On that same task first-placed Fable passed 3 times and second-placed Astra once. In the other direction, Kimi K3 failed all eight attempts at the second-easiest task on the board, where the other models passed somewhere between four and eight times. Which model wins matters less than which model wins on which task. Importing the ranking without knowing which of those ten your own work resembles imports an illusion rather than information.

3.3Each Row Is a Model Plus a Tool, Not a Model

As section 1 noted, every row of Real-SWE carries a model name and a harness name together. A paper posted to arXiv this May takes that problem head-on, and its title is the argument: "Stop Comparing LLM Agents Without Disclosing the Harness." Its status deserves a precise label first. The abstract opens with "This position paper argues," so it is a position paper, with a controlled factorial experiment attached in section 4. Presenting it as an experimental paper would overstate it.

The contrast the paper draws from public leaderboards is sharp. Measured through one standardized scaffold on a public SWE-bench Pro leaderboard, the six leading frontier models "span only 4.9 percentage points" (41.0% to 45.9%). Hold one model fixed and change only the harness, and that same model moves from 45.9% to 55.4%, a gap of 9.5 points. In the paper's words, "the within-model harness range thus exceeds the within-harness model range by roughly twofold." Adding a single search subagent on top of otherwise identical infrastructure flipped the ordering between two models, which the paper reports as well.

The controlled experiment the authors ran themselves is plainer. They took three models that sat tightly clustered on a coding leaderboard, set up three harness configurations labeled minimal, improved and full, and ran a difficulty-stratified 100-task subset of SWE-bench Verified. It is a small experiment, two runs per cell, and that has to be kept in view. The direction was unambiguous anyway. Harness-induced variance came to 7.80 times model-induced variance, and across nine crossed model-pair and harness-pair comparisons the ranking reversed in six. The model that led under the minimal harness was not the leader under the full one.

The same thread showed that nobody outside can audit the harness either. One commenter claimed the benchmark had used a dead tool: "There's no such things as gemini cli these days." Checking the source, that charge does not hold. Google's announcement dates the transition to 19 May 2026, sets 18 June as the day requests stop being served for consumer plans, and states in the same document that organizations with Gemini Code Assist Standard and Enterprise licenses, along with users on paid API keys, keep their access. It would be false to say the benchmark ran on a discontinued harness. The real problem is elsewhere. Which build ran, under which settings, cannot be checked from outside. Even the fact that every model ran on "high" reasoning rests on a single line from a co-founder in the thread.

3.4Spending More Did Not Score Higher

That is why section 1 put the cost column next to the scores. The builders' own subhead says it: "Higher cost does not guarantee a higher resolution rate." First-placed Fable 5.1 costs $6.96 per rollout and third-placed Gemini 3.8 Flash costs $2.50. Paying 2.8 times more buys 7.6 percentage points. Fourth place makes the point harder. GLM 5.3 costs $5.12 for 28.8%, while the cheaper Gemini 3.8 Flash returns 31.2%. A combination that costs more and does worse is right there in the table.

Tokens tell the same story. GPT-6 Astra averages 24k output tokens per rollout and scores 33.8%; GLM 5.3 spends 117k and stops at 28.8%. Nearly five times the burn for a lower score. The sharpest scene for anyone forecasting a budget belongs to Grok 4.6. The two tasks it burned hardest on were one it passed once in eight attempts, at 261k output tokens per rollout, and one it never passed at all, at 315k. Its other eight tasks took between 3k and 20k. The heaviest burn lands on the problems a model cannot solve, which is the concrete shape of cost risk inside a company. One caveat: the page flags incomplete usage data for Grok 4.6 and Kimi K3 and warns that actual cost may be higher, so both figures read as floors.

A single score still leaves things out. One commenter put it directly: "this suggests Fable is better than Astra, but in practice Astra is waaaaaay faster (like 5x; it's not even close), and also waaaay less annoying to talk to." Another answered that the benchmark is "not measuring speed nor annoyance. It's there on the page," which is half right. Real-SWE does measure cost and wall-clock minutes per rollout. What it leaves unmeasured is conversational quality and how often a human has to step in. The co-founder pushed back with "for long horizon engineering tasks would speed still matter?" and got the reply: "Do you want your results in a week or a day?"

Two more things the table cannot separate. The first is a human baseline. No score exists for human engineers on these same ten tasks. A commenter named barbegal put a finger on it: "Without a human to benchmark against it's really tough to gauge how good these models are vs how good the task definitions and existing codebases are." Deciding whether 38.8% is low or high requires something to compare against, and the only comparison on offer is the sentence about engineers who worked on these problems for weeks. The second is the person driving. In another commenter's phrasing, "it's the wizard not the wand at this point given a decent model. These benchmarks don't have the wizard." The gap between people who get results and people who flounder with the same tools on the same codebase never enters the score.

Respecting the limits of the previous section leaves one question open. Is there evidence for the claim that agents which do well on public repositories fall apart on company code? Real-SWE on its own cannot support that sentence, because the benchmark contains no matching measurement of the same models on public code. Dragging a 70-something score from another benchmark to sit next to 38.8% is worse still. It welds together two numbers whose task definitions, grading and harnesses all differ, purely because the drop looks dramatic.

Usable evidence exists elsewhere. Scale AI's SWE-bench Pro keeps a public split and a private split inside the same benchmark and measures the same models on both. Its 1,865 tasks across 41 repositories divide three ways. The 731 public instances come exclusively from strong-copyleft repositories such as GPL, the 276 private instances come from 18 private, proprietary codebases from startups, and the remaining 858 stay held out for internal analysis. The operator explains the copyleft choice as "a legal deterrent against the code's inclusion in model training data," a design that tries to block contamination with a license.

Seven models appear under the same name on both leaderboards, so seven pairs can be matched. The first five carry the same harness marker (mini-swe-agent) on both sides, which satisfies the condition section 3 demanded: compare with the harness held fixed.

All seven pairs point downward, without exception. The size of the drop runs from 4.8 to 15.7 percentage points. Honesty requires reading the intervals here too. In five of the seven pairs the public interval and the private interval do not overlap, which is hard to write off as chance; in the other two they do overlap, so neither case settles anything on its own. The weight of the evidence therefore rests on seven pairs moving the same way, not on the size of any one of them.

One trap waits for anyone citing these pages. Both carry an older figure caption reporting that Claude Opus 4.1 falls from 22.7% to 17.8% and GPT-5 from 23.1% to 14.9%. The private-side values do survive in the current table as 17.75% and 14.86%, but the public table no longer lists those two models at all. The caption is a snapshot from the paper; the table is a later revision. Cite the caption as caption and the table as table. Mixing them manufactures a comparison that does not exist.

So far this report has read the exam results. Now for where the exam paper came from. The company behind the benchmark keeps a second page on the same domain, at /company-data, and the browser tab announces the offer: "Your Company Data Could Be Worth $100K–$1M."

The body is short. "Specific works with companies that own valuable records of how real work gets done. This data can help train and evaluate AI models. Share a few company details for a private fit review." Then comes the form. It collects first and last name, a work email that has to match the company website, company name and website, "software from the company you have access to and can license," the tools that data lives in, peak company size, current company size, years in operation, a phone number, and a fax number.

The interesting part is what the page does not say. How compensation gets calculated, which clause protects confidentiality, how long the data is retained, whether it can be reused or resold elsewhere, whether a company can pull out later: none of it appears. The one commitment stated is a single line: "By sending your details, you agree to our Privacy Policy. No files or datasets are uploaded through this form."

So this report opened that privacy policy. It runs to less than half a page, and all of it concerns a LinkedIn advertising-measurement integration: that a hashed work email, a LinkedIn click identifier and the submission's time and event ID go to LinkedIn to "measure and optimize advertising"; that "a hashed email can still be matched to you; it is not anonymous"; that pending payloads are kept "for approximately 24 hours for delivery retries." Not one sentence covers how licensed code is handled. The $100K–$1M figure comes first, and what a company gives up in exchange appears nowhere public. To keep this precise: the point is not that a contract lacks those clauses, but that none of them are disclosed at the enquiry stage.

5.1One Sentence Serves Both the Benchmark and the Sales Pitch

Back on the benchmark page, one sentence sits there with nothing behind it: "99% of tokens in real-world enterprises are hidden away from the frontier models." No source, no method of calculation. That sentence does two jobs at once. It explains why a benchmark built from public repositories cannot be trusted, and it explains why your company's code carries a price. The justification for the evaluation and the sales logic for the acquisition ride on the same line.

That arrangement knocks down one defense offered in the Hacker News thread. A commenter wrote: "As long as the ones offering the benchmark aren't trying to sell you something and have no affiliation with one of the companies on the page I'll take it as opposed to having the benchmark rendered useless in 3 months when the next models drop." The condition is stated precisely, and in this case the condition fails. This company has something to sell, and something to buy.

Another commenter doubted the acquisition channel itself, describing ads that offer to buy the codebase of a real product or startup even long after it has shut down, with the "offer usually price per lines of code," and suggesting the pool might therefore hold abandoned small and mid-sized codebases, or even fake ones. That claim is an unverified statement from a forum user, and this report could not locate a primary source for any such advertisement, so nothing here rests on it. The co-founder answered in one line: "we manually vet all codebases and companies."

A deeper doubt than sourcing came from traceroute66: "any serious real-world company with a proprietary codebase worth looking at would not be handing out the crown jewels to a third party. License or not." The follow-up argument cuts harder. Companies that keep their code off public repositories tend to be exactly the companies with the strictest rules on language models, and those companies are unlikely to sign off on feeding an entire codebase to a set of LLMs for benchmarking. If that holds, the sample tilts toward code that was safe to hand over. The builders answer on two fronts. The page carries the selection criteria quoted in section 1, and the co-founder commented that "we reached out to companies that were willing to license their codebases. every codebase we used had real users, one of them had 200k+ users and is currently top 100 on the app store." Neither reply is evidence that the sample is unbiased. The doubt cannot be settled from outside while the company names stay sealed, and that is part of the price a private design pays.

Measuring on private code is, for now, close to the most honest way to dodge contamination. The moment it is private, however, nobody outside can check the measurement. The sharpest criticism in the Hacker News thread landed on exactly that. "So TL;DR benchmarking in a completely non-reproducible manner? 'Model X performed great, but we can't possibly tell you anything about the code it was looking at apart from it was a large code base from an unknown company'. So basically pinky-promise benchmarking?"

The rebuttal was honest in its own way: "If it builds up history and perceived reliability, this type of thing can be valuable. You're giving up transparency for it being harder to game." Both sides of the trade sit in those two sentences. The open question is how much was given up.

6.1The Methodology Lives in the Comments, Not on the Page

The section of the benchmark page that describes the evaluation setup is three sentences long: each agent ran in an isolated sandbox, all tasks are in Harbor format, and verifiers are injected at grading time from the codebase's existing tests. Everything else a reader would need to reproduce the work is scattered through the discussion thread. That every model ran on "high" reasoning, that the codebases were vetted by hand, that the tasks represented weeks of work for human engineers, and the central contamination defense as well.

That defense reads: "all the codebases were written pre-2023, so pre when AI got good at coding." As a defense it holds up halfway. "Written before 2023" is a different proposition from "absent from the training data." Older code has had more time, not less, to end up in past crawls and forks. And this claim, too, lives in a comment. Anyone citing it has to mark it for what it is: a claim by the builders, unverified.

6.2Private Does Not Mean Clean

The most practical warning in the thread came from someone who runs a benchmark of their own. "My intuition is that many of the better & bigger 'private' code bases, at least in terms of claude code and codex... are not in fact private at this point. One lesson of running botsbench.com, in a slightly different domain, is to measure for model contamination every time."

That warning attaches to this benchmark with unusual precision. Every model ran inside its own maker's native harness (section 3.3), which means the licensed private code passed through eight vendors' execution environments in order to be graded. The thread asked the question directly: "Does this mean they ended up sharing those private codebases with OAI, Anthropic etc?" The co-founder answered "nope, these were private codebases," and the next comment underneath read: "Private codebases uploaded to the AI labs." Another user put it plainly: "Once we send code & prompts to the providers, it can no longer be considered private." Exposure is not the same as training. Still, ARC Prize, which the next subsection covers, gave this exact situation its own name. A set that was never published but has been exposed to commercial APIs is not called private there; it is called semi-private, not because training has been confirmed but because the leakage risk cannot be ruled out. In that vocabulary, Real-SWE's ten tasks were already semi-private the moment grading finished.

Ways to measure contamination were surveyed here back in June, and the method this commenter uses is among the simplest of them. It has two halves, and both transfer directly to an in-house pilot. The first is cutting off the material: run the same problem "without the DB nor internet access," and "if they still get some right, that's a problem." The second is reading the intermediate traces, checking the mid-run database queries for knowledge the model should not have had given the steps that came before.

The same thread also shows the method to avoid when claiming contamination has been found. One user prompted a chatbot with an internal idiom from a big tech company, got back code in that company's house style, and asked where it had learned the pattern; the model replied "from my learned understanding of CompanyX's internal coding conventions." The correction arrived immediately: "You cannot introspect model training by prompting the model. Whatever answer it hallucinated on your query 'where do you know that from' has almost certainly very low bearing on reality." The original poster agreed. Contamination gets measured against a control, not sensed.

6.3Running a Private Evaluation Responsibly Has Been Done Before

"It's private, so it can't be checked" does not work as an excuse, because somebody has already done the work. ARC Prize splits its tasks four ways: 400 public training, 400 public evaluation, 100 semi-private evaluation, 100 fully private evaluation. The naming starts out honest. Of the third set the report says it is called "semi-private" because "while it hasn't been publicly released, it has been exposed to commercial APIs and thus suffers from a risk of leakage." An exposed set does not get to be called private. One more piece belongs here. The 100 fully private tasks "were originally tested by two people who scored 97% and 98%, and, together, solved all 100%." The baseline section 3.4 found missing was captured here while the set was being built.

The sharper lesson is a numeric threshold for overfitting: "We consider scores overfit if semi-private and public evaluation set scores exceed ±10% absolute difference." Applying it requires publishing both scores side by side, and the report's table does carry both for every entrant. Held against its own table, one entry at 47.5% and 62.8% falls outside the line. Publishing a threshold means showing where your own results cross it.

The report even records that its own set is wearing out. Those 100 tasks were used across all four competitions, "and, as a result, on the order of 10,000 private evaluation set scores have been reported to participants so far. This presents a significant risk of overfitting, since each score has the potential to extract a tiny but non-zero amount of information about the content of the hidden tasks." The operators put in writing that a private set erodes with use.

However well anyone learns to read somebody else's leaderboard, the number that decides anything is the one measured on your own code. What follows turns the evidence from the sections above into a design for an in-house pilot. Seven items, every one of them backed by something argued earlier in this report.

1. Fix the harness and name it. The same model moves by as much as 9.5 percentage points depending on its execution environment, and adding one search tool flips rankings. When a vendor hands over a score, the first question is which harness produced it, at which reasoning effort.
2. Take tasks from real work. A measurement means something only if the task is one a human engineer spent days or weeks on. And grade with the tests already in that codebase, not by human eye.
3. Run each task several times and publish the interval. With few tasks, one sample flips the order. A score with no confidence interval beside it is a chart with no error bars.
4. Classify the failures by type. If missed requirements, integration errors and unverified assumptions dominate, the answer is context supply rather than a bigger model. Writing down the specification, the tests, the contracts with adjacent systems and the house conventions comes first.
5. Put cost and tokens in the same table. The top score was rarely the best value, and one model burned hardest on problems it could not solve. Budget accidents happen in that tail, not at the mean.
6. Measure contamination every time. Cut off access to the material and put the same problem back. Getting it right anyway is the signal. Check the intermediate traces too, for knowledge the earlier steps could not have supplied.
7. Read the contract before any data leaves. Check how compensation is calculated, what covers confidentiality, how long data is retained, whether it can be reused or resold, and how to withdraw. None of that is disclosed at the enquiry-form stage.

Someone in the same thread already runs this design. The recipe is plain: "split the git history to just before the change, sandbox the agent with everything they will need at that commit and lightly modify rules so they don't go searching outside the box. Then they get the same prompt (usually the ticket that began the work) and are graded against the accepted PR." The biggest cost was not compute but finding the examples, because "in my real dev flow it's rarely ticket -> PR -> merge, things bounce around a lot more." The recommendation stands regardless. With the rig in place, a new model can be lined up against earlier ones on their own codebase within days, and their sense of "x model is good at y and bad at z" moved "from vibes to a better heuristic."

Two traps surfaced in the same exchange. The first is the grader. Handing grading to a model is easy to build, but someone has to ask whether that grader is calibrated, whether it is "biasing towards same-model agents," and how many results it takes to rise "above the noise floor." The observation that "models absolutely prefer their own outputs to others" followed immediately. Real-SWE's decision to use the codebase's existing tests as verifiers sidesteps this trap. The second is a model noticing it is under test. The same commenter keeps "a specific audit script that goes through thinking traces to see if the model realizes it's being tested." For an in-house pilot, those two attach to item 2 on grading and item 6 on contamination.

If running the whole list looks expensive, reorder it. Items 3 and 4 alone hand most organizations a number more useful to them than any vendor score they are looking at today. Ten tasks at eight runs each is eighty runs, and that fits inside a single weekend.

Pebblous diagnoses data and issues quality reports on it. That makes this benchmark hard to read as somebody else's industry. The shape of the breakdown, and the place where a price gets attached, both match what we look at every day.

8.1The 94.9% Translates into a Quality Problem

Put section 2's breakdown into the language of data quality and it reads like this. A missed requirement means the specification does not live next to the code. An integration error means the contract with the adjacent system lives in someone's head and not in a document. An unverified assumption means the codebase never tells anyone where to go and check. The shortfall is not in the craft of writing code but in the context that code has to fit into. Those three lines are the concrete form of the sentence "our codebase is not AI-Ready yet."

That translation redirects the buying decision. Evaluations usually open with which model to purchase, and the lever this table points at is not the model. It is putting specifications, tests and inter-system contracts next to the code, which applies to an in-house codebase exactly what we have been saying about training data for years.

8.2The Market Priced Evaluation Data First

Pebblous argues that the bottleneck is data rather than models, and DataClinic clears that bottleneck by turning data itself into something diagnosable and sellable. Real-SWE is the case where the same logic showed up first, and in dollars, on the evaluation side. When a benchmark company quotes $100K–$1M for one enterprise codebase, the market has answered that competitiveness in evaluation lies in the route to the data rather than in the algorithm. What that company put a price on is "records of how real work gets done." We have been calling the same object by a different name for several years.

8.3Two Decisions Left with the Customer

The first belongs to the buyer. Did the score attached to a candidate agent come from somebody else's open source, or from data resembling your own company's code? That one question is already a checklist. The second belongs to the seller. In-house code, logs and documents have started to be priced as evaluation assets, and deciding whether to let them out takes an eye trained on contracts. The list of absences in section 5 is the minimum that eye should look for.

8.4The Uncomfortable Part That Lands on Us Too

This report does not close neatly on "so measure with your own data." A score measured in-house cannot be checked from outside. The contradiction described in section 6 catches us too. For anyone who issues quality reports, the question is unavoidable. By what yardstick was that score measured, how many tasks does the yardstick hold, how wide is the interval, and can the customer see the number? The way ARC Prize wrote down the erosion of its own set looks like the most mature answer the industry currently has: instead of concealing, record what has not been verified.

The figures in this report come from three routes. Items 1 through 4 were checked against raw source HTML retrieved for this report, and where a value was never printed, such as the confidence intervals, the body text says that it was converted from the coordinates of the figure on the same page. The public-versus-private pairs in items 5 and 6 were matched directly from the two leaderboards as they stood on 15 September 2026. Items 8 and 9 are quoted from the public arXiv versions.

The spine of this report (checked against primary sources)

• 1.Snagnik Das, Siddhant Paliwal, Janak Sunil (Specific Labs). "Introducing Real-SWE", September 2026. withspecific.com/benchmarks/real-swe — the eight leaderboard rows, the 10×8 per-task results, the five failure categories and their per-model breakdown, the rollout-length cross-tab, the cost and token figures, and the full tax-task instruction all come from this page. The interval values are not printed as numbers there, so they were converted from figure coordinates. The sample tasks used in the analysis are available through a request form on the same site, which collects name, work email, company, role and intended use, and states that the builders reply after review.
• 2.Specific Labs. "Company data partnerships". withspecific.com/company-data — the "$100K–$1M" asking price, the "how real work gets done" line, the fields the form collects, and the list of absences set out in section 5.
• 3.Specific Labs. "Privacy". withspecific.com/privacy — retrieved 15 September 2026. The entire text concerns a LinkedIn advertising-measurement integration; no clause on retention, reuse or withdrawal of licensed code was found.
• 4.Hacker News thread 49676820, 12 September 2026, 272 points, 154 comments. news.ycombinator.com/item?id=49676820 — the co-founders' comments and the quotations from traceroute66, kadoban, cbg0, lmeyerov, didgeoridoo, strobe, IshKebab, barbegal, CompoundEyes, ttul, springtimesun, bisonbear, prometheus1992, dgellow and ignoramous were verified against the full thread text retrieved through the Algolia API. Plain web requests return 429, so the API route is required for the full text.

The public-versus-private drop

• 5.Scale AI. "SWE-Bench Pro (Public Dataset)" leaderboard. labs.scale.com
• 6.Scale AI. "SWE-Bench Pro (Private Dataset)" leaderboard. labs.scale.com — the split structure (1,865 tasks across 41 repositories; 731 public, 276 private from 18 startups, 858 held out) and the seven pairs in section 4 come from here. The 22.7% and 23.1% figures left in both pages' figure captions are an older snapshot and were kept separate from the current tables.
• 7.The original SWE-Bench Pro paper, arXiv:2509.16941. arXiv: 2509.16941 — this report did not open the paper itself; every figure in section 4 was checked against the two leaderboard pages above.

Harnesses and running a private evaluation

• 8.Yunbei Zhang, Janet Wang, Yingqiang Ge, Weijie Xu, Jihun Hamm, Chandan K. Reddy. "Stop Comparing LLM Agents Without Disclosing the Harness." arXiv:2605.23950v1, 7 May 2026. arXiv: 2605.23950 — a position paper by its own abstract, with a controlled factorial experiment in section 4. The 4.9-point and 9.5-point contrast in section 3.3, the 7.80× harness-to-model variance ratio, and the six ranking reversals out of nine comparisons come from here. The factorial experiment is small: three models, three harnesses, 100 tasks, two runs per cell.
• 9.François Chollet, Mike Knoop, Gregory Kamradt, Bryan Landers. "ARC Prize 2024: Technical Report." arXiv:2412.04604. arXiv: 2412.04604 — the four-way split, the definition of the semi-private set, the ±10% overfitting threshold and the passage on erosion from roughly 10,000 reported private-eval scores are all sentences from this report.
• 10.Dmitry Lyalin, Taylor Mullen (Google). "An important update: Transitioning Gemini CLI to Antigravity CLI", 19 May 2026. developers.googleblog.com — the announcement date, the 18 June cut-off for consumer plans, and the continued access for organizations with Gemini Code Assist Standard and Enterprise licenses and for paid API key users are all in this notice.

Related reading on the Pebblous blog

• 11.Is the AI failing physics, or is the question wrong? — on defects in the answer key and the grader of a public exam. This report covers the opposite case, an exam nobody is allowed to see.
• 12.OpenAI Retired Its Own Coding Benchmark, SWE-bench Verified — on why trust in public coding benchmarks came apart.
• 13.The Test Was Already in the Training Data — connects to the contamination measurement discussed in section 6.2.
• 14.'Perfect Score Without Solving Anything' — How 8 AI Agent Benchmarks Were Broken — used here as the frame for separating gaming from honest error.
• 15.A $400M Valuation for the Company That Grades AI on Real Work — on the business structure of the evaluation industry.
• 16.The Moat of the Medical AI 40% of US Doctors Use Is Licensed Journals — that moat was the product; the moat in this report is the exam.