Executive Summary
Seven weeks have passed since Anthropic shipped Claude Science, a workbench for scientific research. The launch post carried three named success stories, and all three said the work got faster. Exactly one of them carries a sentence saying the results were independently verified. In the other two, the person who used the tool is the person who checked it. The company said from the start that this is not a new model. It runs the same Claude that is already available, and what is new sits underneath it: an execution environment and a recording layer. This is not an occasion to allege a defect. Even inside a single launch post, evidence comes in grades.
The outside world turned up a few things of its own in seven weeks. An independent teardown concluded that the real novelty of this product is not on screen but beneath it. A kernel holds the work state, a daemon holds permissions and history, and the lineage of every artifact is observed during execution rather than reconstructed afterward. The same teardown also recorded two limits: lineage coverage differs by language, and replaying an exported notebook is not recomputation but a playback of recorded responses in order. One independent hands-on review left a sharper scene. A saved review described a risk model built from four factors as though it were the effect of one, the lineage preserved that process faithfully, and the reviewer said nothing. The reviewer was not broken. Checked against the record, nothing was out of place. Catching the error required reading the source paper, and reading source papers is exactly the labor this tool promises to reduce.
What, then, should be measured? Agentic science benchmarks already exist in numbers, and when the measuring is done, performance is still low. None of them, however, measures what this product actually staked. Its claim is not that it solves tasks better but that outputs carry their own evidence, and there is no scoreboard for whether the lineage is right or whether the reviewer catches consequential errors. The party that put its name on a public leaderboard was not the vendor but an open-source alternative that appeared six days after launch. Meanwhile, a recent study pooling more than 25,000 agent runs across eight domains reported that what separates performance and behavior is the base model rather than the harness, and that outcome-only evaluation cannot detect this class of failure. Traceable and sound are two different words. This report measures the distance between them.
Key Numbers
These three numbers measure three different layers: the success the vendor reported, the capability an academic leaderboard measured, and the limit the vendor's own documentation lists. All three return later in the report.
named cases whose write-up
states the results were independently verified
average for the strongest autonomous agent
on an end-to-end autonomous research benchmark
Claude Science events recorded
in the organization audit log (vendor docs)
What's New Is Not the Model but the Layer Beneath It
When Anthropic released Claude Science in beta on June 30, 2026, it stated plainly that this is neither a new AI model nor a model tuned harder for biology. It runs the same Claude models already available today, with no special access and no gating. What the product is, and how it binds literature, databases, code, and high-performance computing into one place, was covered in our piece written the day after launch. This report starts there and moves straight to the next question. If the model is unchanged, what exactly was laid underneath it, and what does that layer guarantee?
The most concrete answer came not from the vendor but from outside it. On July 6, 2026, the technical analyst feitong.phd published a 24-minute teardown of version 0.1.15-dev and concluded that the novelty sits below the screen, not on it. The core sentence is short: the kernel holds the work state, and the daemon holds permissions and history. The chat window the user sees is a thin shell, and a separate layer remembers what may be done and what was done.
The install itself shows that structure. The signed app is 111MB, the staging daemon about 112MB, and the versioned runtime about 95MB, while the separately provisioned execution environment runs to roughly 3.7GB. Of the roughly 3.9GB total, some 95% is not application but room for computation to run. The claim that you are downloading a workbench rather than a chat app is physically visible in that ratio.
1.1Code Is the Orchestration Language
Change the structure and you change how the agent works. The teardown's example is the literature review skill that ships with the product. Facing roughly 240 candidate papers, the model in this app does not make 240 tool calls. It writes one loop that runs inside the kernel and finishes in two code cells. Quality of the result aside, the shape of the transcript changes. The conversation record stays flat instead of ballooning, and the burden on whoever has to read back through what happened drops with it.
Permissions are held behind two layers. The outer layer is the SDK surface the kernel exposes, so calling a capability that does not exist raises an error. That is closer to ergonomic guidance and can be routed around. The real boundary is the inner layer, the daemon's allowlist. Reading the daemon source, the teardown confirmed that a hand-forged call pushed through is refused by the daemon. Even if the model is confused or coaxed, the permission ruling happens outside the model.
1.2Lineage Is Observed Before It Is Reconstructed
The part most worth watching is how lineage gets made. The usual tool digs through logs after the work is done and pieces together what came from what. The analysis SDK in this app wraps the functions that read data and the functions that write results out. The moment a table is read, an array is loaded, or a figure is saved, the source and a version tag travel with it. This is observation during execution, not inference after the fact.
As a result, each artifact carries evidence of a different strength. Below are the four tiers the teardown laid out. Higher tiers are closer to what the machine saw directly, and lower ones closer to inference and open questions.
Original Pebblous diagram, reconstructing the four tiers described in feitong.phd's teardown (2026-07-06). Bar lengths indicate the order of strength, not a quantitative ratio.
1.3Two Limits the Same Teardown Wrote Down
The teardown does not stop at praise. In a footnote the author states that lineage reliability is uneven. Python has the densest object wrappers, while R and Bash lean more on write tracking and after-the-fact reconstruction. On macOS configurations, Bash is not traced through the same path as on Linux. His conclusion fits in one sentence: having lineage should not be presented as a binary guarantee.
"This is portability, not full recomputation. The replayed LLM answer is the answer that was recorded during the actual run. The replayed external tool result is not the result of a fresh query. What the tape proves is that the exported code executes in the recorded order against preserved host responses. It does not prove that a clean environment could produce those responses again from the outside world."
What the export feature actually does turns on this point. When you export a notebook, the app records host responses into a tape file and later replays them in that order. A successful replay therefore means the code runs against preserved responses, not that querying again today would return the same answers. Between what "reproducible" usually means in a lab and what this product guarantees, that is the gap.
What the lineage layer really does is bind claims to a record. Whether that record is accurate, and whether recomputing along it yields the same values, are separate questions. This distinction runs through the whole report. The failure we reach in Section 3 happened in exactly that gap.
Seen against its predecessors, this approach differs in kind. The autonomous authoring systems that set out to write papers on their own competed on the polish of the output, and verification was mostly deferred. Claude Science bet the other way. Instead of competing directly on the quality of the result, it leaves behind a machine-readable account of where the result came from. How far that bet actually pays out is the rest of this article.
The Three Named Cases
The launch post carried three cases with names attached: Manifold Bio, which develops tissue-targeted medicines; Jérôme Lecoq, a neuroscientist at the Allen Institute; and Stephen Francis, an associate professor of epidemiology at the UCSF Brain Tumor Center. All three were chosen by the vendor, and all three are self-reported. That is not a flaw in itself. Self-report is essentially the only evidence a launch post can carry. But the reader has to assign a grade, and the last two columns of the table below are that grade.
| Case | What the company reported | Type of evidence | Who verified |
|---|---|---|---|
| Manifold Bio tissue-targeted medicines |
Used for target nomination in its latest experiment. Assessed surface expression, trafficking, and safety across tissues and targets, then ranked candidates against the company's own internal data. What set it apart from a general coding assistant, the company said, was that it carried the work end to end | Company self-report no figures |
Same party that used it |
| Jérôme Lecoq Allen Institute, neuroscience |
Built a multi-agent review template out of about 20 custom skills. Subagents pull key claims and quantitative results from thousands of papers into an evidence-state database and generate section-by-section narrative. One such review used to take up to two years; he has now produced about ten, many of them over 100 pages | Researcher self-report time and volume figures given |
No external verification mentioned |
| Stephen Francis UCSF Brain Tumor Center, epidemiology |
Analyzed thousands of small-effect germline variants in molecular epidemiology work on glioma. Ran a comprehensive germline workup spanning several approaches in roughly one-tenth of the previous time | Researcher self-report | The research team independently verified the results the only case where the write-up says so |
Taken from the body of Anthropic's official launch post (2026-06-30). All three cases appear in that post; none is a secondhand summary quoted from press coverage. Time-related figures follow the original wording. "Up to two years" is an upper bound, and "roughly one-tenth" is an approximation for one lab's specific workup.
Read the table vertically and one frame remains. The claim of speed appears three times; the claim of verification appears once. In Lecoq's case, what did the verifying was not a person but a review agent he built himself. Putting a separate critic agent alongside the generating agent is the crux, he said, and he is now refining that critic side with domain experts. Which raises the question of how the reviewer that ships with the product is actually designed.
2.1The Reviewer's First Principle Is Not to Recompute
The first principle in the reviewer prompt, as read out by the teardown, is terse: trace, do not recompute. When the agent asserts a number, the reviewer walks back to the cell that emitted it, checks it against the record, and looks for contradictions. The rule goes one step further. A value that cannot be traced within the window is, by itself, not a finding. The design refuses to produce a reviewer that piles up suspicion without grounds.
The standard also shifts with where a claim was made. Saved artifacts such as figures, tables, and reports face a strict bar, while flowing chat prose is flagged only when acting on it as written would materially mislead. And the reviewer runs stripped of nearly all its tools. Planning, delegation, web search, Python, R, Bash, artifact writing, even the thinking process are taken away, leaving three of twelve declared tools.
Original Pebblous diagram. Reconstructed from the independent teardown (feitong.phd, 2026-07-06) describing the reviewer's tool restrictions. It illustrates the stated ratio of three allowed out of twelve; the source does not name which three remain.
What makes that stripping interesting is that it was not a matter of taste. The reasoning survives in the code comments. Extended thinking consumed roughly 72% of the reviewer's output tokens while adding effectively nothing to recall on tracing work, and when 30 actual reviewer rounds were measured, Python accounted for 41% of tool calls, every one of them the recomputation the rules forbid. The harness was cut down against measurements. That fact returns as a problem in Section 4.
2.2Two Lines the Vendor's Own Docs Draw
The reviewer's scope is also written into Anthropic's official documentation, and two sentences there are unusually precise. One says the reviewer does not re-run analyses. The other says it does not judge whether the method that was run was the right choice for the research question. It can flag conclusions that do not follow from the method executed, but it does not ask whether that method was appropriate to begin with.
"The reviewer checks whether claims match the record. It doesn't re-run analyses. It can flag conclusions that don't follow from the method that was executed, but it doesn't judge whether that method was the right choice for your research question."
The second line is drawn across the pricing tiers. According to the same documentation, automatic review is on by default on Max, Team, and Enterprise, and off by default on Pro. Pro users have to switch it on each session, and review itself consumes plan usage. A low barrier to entry is often listed among this product's virtues; the fact that on the cheapest plan the verification layer does not run by default is the other half of that virtue.
2.3What a Showcased Researcher Said a Month Later
In July 2026, The Scientist ran an early-assessment piece based on interviews with three researchers. According to the report, all three said they would keep using the tool, and all three voiced concerns as well: it was sometimes wrong, and the researcher had to review carefully. Lecoq is reported to have noted its limits in judging the quality of a paper, saying the problem is no longer predicting the next token but doubting the next token. Another researcher raised the risk that a lab head under funding pressure would use the tool as a shortcut instead of hiring.
We attempted this article four times with a browser user agent and received bot-blocking responses every time, so the body text was never obtained directly. This is automated-access blocking, not a paywall. The account above was confirmed through a search index, so it should be read as indirect attribution to The Scientist's reporting, and the exact wording of the quotes needs re-checking against the original.
Once that level of access is disclosed, the passage becomes more valuable rather than less. One of the three cases the vendor selected draws a line, a month later, around the same tool's judgment. This is not an allegation that the vendor was wrong. It is a scene in which the self-report in the launch post and the testimony from real use diverge in the same person, which is precisely why evidence has to be read by grade. Who confirms a discovery, and with what standing, is a question we took up separately in an earlier report.
What the Outside World Said in Seven Weeks
The external response over the seven weeks since launch is not one block. Press framing, the community's first impressions, one person's hands-on measurements, reporting on the business strategy, and an open-source alternative that appeared within six days all sit at different levels. Each level can support different claims, so they should not be read as one.
3.1The Press Read It as Productization, Not a First Entry
The analysis R&D Magazine ran on July 28 placed the launch on a timeline. Anthropic shipped a life sciences product in October 2025, widened it to healthcare in January 2026, and stood up a separate research workbench in June. On that reading, this was not a first step into new territory but a place built for work already underway. The piece also called the "AI does research" framing overblown. On launch day, TechCrunch made the same point, noting that while the fact-checking step matters, it is ultimately the same base model checking itself rather than an independent source of truth.
3.2The Community's First Impression Was Skeptical
The Hacker News thread on launch day reached 564 points and 174 comments. Top responses ran mostly skeptical. What science lacks is not paper count but good papers, several argued; citation hallucination came up repeatedly; so did the observation that the preconfiguration leans heavily toward life sciences. PubMed and genomics databases are wired in, while ACM and IEEE material is not. There were reports of the Linux build crashing, and questions about whether Claude Code was not already enough. One user working in developmental biology wrote that the scientific reasoning was clearly improved and better than other chatbots, but that without specifying what is already known it falls short of the frontier of the field, because many recent papers sit behind closed access and never entered training.
Community reports are anonymous and disclose no reproduction procedure, so this report treats them at the lowest grade. They show the scale and spread of sentiment; they are not grounds for adopting any individual claim as fact.
3.3One Prompt, Two Tools, Measured Side by Side
The most valuable material is Takashi Fujino's review, published by Future Stack Reviews on July 21. He gave the same prompt to the regular Claude app and to Claude Science, opened every tab of the saved artifacts one by one, checked key numbers against the primary papers, and ran a controlled test on the reviewer using synthetic data with known answers. The author states his own limits: one topic, one run per tool, one date, one plan, one beta build. He also disclosed the conflict that the review was written with Anthropic's Claude while auditing an Anthropic product. What follows should be read with both caveats attached.
The most concrete scene this exercise produced turns on a single sentence about a protein called SDHB. The prose in the saved review stated that loss of the SDHB protein stratified five-year progression-free survival from 91.5% to 34.8%. But the paper the app had pulled in full text defines that survival stratification through a risk model built from four factors: SDHB expression, primary tumor size, final diagnosis, and Ki-67 index. SDHB is one of the four, and the prose promoted that one into the whole model. Placing three layers of the same claim side by side makes the point of divergence obvious.
| Layer | Actual wording | Accurate? |
|---|---|---|
| Prose in the saved review | Loss of the SDHB protein stratified five-year progression-free survival from 91.5% to 34.8% | Promotes one factor into the whole model |
| Structured record the app left | Five-year progression-free survival: low risk 91.5%, intermediate 41.7%, high risk 34.8% | Accurate |
| The source paper | A three-tier risk model built from four risk factors (SDHB expression, primary tumor size, final diagnosis, Ki-67 index) | The authority |
Reconstructed from what Future Stack Reviews (2026-07-21) recorded when checking against the source. The reference entry itself was correct; only the prose was wrong.
The reviewer did not catch this, and that is by design. The numbers are real and SDHB genuinely is one of the four factors, so nothing is out of place as long as you check against the record. Catching this error requires reading the source paper, and reading source papers is exactly the labor this tool promised to reduce. The data was right, the code was right, the record was right, and the error occurred in the layer that carries meaning over from the source.
What matters just as much is that the same review recorded results in the opposite direction. In the controlled test, the app refused to call a pure noise variable a significant biomarker and corrected two textbook statistical traps on its own. In a separate run, the reviewer caught the model's claim to have verified citations on the grounds that no such step appeared in the log. So it is strong against lies that can be checked against the record and weak against distortions that require reading the source. The two results are not a contradiction but two sides of the same design.
The observations on data access are practically useful too. The "60-plus" figure in the launch post is a catalog, not a promise. The connection wizard lists 42 connectors, 25 of them active by default, and one bibliographic database marked as free to use began requiring a free API key for full-text access in a July update; in the actual run, the app could not find a key and fell back to PubMed alone. On literature matching, Claude Science returned 49 records against the regular app's 8 confirmed plus 11 named. Two of the 49 were preprint and published versions of the same work, making the effective count 47, and the set mixes reviews, case reports, and animal models. With no independent reference set, the author wrote, an honest comparison stops at record counts rather than recall.
3.4Selling the Software While Competing in the Same Market
At an event in San Francisco on launch day, Anthropic's head of life sciences, Eric Kauderer-Abrams, announced that the company will develop drugs itself, starting with neglected diseases. The rationale was put this way: to build the right models, products, and tools to accelerate the industry, the company has to live it too, and nothing substitutes for the shared experience of being in the trenches of drug development. CNBC and The Verge each confirmed the announcement independently.
Structurally, that puts the company in the position of selling software to pharmaceutical firms while competing with them in the same market. What it would do with a promising candidate was not disclosed, and it did not respond to The Verge's request for comment. The expert reservations attached to that same reporting are clear. Frank von Delft of Oxford said AI models are nowhere near making experiments unnecessary, and Matthew Todd of UCL said AI-designed drugs have a long way to go before regulatory approval. No AI-designed drug has yet cleared clinical trials and FDA approval to reach the market.
The same article reported that Anthropic has spent the past year hiring biologists and building its own laboratory. Namshik Han of Cambridge said several colleagues had been approached and that a few had moved over from large pharmaceutical companies and top universities. What is striking is that the technical teardown predicted the same direction. At the end of his analysis, feitong.phd wrote that Anthropic would tighten a feedback loop in which model, harness, and laboratory co-evolve as the company both performs and funds lab research. Someone reading the code and a reporter interviewing people arrived independently at the same conclusion.
3.5An Alternative Built in Six Days Became a Product in Seven Weeks
Six days after launch, an open-source alternative appeared: Open Science Desktop. Seven weeks on, it is not a sketch. It has accumulated 432 commits, shipped a build with browser control and a token authentication gateway on July 21, added split panes on July 24, projects, memory, and history on August 1, and bidirectional Agent Client Protocol support on August 13.
What someone chooses to clone is what the market saw as the core value. The project's own pitch comes down to three lines. Everything traces back: figures, tables, reports, notebooks, and execution outputs link to the exact code, inputs, environment, model outputs, and conversation that produced them. It is reproducible by construction. It is local-first and yours, and by default nothing leaves. Two things set it apart: it is model-agnostic, so you can attach whichever model you want, and it supports Windows in addition to macOS and Linux. Claude Science has no native Windows build.
The clone's pitch summarizes this product's core more precisely than the vendor's own. What people decided to copy was not model performance but the record structure. And as Section 4 shows, the party that put its name on a public scoreboard was also the clone, not the vendor.
The state of external evaluation itself shows through here. One of the most detailed governance commentaries published since launch states in its own text that it is a chatbot-written summary. The event figures and panel remarks it carries could not be confirmed at first hand, so this report does not use them. One line of its conclusion is worth quoting: what Anthropic has demonstrated so far is the acceleration of scientific work, not the acceleration of verified scientific truth. That gathering outside assessments seven weeks after launch leads you to quoting an AI-written summary is itself part of why this article insists on grading evidence.
What Benchmarks Measure, and What They Don't
Is there anywhere to compare tools like this objectively? The hypothesis we started this investigation with was that no one has measured it yet, and that hypothesis was wrong. Agentic science benchmarks already exist in numbers, and some of them assign end-to-end autonomous research. The accurate conclusion comes in three layers: the benchmarks exist, the measured performance is low, and none of them measures what this product actually staked.
4.1The Scoreboards Exist, and the Scores Are Low
Below are the main benchmarks confirmed in this investigation. The last column states how far this report got with the original source. For entries where only scale was confirmed, no performance figure is given.
| Benchmark | Scale | What it measures | Best performance | Level of access |
|---|---|---|---|---|
| AstaBench Allen Institute for AI, ICLR 2026 |
2,400+ problems across 11 sub-benchmarks | The full arc of scientific research. Evaluates 57 agents across 22 agent classes with a controlled search-tool environment and time-invariant cost accounting | The conclusion is the finding. AI remains far from solving the task of research assistance | Abstract obtained in full detail |
| ResearchClawBench InternScience, v5 |
40 tasks across 10 scientific fields | End-to-end autonomous research. Grounded in real papers, with the target paper hidden during evaluation | Strongest autonomous agent averaged 21.5; strongest harness-plus-model combination 20.7; frontier models averaged 26.5 | Abstract obtained in full detail |
| ReplicationBench | Full replication of astrophysics papers | Tasks designed with the original authors, scoring faithfulness and correctness separately | Even the best-performing model scored under 20% | Abstract obtained in full detail |
| CORE-Bench | 270 tasks, 90 papers, 3 fields | Computational reproducibility. The hardest tier hands over only the codebase and requires dependency installation, execution, and reading the output | Precise figures not obtained in this investigation | Scale only |
| ScienceAgentBench | 102 tasks from 44 peer-reviewed papers | One self-contained step of a real workflow. Designed to move away from abstract claims of end-to-end automation | Precise figures not obtained in this investigation | Scale only |
| BixBench | 50+ scenarios, roughly 300 open-ended questions | Bioinformatics. Deliberately selects tasks that are ambiguous, open-ended, and lacking a clear optimization metric | Precise figures not obtained in this investigation | Scale only |
PaperBench, DiscoveryBench, SUPER, LitQA2, and DS-1000 were also identified, many of them folded in as sub-benchmarks of AstaBench. Headline numbers in benchmark papers are usually maxima under specific conditions, so the table above carries only the averages stated in the abstracts.
4.2They Measure Task Success; This Product Staked Evidence Preservation
Every scoreboard in the table answers the same question: how much of the task does this agent solve? But the claim Claude Science made is not that it solves tasks better. It is that outputs carry their own evidence. No benchmark that scores whether the lineage record is actually right, or whether the reviewer catches consequential errors, turned up in this investigation. The SDHB case in Section 3 happened in exactly that blank. The task succeeded, the record was accurate, and one line of prose was wrong. No scoreboard deducts a point for that failure.
More conspicuous is the absence of a name. Claude Science appears on none of the public leaderboards above. The open-source alternative that appeared six days after launch, by contrast, said on July 9 that it ranked first on the average of ResearchClawBench's scored tasks. That ranking is a claim in the project's own documentation and could not be independently confirmed on the official leaderboard. The leaderboard also accepts submissions of externally produced runs, so some entries are self-reported. Even so, one thing is clear: the party that chose to put its name on a public scoreboard was the clone, not the vendor.
4.3The Failures Outcome-Only Evaluation Cannot See
A study published in April 2026 addressed what remains even when you do measure. Ríos-García and colleagues pooled more than 25,000 agent runs across eight domains and analyzed them through two lenses, from workflow execution to hypothesis-driven inquiry. One is a performance analysis decomposing the contributions of base model and scaffold; the other is a behavioral analysis examining the epistemic structure of agent reasoning.
The two lenses returned answers of different kinds. On performance, what separated outcomes and behavior fell overwhelmingly to the base model, and however the harness was tuned, the share explained there stayed slight. The behavioral side is more uncomfortable. More than two-thirds of traces walked past evidence the agent had generated itself, and belief was revised in the face of disconfirming evidence about one time in four. Triangulation, where different methods converge on the same conclusion, was rare. The table below puts the two answers side by side.
| Measure | Value |
|---|---|
| What determined performance and behavior: the base model | 41.4% of explained variance |
| The scaffold, that is the harness, on the same basis | 1.5% |
| Traces in which evidence was ignored | 68% |
| Belief revised on the basis of disconfirming evidence | 26% |
| Triangulation, different methods converging on one conclusion | Rare |
Figures from the abstract of Ríos-García et al., "AI scientists produce results without reasoning scientifically," arXiv:2604.18805 (2026-04-20). This study did not test Claude Science. It evaluated LLM science agents in general, so these numbers must not be read across as one product's score.
The same reasoning pattern appeared whether agents were executing computational workflows or conducting hypothesis-driven inquiry, and it persisted even when near-perfect successful reasoning traces were supplied as context. In epistemically demanding domains, unreliability compounded across repeated trials. The authors' conclusion reads as follows: current LLM-based agents execute scientific workflows but do not exhibit the epistemic patterns that characterize scientific reasoning. Outcome-based evaluation cannot detect these failures, and scaffold engineering alone cannot fix them.
Applying that conclusion directly to this product would be an overreading. It does, however, point precisely at where the product placed its bet. As Section 2 showed, the Claude Science reviewer is a scaffold cut down with care against measurements. Removing extended thinking and blocking Python were decisions made after running evaluations. And the contribution of that layer is exactly what the study above measured at 1.5%. The two facts do not refute each other. Neither says harness engineering is pointless, nor that the reviewer is useless. They point from different directions at where the boundary lies between what a harness can fix and what it cannot, and there is not yet data to adjudicate that boundary.
Return to TechCrunch's reservation on launch day and the loop closes: the verification step matters, but it is ultimately the same base model checking itself rather than an independent source of truth. However dense the record, if the eye reading it is the same model, what remains is proof of consistency, not proof of validity. What is lost when the process does not survive is something we took up separately on the mathematical proof side. This product succeeded at preserving that process, and whether the preserved process is right remains open.
What to Check Before Adopting It
Everything in this section was obtained directly from Anthropic's official documentation. It is a list the vendor itself compiled of what does not work yet, which puts it at a high grade of evidence, and an organization evaluating adoption can lift it straight into a checklist. Because it documents a beta product, though, it carries a timestamp: as of August 17, 2026. The first thing to do when you begin an evaluation is to reopen that list and see what has changed.
5.1Admin Controls Are Still Mostly Empty
In May 2026, Anthropic shipped the Compliance API alongside 28 enterprise security integrations. None of them reaches Claude Science yet, and the reason is largely the same in each case: the data sits on the user's computer, so there is nothing on the server side to act on. The advantage local-first design gives an individual becomes, for an organization, a gap in control.
The eight rows below are better read as two kinds. One kind is simply not built yet but stated in the documentation to be on the roadmap; audit logs and connector allowlists belong here. The other kind has no object to act on at all because the data is not on the server, which is the case for organization data export and retention windows. The first can fill in with time; the second stays as it is unless the storage architecture changes. How much risk an organization has to carry today turns on that distinction.
| Control | Status per the vendor's documentation |
|---|---|
| Audit logs | No Claude Science events are recorded in the organization audit log at all. The docs say only that it is on the roadmap |
| Compliance API | Data export and deletion are not possible. Conversations live on the user's computer, so there is no data on Anthropic's side for the API to reach |
| Organization data export | Data residing on members' computers is not included |
| Retention windows | Nothing is stored on the server, so there is nothing for an automatic deletion window to act on |
| Local deletion signal | When a user deletes local data, Anthropic is not notified, so server-side model traffic logs remain for the standard retention period |
| Connector and skill allowlists | They apply only to directory connectors published by the organization. Connectors a member adds locally cannot be controlled, and this too is roadmap status |
| Session duration | Only the browser login step is limited. After login the app carries its own token past that window and stays signed in |
| Offboarding | Removing a member only blocks login. Data already on that computer is not wiped, and the docs direct you to device management software |
Taken from Anthropic's documentation page "What's not available yet" (accessed 2026-08-17). Wording has been condensed; meaning is unchanged.
5.2Local-First Is Not Local-Only
The data boundary is documented officially as well. Conversations and artifacts are stored only on the device. But every model call sends prompts and responses to Anthropic's servers, where they are logged under the standard retention policy for model traffic. Code and data pushed to remote compute go straight to their destination without passing through Anthropic, and administrators cannot yet restrict whether members connect to remote compute at all. Connectors a user adds locally also talk directly to the app rather than routing through Anthropic. Product usage telemetry covers event counts and timing rather than conversation content, and it can be turned off in device settings.
There is one more boundary: where the computation runs. Work the app runs inside itself stays within Anthropic's sandbox, while work pushed to a lab server or HPC resource leaves that fence.
Work sent over SSH or to HPC runs outside the sandbox
The vendor's documentation states that such work executes outside the sandbox, as your account on the host, with access to everything that account can read and write. The trust boundary moves from Anthropic's sandbox to your own account privileges. The documentation also specifies that when the code tab and the execution log disagree, trust the log.
5.3Who Checks, Who Pays, and What You Can't Use It On
The remaining items are grouped by kind. They bear directly on contracts, budgets, and compliance.
Different people should check each one. Whether the verification layer is on depends on the plan, which makes it the principal investigator's call. Costs accrue outside the app and do not stop when the app is closed, so budget owners need to look. The limits on what data may be handled and which operating systems are supported belong to legal and IT. The three items below follow that order.
Verification layer: check the default
Automatic review is on by default on Max, Team, and Enterprise, and off by default on Pro. Review consumes plan usage as well. And the reviewer does not re-run analyses, nor does it judge whether the method was appropriate to the research question.
Cost: there is no spending cap inside the app
Modal remote compute is billed directly to the user, and the app offers no spending cap setting. Jobs keep running and billing after you close the app. The default container timeout is 12 hours, with a maximum of 23. Plan usage is shared with Claude Code and Cowork, and the background reviewer draws on that usage too. On the other side of the ledger, the AI for Science support program offers up to $30,000 in Anthropic credits to as many as 50 projects, with Modal providing up to $2,000 in compute to some selected projects.
Scope: the data you cannot use and the OS you cannot use
Protected health information is prohibited. The beta is not covered by a business associate agreement. There is no native Windows build, and the documented path runs through WSL 2. The connector list is a catalog rather than a guarantee of response, and the user complaint that preconfiguration leans toward life sciences should be weighed alongside it.
5.4Three Paths, and the One Line That Belongs in Procurement
The competitive field has split three ways. Anthropic sells a broad subscription bundling some 60 scientific databases. In April 2026, OpenAI released GPT-Rosalind, specialized for biological reasoning, as a research preview for US enterprise customers, gated behind eligibility screening and safety review. Google DeepMind bundled around 30 life sciences databases while holding its own foundation models such as AlphaFold and AlphaGenome. One opens wide and leaves verification to the user, one opens narrowly and screens eligibility, and one owns the models themselves.
Whichever you pick, the sentence that belongs in the procurement document is the same: who independently confirms what the tool produced? The grounds for that requirement are that, of the three cases in the launch post, one carried such a sentence. Where the independent verification clause goes in the adoption contract and the PoC design, and whose budget covers the time of the person performing that verification, should be settled in the first meeting. Bolting it on later always costs more.
Why Pebblous Follows This
Editor's Note. What follows is not a research finding from the body of this report but the perspective of the editorial team covering the topic. The five chapters above rest on vendor documentation, an independent technical teardown, hands-on reviews, and academic benchmarks. This chapter states where Pebblous places that material when reading it.
6.1Consistency Checking Is Not Reproduction, and Reproduction Is Not Validity
The SDHB case stays with us longest because of where the failure sat. The data was not wrong and the code was not wrong. The record was accurate and so was the reference entry. What was wrong was the layer that carries meaning over from the source. Separating three levels makes clear what this tool guarantees: a consistency check, asking whether the record and the artifact contradict each other; reproduction, asking whether walking the same procedure again yields the same values; and validity, asking whether that procedure was the right method for this question in the first place. The reviewer does the first. The vendor's documentation says as much.
Original Pebblous diagram illustrating the argument of Section 6.1. What the reviewer actually performs is a consistency check; reproduction and validity are not explicitly covered by the vendor's documentation.
6.2For a Tool to Observe Lineage, the Input Side Needs Lineage Too
The problem Pebblous has been working on under the name AI-Ready Data overlaps with this point. What Claude Science staked was not model performance but an execution environment that leaves evidence behind, and a structure in which code, environment, and history follow every artifact is one way of solving, from the research-tool side, the job of attaching context to data. That lineage, however, begins the moment the app reads the data. Where the data came from before it was read, and whose hands it passed through, sits outside this structure. The teardown's footnote about lineage reliability differing by language therefore reads more broadly. It is a warning that applies to every product sold on a binary of having lineage or not.
From the data quality side, one more worry appears. Drawing a perfectly reproducible figure from bad data gets faster in this structure. The reviewer checks whether the code and the artifact agree; it does not check whether the input data was wrong. That is why quality gates belong at the front of the pipeline, and why efforts to score data readiness up front have been multiplying.
6.3What Remains Is the Habit of Reading by Grade
This article is not a verdict. No conclusion follows here that Claude Science failed, or that the launch was overblown. What seven weeks confirmed is that this product made the record strong. What remains unconfirmed is how much that record guarantees about accuracy. The practical work left for an organization choosing tools is therefore to read by grade. Who claimed it? What was measured? Who confirmed it? Apply those three questions to vendor cases, to benchmark scores, and to this report alike, and most of the overstatement filters itself out.
AI Scientist Series Map
The Pebblous blog has approached the question of AI replacing or assisting research from several angles. This article inspects the evidence behind one product; the others each take a different question. Below is a map in reverse chronological order, with the question each piece answers on the right.
2026-08-17 · this article
Three Showcase Cases for a Science AI Workbench, One Independent Check
What evidence currently backs the announced claims?
2026-08-16
Rewriting Only the Prose Moved AI Reviewer Scores Across 4,080 Papers
Does AI review read the content or the sentences?
2026-08-09
The Proof Was Right. How It Got There Wasn't Recorded
What is lost when only the result remains and the process disappears?
2026-08-06
It Can Reason, But It Cannot Discover
What kind of leap can an LLM not make?
2026-08-03
Who Grades the Discoveries AI Makes?
Who holds the standing to verify?
2026-07-21
AI Knows the Answer, Just Not When to Trust It
Judgment in the face of messy data
2026-07-04
AI Now Grades Your Data's AI-Readiness
How do you measure data readiness?
2026-07-01 · the piece before this one
Claude Science Makes Reproducibility a First-Class Feature of Scientific AI
What is this product?
2026-06-21
AI Changes the Questions, Not the Answers
What changed about discovery?
2026-06-17
AlphaFold Folded Proteins. Now AI Designs Them From Scratch.
Science that crossed into generation
2026-05-31
AI Made Individual Scientists Stronger, Made Science Itself Narrower
What tools did to science overall
2026-04-05
When AI Writes Science: The Reality Behind JAIGP and Sakana AI Scientist
The reality of autonomous authoring attempts
2026-03-28
The AI That Reads Papers and Runs Its Own Experiments
The prototype of autonomous discovery systems
References
Product facts in this report rest primarily on Anthropic's official launch post and documentation. External assessments are cited from press articles, an independent technical teardown, and hands-on reviews, each with its grade stated. Benchmark figures were taken directly from paper abstracts. Where the body text of a source could not be obtained, that is noted at the point of use.
Primary sources · Vendor
- 1.Anthropic. "Claude Science, an AI workbench for scientists" (2026-06-30). The authority for the product's composition, the three cases, and the AI for Science support program.
- 2.Anthropic. "The reviewer," Claude Science documentation. The reviewer's scope and the per-plan defaults.
- 3.Anthropic. "How Claude Science works with your data," Claude Science documentation. The boundary between local storage and model traffic logging.
- 4.Anthropic. "What's not available yet," Claude Science documentation. Source of the table in Section 5.1.
- 5.Anthropic. "Claude Team plan for research labs." Plan terms for research organizations.
- 6.Modal. "Modal integration brings scalable compute to Claude Science" (2026-06-30). Remote compute billing structure and container timeouts.
Press · Independent assessment
- 7.Bellan, R. "Anthropic's Claude Science bets on workflow, not a new model, to win over scientists." TechCrunch, 2026-06-30. The vendor's own framing, the three-way competitive field, and the limits of self-checking.
- 8.feitong.phd. "Dissecting Claude Science" (2026-07-06). Technical teardown of version 0.1.15-dev: the daemon-centered harness, lineage observation, reviewer design, uneven lineage coverage, and the limits of replay.
- 9.Fujino, T. "Claude Science Review: what it is, and how it differs from the regular Claude app." Future Stack Reviews, 2026-07-21. An N=1 hands-on test with a conflict of interest disclosed by the author. Source of the SDHB over-attribution case and the connector measurements.
- 10."Early Verdicts on Claude Science: Faster Workflows, But Gaps Remain." The Scientist, 2026-07. Body text not obtained (automated access blocked). Treated as indirect attribution in the text.
- 11.Yoshinaga, K. "Claude Code for Science? Workflow, not a new model." R&D Magazine, 2026-07-28. Product timeline and criticism of the framing.
- 12.Hart, R. "Anthropic wants to develop its own drugs." The Verge, 2026-07-03. In-house drug development, the conflict of interest, expert reservations, hiring, and laboratory construction.
- 13.Peebles, A., & Capoot, A. "Anthropic launches AI drug discovery program." CNBC, 2026-06-30. Event announcement and direct quotes.
- 14.Hacker News thread, item 48735770 (2026-06-30). 564 points, 174 comments. Cited as anonymous community material with its grade stated.
- 15.ai4s-research. "Open Science Desktop," GitHub. The open-source alternative that appeared six days after launch. Source of the release history and the leaderboard claim.
Academic · Benchmarks and agent evaluation
- 16.Ríos-García, M., et al. (2026). "AI scientists produce results without reasoning scientifically." arXiv:2604.18805, 2026-04-20. More than 25,000 runs across eight domains. Base model 41.4% versus scaffold 1.5%, evidence ignored in 68%, belief revised in 26%.
- 17.Bragg, J., et al. (2026). "AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite." ICLR 2026, arXiv:2510.21652.
- 18.Xu, et al. (2026). "ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research." arXiv:2606.07591v5. 40 tasks, 10 fields, strongest autonomous agent averaging 21.5.
- 19.Ye, et al. (2025). "ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers?" arXiv:2510.24591v2. Even the best-performing model scored under 20%.
- 20.Siegel, Z. S., et al. "CORE-Bench" (270 tasks, 90 papers) · Chen, Z., et al. "ScienceAgentBench" (102 tasks, 44 papers) · Mitchener, L., et al. "BixBench" (50+ scenarios). Only scale and design intent were obtained in this investigation.