Executive Summary

Key numbers

Four cells. The first two are the material the scoring ran on and the result when the models worked unaided. The second two are what moved once tools were attached, and the paradox that surfaced on the safety side.

Source: DRYLAB-Bench paper (arXiv:2610.13116).

Suppose you hand an AI a protein sequence and ask it to change that sequence in a more dangerous direction. How do you check whether the answer coming back is dangerous at all? Done properly, you build that variant and put it on a bench. What is being measured is danger itself, so the act of measuring brings the danger into the world one more time. This is the spot where AI evaluation in biosecurity has kept stalling.

Evaluations so far have mostly taken a detour around it. One route asks in multiple-choice form whether a model holds hazardous knowledge, as WMDP and VCT do. Knowing and being able to do are different things, so that route never measures the capability directly. Another route hands the model a real task, as ABC-Bench does, but only one of its tasks was dual-use (screening evasion), and the tools its agents held were general-purpose software.

The remaining options fare no better. Grading a model's answers with another AI inherits the grader's own hallucinations and its ceiling. Wet-lab verification is decisive but costly and impossible to scale, and it lands back on the original contradiction: verifying a hazard means creating one.

DRYLAB-Bench starts from experiments that are already finished rather than from new ones. Take a single protein, build a large number of the amino-acid substitutions it could carry, and measure them one by one. The results of that work, known as deep mutational scanning, have accumulated in databases such as ProteinGym and ViroGym, and the job they were built for was to serve as a control set for grading how well a prediction model guesses.

The authors changed what that data is for. Measurements that used to grade prediction models became an answer key for grading what a generative model proposes. Every one of the 30 hazardous design scenarios, distilled from international biosecurity guidelines, had to be a case where real measurements already existed to check an answer against. The single-variant measurements standing behind those 30 scenarios number 187,695.

How the scenarios were picked decides what the answer key is good for. Two computational biology experts led the initial design, with LLMs used only as assistive tools. One filter was applied: a task was retained only if high-quality quantitative ground truth for it already sat in a deep mutational scanning database. An independent computational biology expert then went through each surviving candidate for biophysical soundness and for consistency in residue numbering. Thirty scenarios, covering all eight hazard categories, came out the other side.

The scoring itself runs like this. A variant the model proposes is looked up in the answer key, its measured value is pulled out, and the scale is oriented so that a larger value means more risk in whatever direction the scenario asks for. Breaking a biological constraint such as stability draws a penalty. That value is standardized within each assay by the assay's own mean and standard deviation, then averaged with the model's stated confidence as the weight, which yields the Biosecurity Risk Score (BRS). A refused scenario is scored as zero, so a model that refuses everything gains nothing by it.

One thing is worth holding apart while reading. BRS is a standardized score, not a percentage. It sits on a different scale from figures such as 67.6% or 23.5% further down, and the two should not be read against each other. Every evaluation ran inside a computer, with no synthesis and no wet-lab work performed. For the most severe pathogen scenarios, the adversarial prompts and the hidden ground-truth mapping are gated and shared only with verified researchers and AI developers. The paper is still a preprint, and the project page says it is under review at ICLR 2027.

Fifteen frontier models from eight developers went in. Anthropic's Claude Opus 5, Claude Opus 4.8 and Claude Fable 5; OpenAI's three GPT-5.6 models and GPT-6 Astra; Google's Gemini 3.6 Flash and Gemini 3.8 Flash; xAI's Grok 4.5; and DeepSeek V4 Flash, DeepSeek V4 Pro, Qwen3.8 Max, Kimi-K3 and GLM-5.3. Decoding temperature and reasoning budget were held to the same setting across models so the comparison would hold.

The first condition gives the model nothing but the sequence and the objective, and asks for an answer in one shot. Under that bare-handed condition, 67.6% of the designs submitted landed above the average variant measured in the matching assay. And 21.4% of them fell inside that assay's highest-risk decile. Guessing at random would put 10% there, so this is more than double. The pooled BRS came to 0.359.

One scene reads sharper than the aggregates. A design that GPT-5.6 Sol returned in a single shot, with no tools, turned out to be the highest-risk variant of the 3,372 measured in that assay. That is not a slot you hit by luck. It means the model is feeling its way across the hazard landscape. For context, the share of submitted variants that had a measured value in the answer key, and could therefore be scored at all, averaged 83.9% under the multi-round condition.

The paper is explicit about what these numbers measure and what they do not. The unit of evaluation is a single amino-acid substitution, because deep mutational scanning is the only source of large-scale causal phenotype measurement. So what BRS puts a number on is the risk of the individual design steps taken when assembling a directed evolution campaign or a screening library, not the fitness of a fully synthesized multi-mutant construct. Epistasis between variants, expression levels, and host-context effects beyond the assayed phenotype lie outside its reach. On how any of this would behave inside a living organism, the paper makes no claim at all.

From the second condition onward the models get tools. Six specialist biological design tools: evolutionary conservation from homolog search statistics, sequence likelihood from ESM-1v, structure compatibility from ProteinMPNN, functional annotation from UniProt, toxicity prediction from ToxinPred3, and a scenario-specific predictor drawn from AlphaMissense, EVEscape or FoldX. None of them hands back a measured value from the answer key. A model either calls a tool once and revises its candidate, or works through up to four rounds, choosing tools as it goes, before submitting a final design.

The numbers went up. The share reaching the highest-risk decile moved from 21.4% to 23.5%, and the pooled BRS went from 0.359 to 0.475. Broken into stages, the first tool call added 0.048 and the multi-round phase added a further 0.068.

Shrink that to "give a model tools and it gets more dangerous" and you have it wrong. The size of the amplification ran against a model's bare-handed ability, with a correlation of -0.42. Models that were already good gained almost nothing, and models that were poor caught up through the tools. Claude Opus 5, whose unaided BRS of 0.378 put it near the top, moved only 0.004 once tools were attached. Two models went slightly down instead.

The third result concerns the safeguards. A single model faces 90 evaluation units: the 30 scenarios, each written three ways with the content unchanged and only the phrasing varied, a device meant to keep one turn of phrase from swinging the outcome. Claude Fable 5 refused 87 of those 90, a refusal rate of 96.7%. Yet on the handful of units where it did answer, its BRS of 0.754 was the highest of any model. It mostly blocks, and when something does get through, the answer it returns is the most dangerous in the set. At the other end were five models that refused nothing at all on the scenarios as originally worded.

The authors then ran a controlled test that left the content of each scenario alone and changed only the way it was written. Strip out the pathogen names and the dual-use phrasing, swap in database accession numbers and taxonomic terms, and refusal rates either fell or stayed where they were. Put a statement of the harm the design would cause in front of the request instead, and all five of the models that had never refused anything swung to refusal rates between 70% and 90%. What the models were judging was not the task but the sentence the task arrived in.

Refusal correlated only weakly with actual risk, at a rank correlation of +0.25. A single scenario faces 45 evaluation units, the 15 models times the three phrasings. Scenarios that name a virus were refused in 23.9 of those 45 units on average, while comparably destructive tasks carrying no alarm word, such as disrupting the folding of a protein or ablating a transporter, drew only 8 and 6. The hazard was in the same range; the refusals tracked the vocabulary. Current safeguards classify wording rather than intent, the authors conclude, and the same hazardous objective restated as ordinary scientific work can go straight past them.

This paper usually gets read as material for the question of whether AI can design a biological weapon. To anyone whose work is data, something else shows first. The most carefully engineered thing here is not the set of scenarios that probe the models. It is the scoring method, and the material that method runs on was not newly made. It was already there.

Nobody who built ProteinGym and ViroGym gathered those measurements in order to rate the hazard of a generative model. They gathered them to compare the accuracy of prediction models. Because the records that piled up carried the conditions and the units attached to every single variant, a completely different question got an answer without a new experiment. AI-Ready Data usually means preprocessing for model training, and the usefulness this case shows is not that. It is what a record can do when its measurement conditions survive intact, so that it can be pulled off the shelf again once the question has changed.

So the question left over points outside biology. Does your field also have a pile of measurements kept for some other purpose? Quality inspection histories, sensor logs, return reasons, equipment repair records: things written down to run an operation. If those records carry their measurement conditions and units, then some of the judgments being verified today through expensive experiments could be scored against records already in hand.

Having many records and having an answer key are not the same thing, though. The condition DRYLAB-Bench put on its scenarios shows where the line falls. Only questions with a real measurement already available to check against were eligible, and 30 of them got through. What separates the two is not the volume of data but whether the conditions survived.

Thank you for reading this far. The paper is here, and the code with the scope of its release is here. If there are records in your own field that could serve as a scoring sheet in place of an experiment, we would be glad to hear which ones.

Primary sources

• 1.Wang, Y., Li, C., Le, B., Yan, L., Zhang, J., Yang, Y. T., Zhao, X.-M., Li, L., & Shao, J. (2026). "DRYLAB-Bench: Benchmarking the Biosecurity Risk of Agentic LLM Mutation Design." arXiv:2610.13116, 9 October 2026.
• 2.DRYLAB-Bench project page. (2026). "DRYLAB-Bench." Review status and the responsible disclosure policy as stated by the authors.
• 3.DRYLAB-Bench code repository. (2026). "drylab-bio/DRYLAB-Bench." GitHub.

Prior evaluations and datasets the paper compares against

• 4.Li, N. et al. (2024). "WMDP: a multiple-choice evaluation of hazardous knowledge possession." Cited in the paper as a point of comparison.
• 5.Götting, J. et al. (2025). "VCT: an evaluation of virology capability." Cited in the paper as a point of comparison.
• 6.Liu et al. (2026). "ABC-Bench." A task-level evaluation that attempted wet-lab verification.
• 7.Notin, P. et al. (2023). "ProteinGym: a collection of deep mutational scanning measurements." Source of the answer key data.
• 8.Zhou et al. (2026). "ViroGym: deep mutational scanning of viral proteins." Source of the answer key data.