Executive Summary

Key Numbers

Source: Qin et al., What Are LLM Agents Actually Learning on Low-Budget Molecular Optimization Benchmarks?, arXiv:2611.08522 (October 30, 2026)

The two numbers in the first card below are one agent solving one set of tasks, and all that separates them is what the prompt said. The second card shows that with the same information in hand an LLM becomes optional, the third says how much of the answer the test gives away before anyone starts, and the fourth is what all of it cost to run.

PMO, short for Practical Molecular Optimization, is a shared test bed built in 2022 so that algorithms searching for drug candidate molecules could be compared side by side. Twenty-three tasks run through a single oracle interface. The oracle is a scorer that takes one molecule and returns one number, and depending on the task it reports how closely the molecule resembles a particular drug or how well it is likely to bind to a particular target protein. A contestant submits a molecule, reads the score, and edits the next molecule accordingly.

What makes the test distinctive is the cap on how often you may ask. In real drug discovery, evaluating a single molecule is itself expensive, so the value of an algorithm turns on how few questions it needs before it lands on something good. The oracle may therefore be called only 50 times, and the area under the curve of the running best score is measured and summed across the 23 tasks. The faster a method reaches a good molecule, the larger the sum.

Reports of LLM agents far outrunning classical algorithms on this test have been arriving steadily. The one the paper names first is SEISMO, a workflow built on Claude Opus 4.5, which reported a summed AUC of 18.12 at 50 calls in its August 2026 v3. REINVENT, a reinforcement learning method from 2017, earns 7.06 on the same budget, so the gap is more than double. Work reporting 15.42 under a 1,000-call setting is cited alongside it.

The exam paper, however, carries quite a lot of readable text. Fourteen of the 23 tasks spell out a drug name or a target name inside the task identifier itself. Sometimes the drug to be found is written out in full, as in celecoxib_rediscovery; sometimes the abbreviation for the target protein is the whole identifier, as with drd2 and jnk3. A model that read the chemical literature on the open web during pretraining can summon a near-correct molecule from that name alone. The standard prompt then adds the oracle description and the reference compound on top. The reference compound is the exemplar molecule the task wants you to resemble, which on a similarity task amounts to coordinates next to the answer.

The researchers stood up an agent of their own and reproduced the standard condition exactly. Running GPT-4o mini with the task name, the oracle description and the reference compound all in the prompt, they measured a summed AUC of 14.44. The 18.12 in the previous section came from a different model and a different architecture; every one of the four numbers that follow is this single agent, measured again with the conditions switched. Two things were switched: whether the task name and description appear, and whether the reference compound appears.

Once the two axes are split, the asymmetry is plain. Swapping the task name and description for a fake number takes the score from 14.44 down to 13.98, a loss of 0.46, and erasing the name when the reference compound is already gone costs a further 1.22. Removing the reference compound, on the other hand, wipes out 8.37 and 9.13 points at once. Averaged out, the reference compound accounts for 8.75 points and the task name for 0.84. The short version, that the score collapsed once the names were hidden, is half right. What was holding the score up was the exemplar molecule.

The researchers slipped one more condition in between. Here the reference compound itself is withheld and only the shape of the oracle is disclosed, along the lines of maximize ECFP4 Tanimoto similarity to a hidden reference. That corresponds to knowing what kind of problem is being optimized without knowing what the target is, and it scored 5.39, which is 0.54 above the fully anonymous 4.85. Knowing the form of the problem is worth about that much.

None of this rests on a single run. On top of the 2x2 across all 23 tasks, eight representative tasks were repeated with three seeds. The standard condition beat the fully anonymous one on eight of the eight, with a one-sided sign test p-value of 0.0039. The average drop was 3.78 points, a 68% reduction.

If the reference compound explains most of the score, how many points would the reference compound alone collect with no LLM anywhere in the loop? The retrieval-only baseline is the device built to answer that. It finishes in three steps. First it submits the reference compound that came in the prompt. Then it submits up to five predetermined seed molecules. From there it takes whichever molecule has scored highest so far, uses RDKit to change one atom, and sends the variants out in a fixed order. LLM calls: zero. The same input always produces the same sequence of submissions.

That procedure scored 12.16. Set against the 14.44 the agent reached with every field filled in and the same 50 calls, that is 84%. With no model, no reasoning and no planning, simply handing back what it was given filled most of the report card. Whatever the agent did across its 50 calls, the part of it that converts into points comes to roughly the remaining 16%.

There is a reason the procedure climbs so close to the agent. In its final refinement phase, the researchers' own agent fills leftover oracle budget with deterministic one-atom additions whenever the LLM returns proposals that are malformed or duplicate earlier ones. A piece of what the retrieval-only baseline does is therefore already built into the agent. Beyond that, seven of the 23 tasks reach roughly 0.98 AUC the moment the reference compound and the seed molecules go in. Because per-task AUC is the trapezoidal area over the 50 running-best points divided by 50, its ceiling is 49/50, which is 0.98. Those seven tasks are already at the ceiling, and the remaining calls have nowhere to lift them.

The paper sets up a positive control to keep the balance. MolLEO-v4 is a population-based genetic algorithm carrying an LLM batch-mutation operator, structured memory and adaptive planning, and it never pulls in a reference compound. That agent took 7.39 at 50 calls, past REINVENT's 7.06, and climbs to 10.24 when the budget is raised to 100. Strip out the leaked information and meaningful optimization remains possible. The claim here is not that LLMs are useless but that today's measurement cannot tell skill apart from access to information.

4.1A Procedure That Learns Nothing Outscores the Agent

The second track is a setting that looks harder than PMO. ProteinLeadOpt-50 attaches five starting molecules to each of 10 medically important protein targets for 50 pairs, and gives each pair 30 oracle calls. Scoring is a composite: the improvement in the AutoDock Vina docking score, plus a drug-likeness measure (QED) at a weight of 0.5, minus a penalty for drifting too far from the starting molecule. Against that, the researchers stood up a baseline with nothing in it worth calling learning. The policy picks one aromatic hydrogen position and attaches one of eight groups at random, drawn from F, Cl, Br, CF3, OMe, Me, OH and NH2. A fixed rule, in other words a random walk.

The table below puts that random walk up against an LLM agent, the genetic algorithm GraphGA, and plain prompting of GPT, over the same 50 pairs. All four rows are the matched single-seed comparison, and the figures from running more seeds and averaging appear in the paragraph after the table.

A lower docking score means tighter binding, so in the docking change column a larger negative number is better. The procedure that attaches substituents with no strategy at all sits at the top of all three numeric columns. Random walk beat the LLM agent on 33 of the 50 pairs, and averaging over three seeds moves its numbers up rather than down, to a docking change of -1.11±0.04 and a composite score of 0.636±0.008.

The reason lies with the scorer. Hang a chlorine, a bromine or a CF3 on an aromatic position and the Vina score typically falls by 0.5 to 1.5 kcal/mol. The scaffold stays put, so the similarity penalty is avoided too. A 30-call budget combined with this composite score systematically rewards local substitution, and points accumulate without any particular strategy. Where PMO's problem was the hints written on the exam paper, the problem here is the plainness of the scoring itself, which looks like the same illness presenting a different symptom. The paper pins this result to the specific combination of AutoDock Vina, the composite score and the 30-call budget, and writes that an evaluation with more discriminating power would need a stronger scorer such as FEP or Boltz-2, target selectivity conditions, and novelty criteria that demand a change of scaffold.

What the paper prescribes is not a better model but a more honest measurement. There are two prescriptions, and neither needs extra training or special equipment, only a few conditions laid on top of existing code. The $0.63 that the whole check cost in API fees is the evidence for that.

• A retrieval-safe protocol: when results are published, report three numbers side by side, for the standard prompt, for the condition that discloses only the oracle form, and for the fully anonymous condition. Placing the three cells together separates the points earned from molecular identity, the points earned from the shape of the problem, and the points earned from pure feedback optimization. Where the setup shows the reference compound outright, the paper adds that the retrieval-only score obtained without an LLM belongs in the same place.
• A random-walk floor: on docking-based tasks, run a strategy-free variant with the same starting molecules and the same call budget, and publish that score alongside. An agent that cannot clear this floor has no standing to claim it learned anything. Reporting docking change, success rate and composite score together with QED, similarity and scaffold preservation, and stating which substituents were allowed at which positions, lets another group rebuild the same floor.

The evaluation data companies work with is in no better shape. The numbers that justify picking a model or reporting an improvement are produced by an evaluation set, yet what else rides along inside that set is seldom written down anywhere. An item name that hints at the answer, an example attached to the prompt that sits near the answer, a question pulled from public data that already went into the model's pretraining, all land in the same place. How much leakage of this kind actually lifts a score came up in Data Leakage: The Flaw That Makes an AI's Analysis Score Higher, and the case of a broken exam paper was covered in On the AI Exam, the Same Row Carries Two Different Answers.

Carried over to an organization's own evaluation data, the two checks in this paper turn into three questions.

• Does the score hold when the names and descriptions of the evaluation items are erased? A name-blinded condition takes about one line of code, and if the score stays put you have evidence that the names at least were not hints.
• How far from the answer are the examples bundled into the prompt? Without a measurement taken after removing the exemplar, the performance on the table has the exemplar's contribution mixed into it.
• What does a procedure that learns nothing score on this evaluation set? Without that floor there is no way to say where the model's score started climbing from.

All three ask about the evaluation set rather than the model. That is also the place the phrase AI-Ready Data points at. Only when the conditions a value was produced under stand next to the data itself does a score measured on that data come with an account of what it proves.

Thank you for reading this far. The full paper is at arXiv:2611.08522. If you have ever blinded a name or removed an example from an evaluation set you use, we would be glad to hear how far the score moved.

• 1.Qin, J., Luo, Z., Chen, C., & Xu, Z. (2026). “What Are LLM Agents Actually Learning on Low-Budget Molecular Optimization Benchmarks?” arXiv:2611.08522.
• 2.Krüger, F. P., Hunklinger, A., Wolny, A., Adler, T. J., Tetko, I., & Villalba, S. D. (2026). “SEISMO: Explanation-Aware, Trajectory-Conditioned LLM Agents for Sample-Efficient Molecular Optimisation”. arXiv:2602.00663.
• 3.Gao, W., Fu, T., Sun, J., & Coley, C. W. (2022). “Sample Efficiency Matters: A Benchmark for Practical Molecular Optimization”. NeurIPS 2022 Datasets and Benchmarks Track.