Executive Summary

Autoresearch is not a heavy framework. The code Karpathy released on March 7, 2026 is 630 lines of Python, and the core idea fits in a sentence: an agent makes one change to the code, runs a time-boxed experiment, measures a validation metric, keeps the change if things improved and reverts it if they got worse — then repeats. Karpathy ran this loop 700 times over two days on a single GPU, found 20 genuine improvements, and, applied to a larger model, lifted training speed by 11%. The loop caught defects a human had missed by hand for two decades, such as a missing normalization step.

What made the loop catch on so quickly is that it worked beyond Karpathy's own experiments — in someone else's production code. Shopify's Tobi Lütke reported running the same pattern on his company's template engine, Liquid, nearly halving a rendering speed that humans had already tuned and cutting memory use substantially. The result kept only the changes that passed every unit test across hundreds of automated experiments. That is a second piece of evidence that, in real product code rather than a toy model, the loop found improvements seasoned developers had missed. So the next question follows naturally: what if the artifact were not code, but prose?

The direct source for this report is not the code. It is Ole Lehmann moving the same pattern to a content skill. He started from the fact that his landing-page copy skill failed 44% of its quality checks, then had an agent revise the skill's prompt, compare scores, and repeat keep-or-rollback unattended. After four rounds and roughly $15 in cost, the pass rate reached 92%. Three changes survived: a rule to put a concrete number or outcome in the headline, a banned-buzzword list, and — more powerful than any rule — a worked example dropped directly into the prompt. One change was rolled back: a rule that capped character count too tightly, because the copy thinned out and the call to action weakened.

The Five Pieces That Make Up the Loop

Break the pattern into parts and five remain: Mutation, which proposes one candidate change at a time; Execution, which produces the artifact under that change; the Judge, which scores the artifact; the Orchestrator, which keeps the change if the score rises and reverts it if the score falls; and the Changelog, which records what was changed and why. The real product here is not the prompt — it is the Judge. If the Judge wobbles, the whole loop optimizes in the wrong direction.

That is why Ole's checklist design rules matter. Keep items as three to six binary questions: Does the headline contain a concrete number? Is it free of buzzwords? Is the call to action specific? Consistent judgment, not precise scoring, is the design principle. Too few items and you miss quality; more than six and the model games the item scores while real reader quality drops — overfitting. An evaluation function should be a floor that blocks low quality, not a ceiling that suppresses creativity.

The Academic Lineage: Not a New Invention, a Simplification

The discipline of "one change at a time, then keep or roll back by score" is a practical simplification of a line of work already validated in academia. DSPy, which compiles and optimizes prompts as declarative modules, raised accuracy by 32% on multi-hop reasoning and 45% on math reasoning. OPRO, where a meta-LLM iteratively generates new prompts; EvoPrompt, which evolves prompts through crossover and mutation; and TextGrad, which treats natural-language feedback like a gradient, all belong to the same family. Effect sizes range widely with the task — from 7% to 45% — and show up most strongly in structured reasoning. Ole's reported +36pp sits at the high end of that range, but it bears repeating that it remains a single self-report.

The process Pebblous uses to produce a deep-research report is not a single block of prompt but a pipeline of agents working in sequence. When a topic comes in, duplication and value are reviewed in parallel; a human confirms once; then planning begins. From there the work forks three ways to research academic papers, industry developments, and data simultaneously, and a synthesis stage binds those into one document. Writing, a second human review, five stages of quality reinforcement, English rewriting, SEO and social, and finally publishing follow in turn. The very article you are reading went through that process.

Two design choices are worth noting. The first is tiered model assignment. Only the reasoning-heavy nodes — planning, paper research, synthesis, writing — use a top-tier model, while collection, execution, and publishing are handed to a lower tier. The top and lower tiers differ by roughly fivefold in price per output token, yet the gap on coding benchmarks is only a few percentage points. So routing only the heavy-reasoning nodes to the top tier saves 40–60% versus a single model while holding planning quality steady; add caching and batching and the savings grow further. This is Pebblous's implementation of the Autoresearch idea: plan well once, then execute cheaply, many times.

The second is the human confirmation gate. Right after the initial review and right after the first draft, the pipeline stops and waits for human judgment at two points. Reading that as unfinished automation is a mistake. Pebblous's aim is not maximal automation but measurement-driven iteration that keeps human judgment as a core gate. The gate is not a weakness; it is intended design.

We Already Have a Judge

Inside the pipeline, two tools already run that correspond to Autoresearch's Judge. ko-prose-humanizer catches the tells of AI-written prose across eleven markers and scores them out of 110 — patterns like em-dash restatement, a monotony of nominal sentence endings, meta-announcements, and the contrived pivot to the company at the end of a piece. If the score crosses a threshold, it forces an automatic correction; if it falls in the passing band, it lets the text through. seo-check audits four layers — meta tags, OG and Twitter cards, JSON-LD schema, and Search Console — and everything must pass before the work moves on.

Put the two systems on the same axes and what emerges is not a ranking but a difference in character. Autoresearch is a tool for automatically drilling deep into a single prompt; the Pebblous pipeline is a process for completing one article broadly with human judgment woven in. The table below lays out how the two diverge across eight axes.

The difference between the two systems comes down to a single sentence: Autoresearch drills narrow and deep through automated optimization, while Pebblous completes one article broadly, layering human judgment and multiple languages onto a single pass. One is good at fixing the evaluation function itself; the other is good at carrying one long article through to the end, fitted to its context. The two are not competing over the same job.

The real value of the comparison is in exposing each system's blind spot. Each one misses something precise, in the shadow of what it does well.

To put it plainly: Pebblous measures, but it does not use that measurement to fix the skill. We score the prose of a single article, but the step that takes that score and mutates and validates ko-prose-humanizer itself into a better version is empty. Conversely, Autoresearch optimizes a narrow checklist quickly, but the grain of a narrative or an error of fact that the checklist cannot catch will never come into view. Each empty space is precisely the other's strength.

If we already hold half of the evaluation function, the way to fill the other half is not to build from scratch but to close the loop around the judges we already have. No heavy infrastructure is needed — remember that Ole's case cost about $15. The five steps below add quantitative scores at the points where humans already look, without removing the human confirmation gates.

Cost and Failure Modes

The scale is small. You start not with a heavy platform but with a handful of binary checklists, 10–20 golden inputs, and the discipline of comparing scores after one change at a time. The failure modes, though, are clear. First, overfitting: only the items the evaluation function catches improve, and the articles all start to look alike. That is why the evaluation function must be a floor, and the golden set must be refreshed each quarter to prevent the criteria from aging. Second, judge reliability: LLM judgment agrees with humans in the low 80s on general tasks but drops to the mid-to-high 60s in specialist domains, and the verdict shifts when you merely reorder the answers. So the core items keep a human in the loop, and a correction that swaps order and averages is needed.

Primary sources: the Autoresearch original and its application

• 1.Lehmann, O. (2026). "Karpathy's Autoresearch method for Claude Skills." Threads @itsolelehmann. (Original 56%→92% self-report.)
• 2.Karpathy, A. (2026). "autoresearch." GitHub (MIT License, released 2026-03-07).
• 3.Agent Cookbook. (2026). "How to 10x your Claude Skills using Karpathy's Autoresearch method." (Reproduction details.)
• 4.VentureBeat. (2026). "Andrej Karpathy's new open-source autoresearch." VentureBeat; cross-checked with Fortune and NextBigFuture (700 experiments, 11% improvement).

Academic: automatic prompt optimization and LLM-as-Judge

• 5.Yang, C. et al. (2023). "Large Language Models as Optimizers (OPRO)." arXiv.
• 6.Khattab, O. et al. (2023). "DSPy: Compiling Declarative Language Model Calls." Stanford NLP. (HotpotQA +32%, GSM-8K +45%.)
• 7.Zheng, L. et al. (2024). "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." NeurIPS.
• 8.Evidently AI. "LLM-as-a-judge: a complete guide." (Binary vs. scale reliability, +20pp.)
• 9.Galileo AI. "LLM-as-a-Judge vs Human Evaluation." (General 80%+, specialist domains 60–68%.)

Industry, market, and tooling

• 10.Caylent. (2026). "Claude Haiku deep dive: cost, capabilities, and the multi-agent opportunity." (Tiered models, routing savings.)
• 11.Braintrust. (2025). "Best prompt evaluation tools 2025." (Eval-tool ecosystem; Series B $80M, $800M valuation.)
• 12.Precedence Research. "Generative AI in Content Creation Market." ($19.75B → $143.09B, CAGR 21.9%.)

Note: the 56%→92% figure is Ole Lehmann's single self-report, with no independent reproduction; every in-text citation marks it as a self-report. Model prices and performance are described by tier and price ratio to avoid naming a specific version. Some market sizes vary by source depending on the definition (broad vs. narrow scope).