Executive Summary

Key Figures

Four numbers sum up the study: who was most accurate, what was locked away, how much one round of grading cost, and how many people wrote the answer key. That last number is where this article begins.

Source: Benhenda, arXiv:2606.23032 (2026)

SpaceX filed its S-1 confidentially on April 1, 2026, made it public on May 20, and listed on June 12. The offering priced at $135 per share, with 555.6 million shares issued and a target raise of $75 billion. The first-day market cap opened at $1.77 trillion and crossed $2 trillion shortly after the open. By size alone, it is the largest IPO ever.

The filing itself was tricky. The S-1 folded in the full acquisition of xAI, completed in February 2026, so on a consolidated basis it bundled 2025 revenue of $18.67 billion with a net loss of $4.94 billion. Starlink was profitable, with $11.4 billion in revenue and $4.4 billion in operating income, while xAI spent $14 billion the same year and earned $3.2 billion. A profitable business and a loss-making one, acquisition accounting and governance — all overlapping inside one document.

An S-1 for an IPO is a different animal from the 10-K and 10-Q filings a public company files each quarter. On top of historical financial statements, it carries pro forma accounting, capital-raising structure, governance, and acquisition-risk disclosures, and it runs far longer. The existing financial-analysis benchmark, Finance Agent v2, covered only those periodic disclosures. Its retrieval was simple chunk search with no added context, so on a long document like an IPO filing it was easy to miss what mattered.

So this study built a separate exam tailored to IPO due diligence. Working from the SpaceX S-1, it generated 1,000 questions on financial-statement analysis, pro forma accounting, governance, capital structure, and risk disclosure. Of those, only 70 were released and 930 were kept private. The reason for locking them comes later.

Usually the grading criteria for a benchmark are written by hand, by experts. In a field like financial analysis, an accountant or analyst defines, question by question, the facts that an answer must contain. Finance Agent v2 had 537 questions written directly by experts. It is an expensive, slow way to work.

This study automated that work. It had AI generate the grading rubric itself — the criteria that define what each correct answer must include. Instead of writing the answers, people only review the machine-made criteria at the end.

The pipeline runs in five stages. First, several LLMs answer the same question on their own. Candidate facts are extracted from that ensemble of answers, and overlapping criteria are merged into one. Then comes a three-way audit.

Only criteria that pass the audit move on to human review, where the final rubric is locked in. Grading does not end in a single pass either. A stage that grades model answers with the generated rubric (the evaluator) and a stage that re-tunes the criteria from the grading results (the optimizer) interlock and repeat. The grading criteria are refined as they cycle through.

Retrieval was reworked too. In place of the existing benchmark's simple chunk search, the study applied contextual retrieval, which raises search accuracy by attaching surrounding context to short chunks. That difference mattered a lot for finding scattered evidence across a long IPO filing.

Good benchmarks carry a paradox. The better the criteria, the more you want to share them, and the moment you share them they break. Once questions and answers go public, the next generation of models sucks them in as training data. Then a model earns its score by replaying a memorized answer rather than by actually analyzing. This is benchmark contamination.

There is a close precedent. On SWE-bench, which measures code-fixing ability, OpenAI found signs that some models were copying the reference patch verbatim. It ended up halting those score reports. The benchmark was measuring memorization instead of the ability it set out to measure. The scores on a public benchmark lose trust as time passes.

This study's fix is simple. Only 70 of the 1,000 questions are released as examples, and the official ranking is tallied solely on the private 930. Because a model cannot see the answers in advance, its score moves closer to the result of actually reading and analyzing the filing, not to a memorized answer. The locked 930 hold up the trustworthiness of this benchmark.

The grading results cut against expectations. The top score did not go to the most expensive model. Alibaba's Qwen 3.7 Max came first at 79.4%, at a cost of $0.30 per query. Xiaomi's MiMo-2.5 Pro followed close behind at 76.8%, for just $0.05 per query.

The benchmark it was compared against, Finance Agent v2, topped out with Gemini 3.5 Flash at 57.9% and a cost of $2.51 per query. The leading models on the new benchmark pushed that ceiling up by more than 20 percentage points. MiMo-2.5 Pro delivered higher accuracy at roughly one-fiftieth of Gemini's cost.

What this reversal signals is clear. Skill as a financial analyst was decided not by a model's size or price, but by the retrieval architecture that finds evidence in long documents and the quality of the criteria that grade the answers. The same model can score differently depending on how it searches and what it is graded against.

This story is not only about financial analysis. The same question follows everywhere agents start standing in for real work. Who, and with what, grades whether the model's result is right? If the grading criteria are slack, every score and ranking built on top of them stands on sand.

So the trustworthiness of evaluation starts not with the model but with the grading data. If a key fact is missing from the criteria (omission), if a plausible falsehood is mixed in (hallucination), if the same thing is measured several times (duplication), the score fails to point at ability. That is exactly why this study audited all three explicitly. Without managing the quality of the answer key, the grading results cannot be trusted either.

The "publish it and it gets contaminated" paradox runs along the same line. The evaluation criteria are a data asset, and an unmanaged asset rots. Auditing for omissions, hallucinations, and duplicates, and locking part of it away to prevent contamination, are all matters of data quality. The question of whether you can trust an AI loops back, in the end, to whether you can trust the data that graded it.

R.1Academic Papers

• 1.Benhenda M. (2026). "IPO Finance Agent: Evaluation of LLM Financial Analysts beyond Finance Agent v2, with Automated Rubric Generation — the Case of the SpaceX (SPCX) IPO." arXiv:2606.23032.

R.2Official Documents & Press

• 2.Vals AI. "Finance Agent v2 Benchmark." (Existing financial-analysis benchmark, 537 expert-written questions).
• 3.OpenAI. "Introducing SWE-bench Verified." (Benchmark contamination case — reference-patch copying found).
• 4.U.S. SEC / SpaceX. "SpaceX Form S-1 Registration Statement." (2026, SpaceX IPO filing).

Thank you for reading. Each time you meet a result from an AI, the habit of also asking "what graded that score?" is what will let you tell a good-looking report card apart from a trustworthy one. If you have thoughts or counterarguments on this topic, we would love to hear them.

Pebblous Data Communication Team
June 24, 2026