Executive Summary

Three weeks ago we devoted a full report to Together AI, a company that rents GPUs by the hour, earning an $8.3B valuation. The conclusion was this: once models commoditized, the money flowed down to the layer beneath them — the physical compute needed to run models. But the flow did not stop there. This summer, the same capital climbed one more layer up the stack. Not the bottom tier of GPU rental, but the layer above it that actually executes models and serves them as APIs: the inference serving layer.

The numbers lay the move bare. Fireworks was valued at $4B in its October 2025 Series C; in July 2026 it raised $1.5B in a Series D that closed at $17.5B — roughly 4.4× in nine months. Baseten was at $5B in January 2026, then rose to $13B (with some tranches at $11B) after raising $1.5B in a June Series F — 2.6× in five months. Set against Together AI's 2.5× over sixteen months, what stands out is less the multiple itself than the speed of arrival. Capital enters the serving layer faster than it entered the hardware-rental tier beneath it.

Captured on a single page, this terrain resolves into three layers. At the bottom sits the hardware tier that owns GPUs and rents them by the hour (neoclouds like Together AI). Above it sits the software tier that loads models and optimizes them with batching, caching, and routing to serve as APIs (Fireworks and Baseten, the subjects of this report). And at the top sits the data-and-application tier that actually makes that inference worth paying for. What capital re-priced this time was precisely the middle layer.

The investor rosters show this re-pricing was no accident. The Fireworks round was led by Atreides Management, Index Ventures, and TCV, with Nvidia participating as a strategic investor. Nvidia holds a dual position — GPU supplier and investor at once — a point that returns in the Section 5 counterarguments. The Baseten round was led by Altimeter, Conviction, and Spark. Worth noting: it was Baseten's fourth funding round in eighteen months, a micro-signal of just how fast and how repeatedly capital pours into the serving layer.

Two premises underlie capital's climb into the serving layer: first, the commoditization of models; second, the collapse of unit prices. Interlocking, the two produced a single judgment — that what carries value is no longer the model itself.

2.1Open weights closed the gap, but they are not still closing it

Open-weight models — Llama, Qwen, DeepSeek chief among them — have narrowed the performance gap with closed frontier models to a low single-digit percentage. By the Stanford AI Index, the benchmark gap between the top open-weight and closed models stood at roughly 3.3 percentage points in March 2026. In coding, the gap has all but vanished. At that level, for most practical work "which model" is no longer the decisive variable — the commoditization premise holds.

The commonly repeated claim that "the gap keeps closing," however, is inaccurate. The same metric read 0.5pp in August 2024, so March 2026's 3.3pp is actually a slight widening. Epoch AI's tracking agrees: the top open-weight models trail the closed frontier by about four months, and that lag grew slightly from three months at the end of 2025. In reasoning tasks, closed models still hold a 3–8pp edge. So the precise statement is this: "the gap is small enough — low single digits — for commoditization to hold, but the trend is not one of continued closing."

2.2Prices collapsed — so why does capital pile in?

Token prices have fallen by the high-90s percent over the past three years (94.5–99.7%, depending on the estimate). GPT-4-class performance that fetched $30 per million tokens in early 2023 had dropped into sub-dollar territory by 2026. Below is that decline reduced to a few milestones.

▲ Milestones in the collapse of token prices | Source: Stanford HAI AI Index, Epoch AI (LLM inference price trends). Prices vary by model and conditions, so these serve only as milestones.

If prices fell that far, you would expect the serving business to shrink. In practice the opposite holds, because falling unit prices are overwhelmed by surging volume. Goldman Sachs Research projects total token consumption growing 24× between 2026 and 2030, reaching 120 quadrillion per month. Each token gets cheaper, but the count to be processed explodes, so the total serving market grows. The reversal of the training-versus-inference compute split follows the same logic: inference's share of AI compute spend is projected to flip from about 33% in 2023 to 70–80% by end-2026 (estimates vary widely across institutions, so we fix only the direction and cite a single representative figure).

The divide runs like this: training concentrates and fixes on a handful of large model companies, whereas inference demand disperses and explodes across every application. So capital began betting less on "who builds the next frontier model" than on "who runs that exploding inference cheaply and fast." That is why the serving layer became the new battleground.

A natural doubt arises here. If inference serving is, at bottom, "running a model on a GPU," isn't it just GPU rental with a thin shell around it? If so, a $17.5B price tag is nothing but froth. But serving optimization is an engineering discipline with its own, independent difficulty. You have to grasp that fact to judge the character of these valuations.

3.1Serving is systems engineering, not "running a GPU"

What made inference serving fast and cheap over the past few years was a body of systems research. Rather than reciting numbers from individual papers, the lineage below shows what is hard, and why.

• PagedAttention (vLLM) — manages the KV cache like an operating system's memory pages, cutting waste and lifting throughput. The emblem of the idea that "serving = mere execution" is false — that it is a distinct system design.
• Continuous batching — schedules requests dynamically at the token level rather than in fixed batches, maximizing GPU utilization. The key lever that separates throughput and cost.
• Speculative decoding — a small draft model proposes several tokens ahead and a large model verifies them, cutting latency. The technical substance behind the "speed" serving vendors sell.
• Multi-LoRA serving (the S-LoRA family) — stacks hundreds to thousands of per-customer adapters on a single base model and serves them concurrently and economically. This one you will meet again in Section 4.
• RadixAttention (SGLang) — reuses the prefix cache to accelerate structured generation and multi-call workloads.

Keep the last item — multi-LoRA serving — in mind. This technique, which loads a different adapter per customer on one base model and runs them all at once, is precisely the mechanism that serves "models specialized on customer data" cheaply. Section 4's 95% figure does not hold economically without it. Serving optimization (Section 3) and data specialization (Section 4) join into one at exactly this point.

3.2Optimization itself earns standalone venture value

The market's answer to "is serving optimization really that hard?" arrived as capital. In January 2026, the core development team behind the open-source serving engine SGLang spun out as an independent company called RadixArk, earning a valuation of roughly $400M in a round led by Accel. Independent of any hardware rental, it is direct evidence that "the technology that makes serving fast" carries venture value of its own. The scale of the open-source ecosystem backs this up: vLLM has become a de facto industry standard, drawing 75,000 GitHub stars and more than 2,000 contributors.

The operating scale of commercial serving vendors is a long way from the "thin shell" suspicion, too. Baseten says it handles more than a billion inference requests a day across 87 clusters and 18 cloud environments (company-reported). Orchestrating latency, cost, and regulatory requirements simultaneously across multiple clouds is a different order of problem from renting a single GPU. Speed, multi-LoRA serving, routing, compliance, multi-cloud — these five make up the serving layer's moat.

Where the serving layer's value truly comes from — the operators themselves offered the answer as a metric. In its Series D announcement, Fireworks disclosed that the tokens it processes daily had grown roughly 3×, from 15 trillion to 40 trillion. That much is a story about scale. The decisive sentence is the next one: of those 40 trillion tokens, more than 95% originate from models specialized on customers' proprietary data and optimized for specific tasks (company-reported; no third-party audit confirmed).

Why this figure matters becomes clear the moment you picture the opposite. Had most of a serving vendor's traffic been general-purpose open weights run as-is, the vendor's role would be close to "a proxy that switches on a free model anyone can download" — and that commoditizes easily. But 95% of the actual traffic comes from models each customer tuned on their own data. So what the serving vendor gets paid for is not "the act of switching on a model" but "the ability to run hundreds or thousands of models specialized on each customer's data, economically, on one infrastructure."

And what technically underpins that ability is the multi-LoRA serving flagged back in Section 3. Without this structure — stacking a different adapter per customer on one base model and serving them concurrently — the 95% of custom traffic does not hold economically. The causal chain, put plainly: open weights became free → so value migrated to "what data you tuned on" → and multi-LoRA serving, which runs those tuned models cheaply, became the serving vendor's real product.

It would be a mistake to read the argument so far as "so climb aboard inference serving." These valuations face real counterarguments in several directions, and only by laying those alongside does the picture stay honest.

5.1The overvaluation case

The macro figures are uncomfortable to begin with. Global AI infrastructure investment is approaching $400B a year, while enterprise AI revenue sits at only about $100B. That gap is what feeds the "capacity bubble" diagnosis, the sense that capacity is running well ahead of demand. Surveys find 40% of fund managers view AI stocks as overvalued. There are individual cases, too. Chip company Cerebras priced its 2026 IPO at roughly 95× its 2025 revenue (though that is the hardware tier, to be distinguished from the software serving tier of Fireworks and Baseten). Fireworks' $17.5B is itself a revenue multiple of about 17.5× against its company-reported $1B ARR — clearly optimistic pricing.

The character of the round bears noting as well. As Section 1 described, Nvidia joined the Fireworks round as a strategic investor — and Nvidia holds a dual position: the supplier that sells the GPUs, and the investor that funds the companies serving models on top of those very GPUs. Because it is investing directly in an ecosystem that sustains demand for its own chips, this valuation may embed not purely financial pricing but the supplier's strategic interest. That is why the $17.5B figure should not be read solely as the market's cool assessment, and why caution is warranted before generalizing from one headline multiple to the price of the whole serving layer.

The structure of the Baseten round emits a subtle signal too. Its valuation was a split tranche — $13B and $11B — suggesting investors differed on timing of commitment and on what they were willing to pay. It can be read as a micro-sign that pricing power is beginning to wobble in an overheated market. That said, there is a decisive difference from the dot-com bubble: back then Cisco's P/E was 472×, whereas Nvidia today stands at 44×, resting on an actual earnings base. Rather than the binary of "bubble or not," it is more accurate to ask "how heavily is this leaning on optimistic assumptions?"

5.2Commoditization pressure hits the serving layer too

The more fundamental counterargument is that the serving layer itself can commoditize. If the hyperscalers absorb open-weight hosting and serving optimization, independent vendors' premium thins. The academic diagnosis lands on this point. One multi-method evaluation of the AI bubble (arXiv 2606.01575) holds that risk for inference startups concentrates "when revenue grows fast but gross margins collapse beneath inference costs, when customer churn is high, or when model performance commoditizes." All three conditions are realistic threats to serving vendors.

There is a reason we keep tracing capital's movement — from GPU rental through to inference serving. The endpoint of this flow touches the work of teams who handle data. The reasons are essentially four.

First, as interim evidence of a market reset. The claim that "models are the commodity, data is the differentiator" has often sounded like wishful thinking from data companies. This time, though, the flow of infrastructure capital and the serving vendors' own metrics back it up. However fast a serving vendor runs, what it ultimately runs are models fine-tuned and grounded on customer data (Fireworks' 95% is the measured proof), and that is the point where market demand for verified, refined training and tuning data shows up as a capital flow.

Second, that data quality is the upstream bottleneck of inference economics. However efficient multi-LoRA serving is, if the data that trained those adapters is stale or corrupted, the output has no value. Serving optimization solves the problem of "running cheaply," but "what is worth running" is decided by data quality. At the very front of the chain that runs from the properties of training data, through the model's internal representations, to the value of inference output, sit the questions of data freshness, accuracy, and representativeness.

Third, that the enterprise's operating question is changing. Once open weights and cheap serving become universal, competitive advantage shifts from "which model you use" to "how well you refine, verify, and keep current your own data for tuning and grounding." The further serving prices fall, the more the room for differentiation is left to data alone. This is also the ground that justifies the ROI of investing in data preparation and validation pipelines — because when the price of models converges on zero, the only remaining justification for spend sits on the data side.

Fourth, as a coordinate for what comes next. Where the capital climbing GPU → serving → data ultimately settles, we intend to observe in advance from the top of the stack (data and grounding). The open question this report poses is this: in a world where serving has fully commoditized, how wide will the output-value gap grow between "inference grounded on verified, fresh data" and "inference that is not"? Quantifying that gap, we believe, is the problem data teams must solve next.

Press · Primary (valuations · operating metrics)

• 1.Fireworks AI. (2026). "Series D Announcement." Fireworks AI Blog.
• 2.CNBC. (2026-07-16). "Fireworks AI valued at $17.5 billion in Nvidia-backed round." CNBC.
• 3.Value Add VC. (2026). "Fireworks AI Valuation 2026: $17.5B Series D and the $1B ARR Inference Business." Value Add VC.
• 4.Baseten. (2026). "Announcing our Series F." Baseten Blog.
• 5.PYMNTS. (2026-06-22). "Baseten Nears $1.5 Billion Funding Round as Inference Demand Surges." PYMNTS.
• 6.TechCrunch. (2026-07-01). "Neocloud Together AI raises $800M, leaps to $8.3B valuation." TechCrunch.

Serving-Layer Ecosystem

• 7.Yahoo Finance (via TechCrunch). (2026-01). "Sources: Project SGLang spins out as RadixArk with $400M valuation as inference market explodes." Yahoo Finance.
• 8."vllm-project/vllm." GitHub (open-source serving engine repository).

Market · Data

• 9.Epoch AI. (2026). "Open vs. closed model capability gap (ECI)." Epoch AI Data Insights.
• 10.Epoch AI. (2026). "LLM inference price trends." Epoch AI Data Insights.
• 11.Stanford Institute for Human-Centered AI. (2026). "The AI Index Report 2026." Stanford HAI.
• 12.Grand View Research. (2026). "Artificial Intelligence (AI) Inference Market Report." Grand View Research.
• 13.Goldman Sachs Research. (2026-05). "AI Agents Forecast to Boost Tech Cash Flow as Usage Soars." Goldman Sachs.

Background · Counterarguments

• 14.Crunchbase News. (2026). "North America Startup Funding And M&A Shattered Records On AI In Q2 2026." Crunchbase News (background statistics).
• 15.Forbes. (2026-05-14). "Cerebras Priced The Biggest IPO Of 2026." Forbes.
• 16.Forbes. (2026-07-18). "Open Weight Models Are Turning Inference Into A Control Point." Forbes.
• 17."Boom, Bubble, or Buildout? A Multi-Method Evaluation of Whether AI Is in an Ongoing Financial Bubble." arXiv:2606.01575.