Executive Summary

BoSS was posted to arXiv at 19:47 UTC on 27 October 2026. There are five authors: four at the Center for Computational Mathematics at the Flatiron Institute in New York, one at Cornell's Department of Statistics and Data Science. The work was supported by the Simons Foundation and the Flatiron Institute's Scientific Computing Core.

Start with its status. This paper is a preprint. The review link inside it is still the placeholder openreview.net/forum?id=XXXX. It wears the dress of a journal submission and has not been accepted anywhere yet. Every figure quoted below is from this version (v1) and could change if a revision appears.

What is open and what is not also needs separating. The data is CC BY 4.0; the code and the trained models are Apache 2.0. The metric code installs with a single pip install boss-bench, and three trained models (loss-fraction prediction, heat-load sampling, inverse design) sit on Hugging Face with ONNX exports included. The 3TB data itself, by contrast, has to come through Globus and requires an institutional, Google or ORCID login. And the code that generated the data has not been released. Section 8 of the paper records only that it will be "available upon publication". What the license says is open and what can actually be laid hands on are not the same thing.

1.1How Far Does the "First" in BoSS Actually Reach

The shortest way to introduce BoSS is "the first large-scale dataset", but the paper does not put it that way. The abstract says "the first large-scale dataset of guiding center simulations", and the contributions list in §1 narrows the claim once more to "the first publicly available large-scale dataset of stellarators with GC simulation data and trajectories". First along the axis of guiding center simulation, then, not first among stellarator datasets. Larger things already exist in the same field.

Drawn from the related-work section of the BoSS paper together with the primary source for each dataset. QUASR's size is given as a range because it is updated continuously, so 200,000, 300,000 and 370,000 all circulate depending on when a paper cites it. Landreman 2025 and BoSS sit on the same 2025–2026 stretch of the axis and are dropped to a lower line so they do not overlap.

Put the same thing in a table and the column BoSS added becomes obvious. On size alone, Landreman 2025's 200,000 gyrokinetic runs are larger, and attaching tasks and a leaderboard to a dataset is something ConStellaration did sixteen months earlier. One seat was left, and BoSS took it. Nobody had precomputed and published, at scale, where the fast ions leak away.

Taken from each dataset's paper and distribution page. Landreman 2025's "over 200,000 runs" is the paper's own wording, and the composition behind it (how many runs on which shapes) could not be confirmed in public material, so it is not broken down here. ConStellaration's leaderboard takes the boundary you submit and re-evaluates it with the benchmark code; the evidence for that is in section 6.1.

The parenthesis on the last row is the setup for what comes later. ConStellaration has a leaderboard and BoSS does not. Where the two datasets part company is not whether scoring calls a physics code but how far it calls one. ConStellaration's scoring stops once the equilibrium has been re-solved; BoSS solves the equilibrium and then traces the alpha particles again on top of it. The dataset that reaches for the more expensive measure is the one without a leaderboard. Section 4 is where that lands.

A stellarator is a fusion reactor that confines plasma in a twisted three-dimensional magnetic field. A tokamak drives a current through the plasma to produce part of its field; a stellarator produces confinement from the shape of the external coils alone. Not leaning on a plasma current is an advantage for steady-state operation, and the price is a far more complicated shape. In this field, then, the shape design is the whole contest, and it takes the form of an optimization problem with dozens of design variables.

What makes a design good? Several criteria compete, but one of them does not bend. Fusing deuterium with tritium produces a 3.52 MeV alpha particle, and that particle has to stay inside the plasma for the plasma to heat itself. A particle that escapes does no heating and strikes the wall, damaging the machine. So the key metric is loss fraction: the share of fast ions, the alpha particles, that leak out within a cutoff time. BoSS treats 10% or less as good performance.

The trouble is what that metric costs to measure. Getting a loss fraction means integrating the orbits of thousands or tens of thousands of particles through time. The orbits are chaotic, so a tiny change in initial conditions sends paths apart, and because only a finite number of particles is launched, measuring the same device twice gives slightly different answers. On top of that, nudging a design variable does not move the metric smoothly. It is the combination an optimization algorithm hates most: slow, noisy, and with gradients you cannot trust.

So designers reached for a different ruler. They optimized a cheap proxy such as quasisymmetry and pulled the expensive metric out only at the end, to check a finished design. The paper records the practice in two places. The abstract says that the key performance metrics derived from guiding center simulations are too costly to put directly in the design optimization loop and are "almost exclusively used as diagnostics", and the end of §3 in the body puts it more plainly.

"Correlate only imperfectly" is the sentence this whole report hangs on. Start measuring something cheap that resembles the real goal, and the model gets good at the thing being measured. It is the same structure as the reward hacking the Pebblous blog covered in Why AI Aces the Test but Can't Do the Job, which collected cases of models tuned to hit a cheap metric and falling apart on the actual task. In stellarator design that structure had already arrived, not in a model, but in how people design.

2.1Who Had Been Scoring With the Cheap Ruler

One thing catches the eye here. In November 2025, another paper with BoSS's first author on it put the problem of designing stellarators with generative models in front of the machine learning community. It is called Diffusion for Fusion (arXiv:2511.20445), and it scored the designs it generated against a threshold of "less than 5% deviation from quasisymmetry and the target characteristics". A cheap proxy metric, in other words.

Eleven months later the same research lineage released BoSS, and swapped the ruler on its inverse-design task for the expensive real metric. A generated boundary is rescaled to a reference size, the equilibrium is re-solved, the fast ions are traced again, and the resulting loss fraction becomes the score. The people who changed the ruler are the people who proposed the cheap one eleven months earlier. That is the part of BoSS this report found most interesting.

Put simply, here is what BoSS did. The calculation a designer used to run once, at the very end, has been run in advance for 180,000-odd devices. The pipeline has three stages: pick a source design and randomize its size and profiles, solve the magnetohydrodynamic (MHD) equilibrium, then trace fast ions inside that equilibrium. What is left at the end is a loss fraction, a heat-load pattern and trajectories.

The data generation procedure and cost description in §4 of the paper, redrawn as a diagram. Times are per device and assume one CPU and one GPU working together. The average is 15 minutes without saving trajectories and about 20 minutes with them.

The total came to 25,000 GPU hours on A100s, more than 500,000 CPU hours, and over 3TB of storage. Buying the same work on-demand from a public cloud would put the compute alone somewhere around $70,000 to $140,000 (KRW 100–200 million). That conversion rests on a stack of assumptions. Hyperscaler A100 pricing and specialist GPU-cloud pricing differ by more than a factor of two, and the Flatiron Institute runs its own cluster full time, so the actual outlay has nothing to do with this range. Treat it as an order of magnitude and no more. What is unusual is that the figures are written down at all. Survey nine public datasets of similar character and the ones that state their compute cost in hours are BoSS and OMat24, roughly speaking.

3.1The Number of Distinct Shapes Is Not 180,000

One number needs correcting here. 180,000 is the count of equilibria, not the count of distinct boundary shapes. That is exactly what stage ① does. Take one source device, redraw its minor radius somewhere between 0.4 and 2.4 m and its mean field somewhere between 4 and 10 T, swap the pressure and current profiles, and register it again. The same shape enters the dataset on several rows, differing only in size and profiles.

The paper knows this. When it splits train from test it says it filters out devices that share a surface shape and differ only in size or profiles. Leakage spotted and blocked, which is the right call for a dataset. And yet the number of unique boundary shapes appears nowhere in the paper. The blocking is written down; the amount blocked is not. It has the same shape as the near-duplicate problem Pebblous DataClinic keeps turning up in customer datasets, where the gap between row count and unique-row count moves performance figures wholesale.

Where the shapes came from is worth writing down too. BoSS did not build them from nothing. It drew candidates from more than 1.5 million existing public designs across QUASR, ConStellaration, Kappel et al. (2024) and the omnigenity database. What BoSS laid on top is four things: pressure and current profiles, the guiding center simulations, over 21,000 additional devices collected by active learning, and roughly 8,000 randomly generated rotating-ellipse devices. The stacking was possible because other people had published shapes first, and the layer BoSS added is evaluation, not geometry.

3.2No Guesswork, but Still Plenty of Choices

In a field where a physics code computes the labels, no human guess gets into the ground truth. On that count BoSS stands in a rare position. The case Pebblous covered in Drug Discovery AI Learns From Hydrogen Atoms X-Rays Cannot See was the exact opposite. There, where a hydrogen atom sits was decided by human rules and written in as a label, and the guess flowed straight into what the model predicted.

No guesswork does not mean no choices, though. BoSS's "ground truth" has a row of human decisions buried in it. The list is below, and each line changes what the label means.

• A cutoff time of 0.1 seconds. A particle that has not escaped by then counts as confined. Set the cutoff longer and the loss fraction rises.
• A loss surface at s = 0.98. A particle crossing that flux surface counts as lost. The prose describes the criterion in terms of the last closed flux surface (s = 1), but the line the code actually draws sits inside it.
• 15,000 particles per device, dropping to 6,000 when trajectories are saved. Birth surfaces are three: s = 0.1, 0.25 and 0.5.
• Collisions ignored. Particle-particle collisions are left out of the calculation. Real plasmas have them. The paper itself lists simulations with Coulomb collisions in §7 as a task for the next release.
• Devices whose equilibrium did not converge are discarded. What kind of design got thrown away leaves no trace in the data.
• One reference size for rescoring, ARIES-CS (minor radius 1.7 m, mean field 5.86 T). Every score quoted later is measured after rescaling to that size.

There are two ways to read this list. One is "so none of it can be trusted"; the other is "so the fact that it is written down matters". The second is the right one. No label is free of choices. Some labels have those choices documented and some do not, and BoSS is in the documented group. One item on the list comes back later. Trajectories were saved only out to 1 ms, while the cutoff that decides confinement is 0.1 seconds. One hundredth of the window.

This is where the setup from the end of section 1 pays off. ConStellaration, released in June 2025 by Proxima Fusion and Hugging Face, made this boast in its abstract: models trained on their dataset can efficiently generate novel, feasible configurations "without querying expensive physics oracles". Not having to call the physics code was the selling point.

What that sentence points at is generation. It means a trained model does not have to call a physics code while it is producing shapes; it does not mean nothing gets scored. In practice ConStellaration's leaderboard feeds a submitted boundary into the benchmark code, re-solves the equilibrium and recomputes the metrics. Entrants upload a boundary, not a score, and a pinned version of the code does the grading on the server.

BoSS's inverse-design task goes one stage further. When a model produces a boundary shape, that shape is rescaled to ARIES-CS size, the equilibrium is solved, and rather than stopping there, 5,000 alpha particles are traced again to produce a loss fraction (fewer than the 15,000 used when building the dataset). The expensive oracle that had been pulled out of the design process is put right back at the scoring step. Same field, sixteen months apart, two different institutions answering the question of how far to call the physics code differently, and the difference shows up as numbers.

4.1Same Model, Same Ruler, Different Conditions

BoSS entered two methods in the inverse-design task. One is a diffusion model (Diffusion), the other an optimization driven by a surrogate model (SurrogateOpt). Both take a desired loss fraction, shape, field and profiles as conditioning and return a boundary. The headline column of Table 1 is the one the paper calls the low-loss rate: "the fraction of designs whose traced loss fraction is below the target value", with the target set at 10%.

The diffusion model scored 43.6% under vacuum conditions. More than four usable designs out of every ten reads as a decent result. But the results section in §6.1 writes two more numbers alongside it. One is the value under different conditioning; the other is a baseline. The three only mean something read together. All three share a denominator: the share of designs whose loss fraction is under 10% when re-evaluated at ARIES-CS size.

Figures from Table 1 and the results in §6.1 of the paper, gathered onto one axis. The three bars share a denominator, and that denominator is written in the paragraph just above. The middle bar is not generated designs at all but the score of existing devices pulled from the test set, which comes to 12% if measured at their own original size. Widen the loss target to the full 2–20% range and the right-hand value becomes 7.2%.

The 43.6% on the left comes from conditioning on the property vectors of test devices whose loss fraction is already under 10%. That amounts to knowing a good answer, handing over its features, and asking for something like it. The 6.7% on the right comes from taking the conditioning of a random test device and overwriting only the loss target with 10%. Same model, same ruler; the only change is where the conditioning was drawn from. And that value sits below the 10.8% you get from pulling any device at random. Since the middle bar is the score available with no model at all, under the right-hand conditioning there is a reading in which the model is not failing to contribute but actively subtracting.

The paper explains the split in two places. The first is its reason for choosing the 43.6% conditioning that way. Draw the conditioning from a low-loss device and every other property arrives from that same device along with the loss target, so the request does not contradict itself. In the paper's own words, "this ensures the conditions don't preclude the possibility of generating devices with low loss fraction". The other side is a request built from an arbitrary device's property vector with one cell, the loss target, overwritten to 10%, so the target flies off on its own, out of step with everything around it.

The second is a measurement in Appendix D. Checking how faithfully the generated designs honor the conditioning they were given, the median absolute error on loss fraction is 0.026, with 2.0% on mirror ratio and 0.5% on aspect ratio. Small, taken overall. But the error is not uniform across the target range. Where the requested loss fraction is at or below 2%, the median signed error is +0.085, against +0.003 for mid-range targets. The appendix reads this as "at very low-loss targets, the model regresses toward typical devices", adding that the small overall error partly reflects how many of the conditioning rows describe devices of moderate rather than extreme performance. The further the request sits outside the distribution, the further the model retreats toward the average. That is the same place where the score collapsed when the conditioning changed.

What matters is that all three figures were printed side by side. Writing down 43.6% and stopping there would have left nobody any the wiser. The reason that did not happen is that the scoring rules contained a random baseline. With no baseline in the table, 43.6% is a good score; with the baseline in the table, 6.7% is a bad one. Two conclusions out of the same data, and which one you see is decided not by the data but by the scoring rules.

4.2In Fairness, What the Models Got Right

Stack up only the criticisms and the table gets misread. Table 1 has cells where the models clearly did well. Surrogate optimization beat the diffusion model under vacuum conditions at 52.6%, and it does not suffer the conditioning collapse at all. The diffusion model hits aspect ratio with a median error of 0.3%, and relaxing the mirror-ratio constraint by a factor of 1.1 brings 97.4% of its designs inside. Under finite-beta conditions the diffusion model climbs to 60.9%. The table below shows where the two methods differ in character.

Table 1 of the paper carries five metrics under two conditions, vacuum and finite beta, for ten rows in all; the table above takes the five of those where the two methods differ in character and adds one cost line from the body. The five rows left out are the two at the 15% low-loss threshold (vacuum 48.4% against 64.9%, finite beta 71.6% against 50.5%), the two finite-beta mirror-ratio rows (20.7% against 29.4%, 98.8% against 45.0%), and the finite-beta aspect-ratio error row (0.4% against 3.6%). All five run the same direction as the other condition's row for the same metric, so nothing in the conclusion changes. The paper reports the 15% threshold separately in order to count near misses that a local correction could push over the line. Training cost is about 80 hours on four A100s for the diffusion model and about 2 A100 hours for the surrogate. SurrogateOpt then ran its optimization repeatedly to produce 206 solutions with a predicted loss fraction under 10%.

One more cell belongs here that the table does not hold. The first question anyone asks about a generative model, whether it is simply copying the training data, gets a numerical answer in the paper. Appendix D measures the distance between each generated boundary and the nearest boundary in the training set, in units of minor radius. The median is 0.126; 35% exceed 0.2 and 20% exceed 0.3. Of the 4,420 designs evaluated, not one landed within 0.005 of a training boundary. The five designs the paper picks out in Figure 7 have loss fractions of 5.8%, 4.5%, 4.9%, 0.0% and 0.7%, and all five sit at least 0.17 minor radii from their nearest training boundary. The paper's summary is that the model finds well-confining designs mostly in the neighborhood of the training data without copying from it. The reading that discounts 43.6% and these numbers stand separately.

The two methods are in different businesses. The diffusion model spends heavily on training and then generates for essentially nothing. Surrogate optimization trains cheaply but has to spin up CPUs again for every solution it produces. If the job calls for tens of thousands of design searches the first is right; if a few hundred precise candidates will do, the second is. SurrogateOpt, which never hits the conditioning collapse, has a pitfall of its own instead: the optimizer can land in a bad local minimum of the modeled objective, or in a region where the surrogate generalizes poorly. It is a well documented problem with the method. The diffusion model samples where its models are reasonably confident and avoids the trap. Which is why the paper writes that practically, Diffusion is much easier to use than SurrogateOpt.

The paper declines to declare a winner, and names out-of-distribution generation as the homework still outstanding. The 6.7% is the size of that homework. It also notes a direction for doing the homework. The diffusion model's low-loss rate stopping where it did points to room for post-training, and reinforcement learning, online or offline, could raise the probability of sampling low-loss devices. Which is to say the current numbers are not being claimed as a ceiling imposed by the architecture.

The structure of validating generated designs back against physics shows up in other fields too. In AlphaFold Folded Proteins. Now AI Designs Them From Scratch. the final grade was pushing a generated protein all the way to synthesis in a wet lab, and SELECTBENCH was a case of re-measuring drug candidates on a new scoring axis and watching the existing ranking change. BoSS's 6.7% belongs to the same family. Change one scoring axis and the scorecard changes.

BoSS's other two tasks are prediction rather than inverse design. One is guessing where on the wall the lost ions land, the heat-load pattern; the other is pushing the particle trajectories themselves forward through time. The results table for each of those two tasks carries one row the inverse-design one does not have. The first is the wobble in the ground truth, and the second is a predictor that does nothing. This section is about the two of them.

5.1First They Put a Number on the Wobble in the Ground Truth

As section 3 noted, loss fraction is measured by launching a finite number of particles, so the same device measured twice gives slightly different values. The heat-load pattern behaves the same way. So even a perfect predictor cannot drive the metric to zero; the sampling noise remains. That remainder is the Monte Carlo floor: the value a perfect predictor is stuck above because of sampling noise. BoSS computed it first and printed it in the scorecard.

Table 2 of the paper. The test sample is 74,763 for heat flux and loss fraction, 72,512 for energy distance. The rightmost column is the IID model's value divided by the floor; it is not in the paper but computed here from the two columns next to it. The caption on Table 2 describes this performance as almost reaching the floor, while the actual gap runs from 1.5× to 1.8×.

The gap between how the caption puts it and what the numbers say is worth noting. What matters more, though, is that a reader can compute that gap at all. Without the floor column in the table there is simply no basis for deciding whether 0.3447 is good or bad. Reading model error against label noise is the first instrument BoSS stood up in its scorecard.

5.2At the Long Horizon, Two of Three Lost to the Baseline

Table 3, for the trajectory task, has a row called Stationary. It returns the particle's initial state and stops. It learns nothing and computes nothing. Standing a null baseline like that in the table is not unusual in machine learning for science. Weather forecasting benchmarks keep persistence, pushing the last observation forward unchanged, as the minimum test, and use that line to decide whether a model actually added any information about the dynamics. Table 3 of BoSS is where that practice arrives in fusion reactor design.

The normalized root mean square errors from Table 3 of the paper, joined up across horizons. Lower is better, and the heavy black line is the do-nothing baseline. The vertical scale is cut at 1.2, which bunches the lines toward the top, but all three horizon values fall inside the frame. Bootstrap standard errors are under 0.0004, smaller than the line weight. Table 3 carries two more rows besides these three, loss fraction error and confinement time distance, and because those two run the opposite way they are kept out of the figure and written out separately in the text below.

The left edge and the right edge of the figure tell different stories. At the 10 µs horizon the autoregressive MLP posts 0.261 against the baseline's 1.019, beating it comfortably, close to four times more accurate. At the 999 µs horizon the autoregressive MLP is at 1.046 and the autoregressive transformer at 1.094, both above the baseline's 1.038. Two of the three models are worse than doing nothing. The one that stays under the baseline all the way out is Direct-MLP at 0.862, which predicts the whole horizon in a single shot. The paper does not flinch here either, writing that "no model predicts any channel reliably to 1 ms".

The reason is not itself surprising. An autoregressive model takes its own prediction back as input and keeps going, so error accumulates, and past some point staying put is the better bet. The two autoregressive models were also trained on a window out to 100 µs, while only Direct-MLP saw the full 1 ms. All three were sized to about 10 million parameters. What is surprising is not the result but the fact that the result is in the scorecard. Without the baseline row, "error of 1.046 at 1 ms" would have been one more number.

Table 3 does not end with the three rows the figure draws, though. Two more sit below the horizon-by-horizon trajectory errors, and on those two all three models beat the baseline. Mean absolute error on loss fraction is 0.183 for Stationary against 0.104 for AR-MLP, 0.118 for the AR-Transformer and 0.137 for Direct-MLP. The Wasserstein distance on the confinement-time distribution is 159.8 µs for Stationary against 79.4 µs for AR-MLP, less than half. The very AR-MLP that lost to the baseline at the long horizon comes first on both of these rows.

The paper makes the mismatch its conclusion: a model with poor trajectory accuracy can still get the statistics right. Following one particle's path for 1 ms and getting the distribution of when and where particles escape are different jobs. So "two of three lost to the null baseline" is a statement about the trajectory-accuracy axis, and on the statistical axis of the same table all three clear the baseline. What the null baseline row revealed is not that the models cannot do it, but that it splits by axis, working on one and failing on another. That split is only visible if the baseline is in the table.

The paper narrowed down why they lost, too, with an experiment. The training-horizon ablation in Appendix G finds that lengthening the training window extends the range over which long-horizon predictions stay useful without sacrificing short-horizon performance. That suggests the collapse of the two autoregressive models at 1 ms is less an architectural ceiling than the consequence of having learned from a 100 µs window. Training on the full range raises the compute cost sharply, though, and doing it, the authors note, would mean rethinking the architecture and the training procedure. There was also more than one baseline. Alongside Stationary, a device mean that returns the average state of each device's particles served as a second reference predictor, and the only model to stay under that line at long horizons is again Direct-MLP.

Pebblous has counted what a validator in its place reveals on design tasks in other subjects too. In BrickBench, a LEGO design test, taking away the tool that checks whether a build is assemblable collapsed most of the designs, and grading physics engines against motion capture exposed the limits of using physics itself as the gauge. The baseline row in BoSS is an instrument of the same family. Put a baseline and a noise floor in the scoring rules and the failures become visible.

And the gap flagged at the end of section 3 returns here. The 1 ms the trajectory task sees is one hundredth of the 0.1 seconds the confinement verdict runs on. The trajectory task never reaches the timescale on which the headline metric is defined. And inside that hundredth, two of three had already lost to the null baseline on trajectory accuracy.

If the previous sections counted what BoSS put in place, this one counts what it has not put in place yet. Four seats are empty, and the four are different in kind. One is a limit in the data, one a hole in the documentation, one the distribution route, and one the part nobody knows yet.

Compiled by reading §4 through §8 of the paper against the distribution instructions on the official site. None of the four is something the paper concealed; each is either recorded or simply not recorded, and the table separates the two. Only the count of unique shapes belongs to the second kind.

6.1The Scoring Code Exists, the Place That Scores Does Not

The third row shows the current state of this field best. BoSS has metrics and it has scoring code. Install it with pip install boss-bench and you can measure your own model. What it does not have is a service that takes submissions, revalidates them with physics and ranks them. Section 7 of the paper lists such a service as future work: packaging the stellarator design pipeline and metrics "as an openly available service" would let ML researchers validate their designs without running physics codes, while ensuring that claims of physical significance are "verifiable and not overstated". Which is also a statement that it does not exist today.

On the other side there is ConStellaration. Its leaderboard runs permanently on Hugging Face and was still operating in October 2026. And that leaderboard is not a place where scores get written down on trust. Read the published code and what an entrant uploads is a JSON holding a boundary shape, which the server feeds into the benchmark package and runs through the evaluation function for each of the three tasks. All three go through an equilibrium computation, and there is even an admin path for the operator to re-score existing submissions. The version is pinned as well: the repository notes that the leaderboard evaluates every submission with exactly that version, so a score computed locally on the same version matches the leaderboard value. Participation is thin, though. Submissions across the three tasks number 69, 44 and 51, from 39, 20 and 6 users. Nearly fifteen months in, no winner has been declared.

So the state of this field does not divide into having an oracle and not having one. The division is over which oracle was picked up as the ruler, and whether it was left open for others to use. ConStellaration opened the ruler that reaches as far as the equilibrium, through a leaderboard. BoSS picked up the more expensive one, which goes on to trace alpha particles, and there is as yet no counter where anyone else can use it. Nowhere holds both the expensive ruler and the open counter. Nor is BoSS unusual in shipping without a service. Of nine public datasets of similar character, the ones with a submission-and-scoring service are the Open Catalyst family and ConStellaration, two in all; the rest publish data and nothing more. "Data only" is this field's default.

6.2The License Is Open and the Door Is Half Shut

The two routes separated in section 1 become a problem again here. The door the license opens and the door the distribution route opens are different widths. CC BY 4.0 lets anyone reuse and redistribute with nothing asked but attribution, while getting hold of the thing itself means creating an account and logging in. The three trained models, by contrast, download straight from Hugging Face. So the BoSS most people actually meet is likely to be those models rather than the 3TB.

Back to cost once more. The six-figure compute bill from section 3 is expensive if it is used once and thrown away. It is recovered only when several teams reuse the same data repeatedly. Which makes the economics of BoSS less a return-on-investment calculation than public infrastructure, built once and lent out many times. The trouble is that there is currently no way to measure the reuse. With no scoring service, nobody tallies who used it; the Hugging Face model card still shows zero likes and no download count registers. Coverage in English and in Korean is zero. In fairness, six days is short. This paragraph is a reference point, not a verdict.

6.3The Warning the Authors Wrote Down Themselves

The last seat the paper wrote out itself, in the Broader Impact section.

Models confidently filling in physically impossible values have been observed in other sciences. Materials Models Are Predicting Piezoelectric Coefficients That Must Be Zero, which Pebblous covered, is one such case: symmetry forces the value to zero and the model returned something else, and from the data alone it looked plausible. The warning in BoSS points the same way. A design that passes the scoring is not a design that understands the physics. The paper also discloses separately, in §9, that the authors used large language models for script development, running experiments and reviewing the manuscript. The impression of a document that chooses to write things down holds here as well.

Why This Matters to Pebblous

What to Build When You Cannot Compute the Answer

BoSS is a textbook case of turning the evaluation into data before building the model. The textbook cannot be copied straight across, though. Nothing in the data Pebblous handles will chase the alpha particles on our behalf. So reading this report and concluding "we should run simulations too" gets you nowhere. What transfers is not the simulator but the structure of the scorecard. The three instruments BoSS built all stand up again without a physics code.

The left column is the instrument as it appears in the paper; the right column moves that instrument into data whose labels are not computed. The right column is this report's reading and is not stated in the paper.

Where a simulator does supply the labels, the distance closes by a step. PebbloSim is one such case. There the specification BoSS published becomes a reference blueprint directly: where to cut the time limit, how many particles to launch, what to do with samples that did not converge, whether to standardize on a single reference size for re-evaluation. The six lines in section 3.2 are that list.

Three Things That Read as Data Quality

Three data quality lessons are lodged in this report. First, a row count is not a unique-row count. Those are the two numbers counted in section 3.1. Introducing a dataset calls for not one number but two, and the BoSS paper has only the first of them. Anyone buying data, or taking delivery of it, should be asking about the gap between those two before anything else, and where the seller has not written both down, diversity cannot be verified from outside.

Second, the ground truth has noise in it too. The Monte Carlo floor is label noise by another name. The order matters: measure it first, then read model error against it. With human-labeled data the order matters more. Report 95% model accuracy without having measured inter-annotator agreement and nobody can say whether that 5% is the model's fault or the ground truth wobbling.

Third, the authors knew their sample was skewed and filled the gap. QUASR and ConStellaration hold only stylized, well-optimized devices, which leaves the design space sparse. The paper says in its own words that models trained on this data "may have poor predictions in sparsely sampled regions of design space", which is why active learning was used to collect over 21,000 more devices. How they chose which ones overlaps oddly with section 2 of this report. Finding out which designs are hard means finding where loss-fraction prediction error is large, and measuring that error is precisely the expensive simulation. So they used the relative error of a cheap surrogate predicting aspect ratio instead. Aspect ratio is pure geometry and therefore easier to hit than a loss fraction that depends on particle transport, so the logic runs that wherever even aspect ratio is wrong, loss fraction will be more wrong. At thresholds of 0.5% and 1%, efficiency over random sampling rose by 20% and 40% respectively. Use a cheap proxy as the thing you grade with and you get the problem from section 2; use it to decide what to buy and it saves money. The same tool splits on where it is placed. Deliberately buying the hard regions is textbook data procurement. How differently readiness is measured from field to field was taken up separately in AI-Ready Data Means Something Different in Every Scientific Field.

Four Questions to Ask When a Model Scorecard Lands on Your Desk

Move the contents of this report into your own organization and it becomes a checklist. Anyone deciding whether to adopt a model built elsewhere, on the strength of the evaluation results handed over with it, should be asking four questions. For customers in finance, the public sector and manufacturing the same four work directly as a model review checklist.

• What is the score measured against? Are the scores of a do-nothing baseline and a random draw in the same table? Without 10.8% next to it, 43.6% is a number nobody can read.
• Who picked the conditions? Were the evaluation conditions drawn from somewhere that favors the model? One condition is the whole reason the same model moved between 43.6% and 6.7%.
• Was the wobble in the ground truth measured? Label the same input twice and how far apart do the answers land? Is the model error larger or smaller than that?
• Does it hold at the end of the horizon? Does a model that looked good over a short window stay ahead of the null baseline over a long one? The figure showing 0.261 at 10 µs becoming 1.046 at 999 µs is this question in picture form.

For Once, Counting What Works Rather Than the Flaws

When the Pebblous blog has taken on benchmarks, it has mostly been to reproduce the flaws in benchmarks other people built. Pulling the checking tool out of a LEGO design test, grading physics engines against motion capture, re-measuring drug candidates on a selectivity axis, pointing at the place where human-guessed atomic coordinates went in as labels. This one runs the other way. It opens something built well and counts why it works. The aim is not praise but a list of transplantable parts, and that list is the three rows of the table above and the four lines of the checklist.

References

Primary sources, BoSS

• 1.M. Padidar, T. Huang, M. Goldstein, M. Czekanski, J. Soules, Generative modeling for fusion: a large-scale benchmark of stellarator fast-ion simulations, arXiv:2610.37910v1, 27 October 2026. ⚠️ Preprint. The review link in the body is still the placeholder openreview.net/forum?id=XXXX. Every figure in this report is from v1. The full text was retrieved from arxiv.org/html/2610.37910.
• 2.The official BoSS site, boss.flatironinstitute.org. Licensing (data CC BY 4.0, code and models Apache 2.0), citation guidance, installation instructions and the declaration that the database will continue to grow all live here. The 3TB data itself goes through the Globus route at /download.html and requires an institutional, Google or ORCID login.
• 3.The boss_bench repository and PyPI boss-bench v1.0.0 (Python 3.9+). This is the metric code. The data generation code is not included.
• 4.The three trained models, Hugging Face mishapadidar/boss, Apache 2.0: loss-fraction prediction, heat-load sampling and inverse design, with ONNX included. Checked on 2 November 2026: zero likes, no download count.

Comparison cases and lineage

• 5.S. Cadena et al., ConStellaration: A dataset of QI-like stellarator plasma boundaries and optimization benchmarks, arXiv:2506.19583 (v1 24 June 2025, v2 26 November 2025), NeurIPS 2025 Datasets & Benchmarks Track. The "without querying expensive physics oracles" that opens section 4 is verbatim from this abstract. The size of about 158,000 follows the paper's own wording.
• 6.ConStellaration Design Leaderboard, Proxima Fusion with Hugging Face, still operating in October 2026. The 69, 44 and 51 submissions and 39, 20 and 6 users in section 6.1 come from here. The evidence that the server re-evaluates submitted designs is this Space's evaluation.py (it passes the submitted boundary JSON to the per-task evaluate() in the constellaration package), the admin path in app.py for re-scoring existing submissions, and the pinned-version note in the repository documentation.
• 7.M. Padidar, T. Huang, A. Giuliani, M. Spivak, Diffusion for Fusion: Designing Stellarators with Generative AI, arXiv:2511.20445 (v1 25 November 2025, v2 12 December 2025). The "less than 5% deviation from quasisymmetry" scoring criterion in section 2.1 is verbatim from this abstract. ⚠️ Not the same model as the diffusion model in Table 1 of BoSS.
• 8.A. Giuliani, the QUASR database (Zenodo DOI 10.5281/zenodo.10050655). ⚠️ Continuously updated, so its size is written as 200,000, 300,000 or 370,000 depending on when it is cited. That is why this report gives a range.
• 9.M. Landreman et al., Journal of Plasma Physics 91, E120 (2025), Zenodo DOI 10.5281/zenodo.14867776. Over 200,000 nonlinear gyrokinetic ITG turbulence simulations. ⚠️ The composition behind that 200,000 is written differently in different sources, so this report uses only the paper's verbatim total. R. M. Churchill et al., arXiv:2609.28730 (2026) is not an expanded edition of this work but a separate dataset run with different code.
• 10.S. Rasp et al., WeatherBench: A benchmark dataset for data-driven weather forecasting, arXiv:2002.00469. Referenced in section 5.2 as outside grounding for the null-baseline practice. The convention of keeping persistence as the minimum test was standardized here.

Background

• 11.The set of public datasets used for the cost comparison: Open Catalyst OC20 and OC22, OMat24, Materials Project, AlphaFold DB, The Well. "The ones that state cost in hours are BoSS and OMat24" in section 3 and "only two have a scoring service" in section 6.1 are the result of counting this set. ⚠️ Core-hour figures for the OC family vary widely between sources, so no number is quoted.
• 12.Fusion Industry Association, The Global Fusion Industry in 2026 (July 2026). The source for private fusion investment context. ⚠️ Funding totals for individual companies differ between sources and are not used in the body.
• 13.Korea's Ministry of Science and ICT, 2026 fusion R&D implementation plan (21 January 2026). It sets the 2026 fusion R&D budget at about $78M (KRW 112.4 billion), up from $39M (KRW 56.4 billion), with about $3.1M (KRW 4.5 billion) for new AI-integration projects, and states an expansion of support for non-tokamak approaches including spherical torus, field-reversed configuration and stellarators. ⚠️ No stellarator-specific budget is published separately.

Related reading on the Pebblous blog

Drug Discovery AI Learns From Hydrogen Atoms X-Rays Cannot See (28 September 2026), the opposite pole, where humans guess the label. · Why AI Aces the Test but Can't Do the Job (10 June 2026) · AlphaFold Folded Proteins. Now AI Designs Them From Scratch. (17 June 2026) · SELECTBENCH: AI-Designed Drug Candidates Bind the Wrong Proteins Too (21 October 2026) · BrickBench: AI Designs LEGO That Stands, Until You Take the Checking Tools Away (11 October 2026) · Graded Against Motion Capture, No Physics Engine Was Accurate Everywhere (10 August 2026) · AI-Ready Data Means Something Different in Every Scientific Field (22 July 2026) · Materials Models Are Predicting Piezoelectric Coefficients That Must Be Zero (23 August 2026).