Executive Summary

Key Figures

Source: Tenachi et al., Do Tabular Foundation Models Know Physics? arXiv:2609.02766v1 (September 2, 2026), main text and Appendices A through E

Tabular foundation models are trained much as language models are. Entries in a table are masked and the model learns to recover them from the surrounding context. They are invariant to the order of rows and columns, and most of them emit a full predictive distribution per cell rather than a single number. A new task requires no retraining at all. The data goes into the context window and one forward pass returns the answer. The training material is usually a mass of synthetic structural causal models, random graphs in which every node is generated from its parents plus independent noise.

One property follows from that construction. Pretraining on samples from a prior under a proper scoring rule makes the forward pass an approximation to the posterior predictive. So the question worth asking about such a model is not its accuracy but the contents of its prior, and that is where the authors start. What does the model believe before it sees anything? The cleanest way to ask is to hand it data whose answer is already known, which means tables drawn from physical laws.

That regime was missing from the tabular literature. Tabular foundation models are already used across scientific fields including astrophysics, almost exclusively for interpolation, and recent benchmarks have pushed toward extrapolation. One of them, BeyondArena, drops deterministic-function data as artificial. Pretraining tells the same story: no current tabular foundation model has been pretrained on a physical law with units or on a continuous physical target.

The bench was built from 316 equations. There are 120 from AI Feynman, 111 from LSR-Transform, which rewrites the same equations into unusual forms, and 85 from LSR-Synth, which composes novel terms. No published table was loaded. Every table was regenerated from its equation, which left sample count, noise level and sampling domain free to control. Training points came from each equation's sampling box and test points from a shell outside it, with any point falling back inside the box rejected. Without that rejection, a study claiming to measure extrapolation would mostly be measuring interpolation again.

Noise was applied multiplicatively, in proportion to the value, matching the way real instruments carry relative rather than absolute error. Three noise levels, three sample sizes and three seeds ran across ten models, at a measured cost of roughly 1,300 GPU-hours and 4,200 CPU-hours. The definition of the out-of-domain region, the noise model and the failure rule were all fixed before any result was inspected.

The paper is a five-page preprint posted to arXiv on September 2 and submitted to the Representations for the Physical Sciences workshop at NeurIPS 2026, so it is a submission rather than an accepted or published paper. All four authors are at Université de Montréal and Mila, with no connection to the companies that built the models under test.

The comparison set is six methods that remain state of the art on tabular data: RealMLP, TabM, a plain MLP, CatBoost, a Gaussian process and ridge regression. Against them stand four tabular foundation models, TabPFN-3, TabICLv2, TabDPT and Real-TabPFN-2.5. Run at library defaults, the result reproduced the ordering reported by the earlier benchmark. The foundation models led in every stratum.

BeyondArena had reported that this ordering reverses once the trained baselines are properly tuned, and the authors set out to reproduce that reversal directly. It did not occur. Even with the trained models given 25 random-search configurations plus post-hoc greedy ensemble selection, and the foundation models given nothing to tune, TabPFN-3 led under all three pipelines in every stratum. There was one exception in one corner. At the largest context size, and only against the foundation model's cheapest setting, the Gaussian process came out ahead.

Applying both conditions that favor the trained models at once, extrapolation and the largest context size, narrowed the margin without flipping it. Over 314 tasks in that stratum the mean rank was 8.47 for TabPFN-3 against 9.85 for the tuned and ensembled Gaussian process, the best baseline there. Ranks are computed by placing all 26 model and pipeline entities in one pool, so they run from 1 to 26 and lower is better. The clearest breakdown in ordering came not from tuning but from extrapolation. Outside the sampling box the whole field compresses and the ranking reshuffles, with models converging on a common failure rather than degrading in place.

What happened to the tuned baselines is recorded in the appendix. Post-hoc ensemble selection changed almost nothing beyond RealMLP, and raising the selection budget from 25 rounds to 200 returned results identical to four decimal places. The authors attribute this to the data rather than to the protocol. Ensemble selection profits from configurations that make different errors, and on noiseless deterministic targets differently tuned fits of the same model converge on the same function and therefore err together. In the other direction, at zero noise every model except ridge regression improved with tuning, because library defaults calibrated for noisy real-world tables are, in the authors' phrase, "furthest from appropriate" once the noise is gone.

The authors also write down an imbalance in their own experiment. Under the tuned and ensembled pipelines the trained baselines carve off 20 percent of the samples for model selection and fit on the remaining 80 percent, while the foundation models, having nothing to select, condition on all of it. Rather than equalize the difference, the authors disclose it. Capping the foundation model's context would change the quantity being measured, and handing the baselines a separate validation set would change the sample budget. The gap is widest at 50 samples, where the trained models see 40 rows against the foundation model's 50, and that is also the most likely place for the benefit of tuning to look smaller than it is.

Before a win like this can be reported, one explanation has to be ruled out: the models may simply have seen the exam. The worry is not idle. Feynman-equation datasets account for about 118 of the 1,053 datasets surveyed as candidates for TabArena, which puts them close to this ecosystem. They were excluded from the final 51-dataset suite, but the proximity remains.

Only the two models that publish their corpora could be audited: 122 datasets for TabDPT and 43 for Real-TabPFN-2.5. The main text puts the first number at 123 while the list printed in the appendix holds 122, and this article follows the list. Neither corpus contains a Feynman equation, and neither contains a regression target generated by an analytic function. In the TabDPT corpus the six datasets labeled physics or astronomy are all classification tasks, such as particle-collision discrimination and satellite land cover, and none of the nine labeled deterministic and simulated has a continuous target. Physics is in these corpora, but never as law: it is there as detector event discrimination.

The Real-TabPFN-2.5 list is simpler still. Every one of its 43 entries is a classification task, so the corpus holds no real-world regression table at all, and the two Sloan Digital Sky Survey catalogues in it were used to separate stars from galaxies and quasars.

A table resampled from the same equation under a different name passes both a name filter and a content hash, so matching names is not enough. Instead of asserting that no contamination exists, the authors measured exposure. Each model was compared on datasets drawn from its own corpus against comparable datasets outside it, and that gap was read against models that had seen neither. Both candidates landed inside the unexposed band at every context size. What little movement remained ran the wrong way: prior exposure should matter most when the context is thin, yet the gap grew as the context got larger.

Models with no pretraining at all showed larger seen-versus-unseen gaps than the exposed models did, which means the statistic is dominated by which datasets fall into which pool rather than by exposure. The precision of the two audits is also uneven. TabDPT offered 18 regression datasets inside its corpus against 37 matched datasets outside it, while Real-TabPFN-2.5 had effectively one comparable table, leaving that panel far noisier.

On noiseless data drawn from a deterministic law, the correct posterior predictive is a delta function with no width at all, because the observation noise is zero and sufficient context determines the function. The models that emit a predictive distribution reported a nonzero width regardless, and that width stopped shrinking as samples were added. Their own error, measured on the same runs, kept falling. In this limit the uncertainty a model reports about itself and the amount by which it actually misses part ways.

This is not a defect in the estimator. At a noise level of 0.1 the same models sit exactly at the expected calibration width. The failure is specific to the fully noiseless limit.

In the damped harmonic oscillator experiment in the appendix, the models were given 200 noiseless samples over t between 0 and 2 and queried out to t equal to 6. Inside the sampled range the Gaussian process interval was exactly zero. The three distributional models were not: TabICLv2 held a width of 0.010 and the two TabPFN variants between 0.019 and 0.020. These are normalized values, so the magnitudes are small, and what matters is that anything remains in a place where nothing should.

Adding TabDPT to that list would make the sentence false. It emits point estimates only, so there is no width to speak of. In the authors' words, "it does not report zero uncertainty, it reports none." The noise-floor result applies to the three models that return a distribution.

Outside the sampled range all four collapsed, losing the oscillation within roughly half a period and relaxing to a constant. Not everything was lost, though. The predictions of TabPFN-3 and TabICLv2 decay toward zero as the true signal does, unlike CatBoost, which simply holds the last value it saw.

The predictive density of TabICLv2 even keeps a banding that echoes the scale of the oscillation, and the authors decline to call that a recovery of periodic structure. Sliding the true continuation across 201 lags spanning one period, the unshifted curve, which is the correct answer, ranked 98th for TabICLv2 and 157th for TabPFN-3. That is a model hedging across a range of values, not one that has locked onto the phase. It points the same way as a 2024 observation that found no evidence of TabPFN detecting periodic structure.

The cause lies in the shape of the prior. The structural causal models used in pretraining are built by putting independent noise on every node. Such a prior assigns no probability mass at all to the noiseless region, and no amount of context can move a posterior somewhere its prior never went. This is not error from insufficient data. It is the result of asking a model to represent a state it cannot represent.

The second absence is units. Physical quantities carry dimensions, and by Buckingham's theorem a law written in five variables may depend on only two dimensionless groups. The effective dimensionality of a physical problem is often far below its column count. A model that does not distinguish the order of columns and cannot see units has no entrance to that structure.

The authors built the matrix of SI dimension exponents for the variables from the Feynman units table and obtained the number of dimensionless groups by subtracting the rank of that matrix from the number of variables. Then they asked which of the two quantities the error tracks, the raw column count or the number of dimensionless groups. For every model the error tracked the raw column count first, and for most of them dimensionless dimensionality added nothing further.

The direction of the exception is the most counterintuitive result in the paper. TabPFN-3, the strongest foundation model, was significantly worse on tasks with more dimensionless groups, which are the tasks with more room for dimensional reduction. On those same tasks RealMLP and CatBoost were significantly better. Physical low-dimensional structure that a trained model converts into an advantage comes back as a penalty for the strongest foundation model.

Two qualifiers, one of scope and one of method, travel with this result. What degraded was TabPFN-3, not tabular foundation models as a class. And the authors did not feed unit information to any model to see whether performance improved; they observed how performance varies with the number of dimensionless groups. The sentence that supplying units would fix this does not appear in the paper.

The last line of the paper ties both findings together: "They are excellent amortised interpolators; a physical model would need a prior that contains physics." Two facts hold at once. A model never pretrained on a physical law fills in physical data this well, and the same model cannot hold either of two properties, a mechanism without noise and dimensional structure.

From here on this is not what the paper tested. It is this article laying that result over the data an organization actually keeps.

Open a table coming out of a laboratory or a factory and most of what is there is values. Column names exist, units sit in parentheses or nowhere, and nothing in the table marks which column was read off an instrument and which was derived by calculation. The precision of a measurement and the presence or absence of noise live in someone's memory or in a separate document. A model sees none of that. It sees the numbers written in the table.

What this paper shows is that interpolation works well even in that state, which is why early adoption tends to produce a good report card. The trouble comes afterward. The moment the work moves outside the observed range, or requires judging which value is settled, the model breaks down, and if the cause is a demand made on a prior rather than a shortage of data, collecting more data will not answer it.

A team considering a tabular foundation model can start by checking three things.

• Is the unit of each column written into the schema? It has to sit somewhere a machine can read it, not in parentheses beside a column name or in an engineer's memory, before it can be passed downstream.
• Is there a distinction between columns that are noisy measurements and columns that are noiseless calculated values? Both results in this paper trace back to that distinction being absent from the prior.
• Is the model evaluated inside the range of its training data or outside it? Measured only from the inside, the failure this paper exposes will not appear on the report card.

The business side of the same technology is covered in our report on SAP's acquisition of Prior Labs, and the question of assembling what is attached to the values continues in The Conditions for AI-Ready Data.

Thank you for reading this far. The full paper is available at arXiv:2609.02766, and every figure in this article was checked directly against the main text and Appendices A through E. If your team already records units alongside its tables, we would be glad to hear how you write them down.

Pebblous Data Communication Team
September 6, 2026

• 1.Tenachi, W., Hezaveh, Y., Perreault Levasseur, L., & Bacon, P.-L. (2026). "Do Tabular Foundation Models Know Physics? Contamination, Units, and the Deterministic Limit." arXiv:2609.02766 (submitted to the Representations for the Physical Sciences Workshop @ NeurIPS 2026).