Executive Summary

Key Numbers

Four numbers out of the paper. The first is a value carried over from prior work, and the other three are values the authors computed and printed in their own tables.

Source: arXiv:2610.38797 (2026-10-28)

The first evidence the introduction reaches for is a reanalysis experiment by Aczel and colleagues. They took 100 published social science findings, handed independent analysts the same data the original papers had used, and had each finding reanalyzed at least five times. Only 34% of those reanalyses closely matched the original result, and 26% either found no effect or reached the opposite conclusion. The data was the same and the question was the same. What differed was the judgment calls analysts made along the way.

This sensitivity is not a new observation. The authors add that it is not confined to the social sciences either, since the natural sciences face the same challenge. The trouble is that conventional statistical inference largely ignores this source of variability. Analysts have no tool for measuring and explaining it, so conclusions reach publication looking more confident than they are.

The authors expect the problem to grow. The abstract states that the rapid adoption of AI and agentic tools threatens to amplify the reproducibility crisis by reducing the transparency of the decision-making process. The introduction gets more specific. Agentic programming tools introduce methodological decisions that analysts accept without full scrutiny. When tools start making the calls a person used to make, the calls do not disappear. They move somewhere nobody is looking.

The established methods that confront this variation share one strategy. List every point where an analysis could branch, run the analysis once for each combination, and inspect how the outcomes scatter. Multiverse analysis, specification curve analysis, and vibration of effects all work that way. The Predictability-Computability-Stability (PCS) framework, which assesses how documented judgment calls across the data science life cycle affect conclusions, sits in the same place. The object these approaches produce is what the paper calls a multiverse, and each combination inside it is a universe.

The paper does not leave the multiverse as a name. It writes out a formal definition. With d decisions and a set of plausible options attached to each, a universe is the ordered list of options chosen, and the multiverse is the Cartesian product of those option sets. The authors write that this is, to the best of their knowledge, the first time such a notation has been made available. They give a reason for providing it. A common language for reasoning about the multiverse is what lets methodology for multiverse data accumulate afterwards. Every calculation DECIDE performs rests on this notation.

Knowing how far the outcomes scatter is one thing. Knowing which decision produced the scatter is a separate question and wants its own tool. The common move so far has been to train one more model that takes the decisions as inputs and the outcome as output, then read its coefficients or feature importances as each decision's influence. The paper calls that extra model a surrogate model.

The move assumes the surrogate gets reality right. The authors checked that assumption. Fitting a linear regression as the surrogate on two well-studied multiverses, the adjusted R² came to 0.816 on the multiverse about the human reproductive cycle and voting behavior, a reasonable fit, but only 0.490 on the hurricane multiverse. The residual plots also held non-random patterns, which means the assumptions behind a valid linear regression were violated. One surrogate fits well on one dataset and half as well on another, and its coefficients have been read as decision influence either way.

Real multiverses come without an answer key, so the authors built a toy multiverse where they knew the answer. Three binary decisions (c₁, c₂, c₃) make eight universes. The design works like this. c₁ and c₂ change nothing on their own and move accuracy only when the two act together, while c₃ does the opposite, acting purely through its main effect with no interaction at all. No matter the background, flipping c₁ or c₂ shifts accuracy by 0.05, and flipping c₃ shifts it by 0.04. The true ordering therefore puts c₁ and c₂ level with each other and c₃ below both.

The usual methods were run against this multiverse. A linear regression without interaction terms, two feature importances from a random forest (mean decrease in impurity, or MDI, and permutation feature importance, or PFI), and functional ANOVA (fANOVA), which partitions variance over the random forest's predictions. A regression without interaction terms counts as the fair baseline here, and the authors say why. Expecting an analyst to know the relevant interactions in advance is unrealistic for a large multiverse, and putting in every possible interaction saturates the model, because the number of features then equals the number of data points, which makes inference impossible.

Only DECIDE got the order right. Every other method named c₃ the most influential decision. The reasons differ from method to method. Linear regression and fANOVA main effects give c₁ and c₂ almost nothing, because a pure interaction yields no individual effects. MDI fails because of the greedy splits a tree makes. A variable that works only through an interaction does not reduce impurity at the root node, so the main-effect decision c₃ dominates the early splits and takes the credit with them. PFI and fANOVA total fail because of bootstrapping. Some universes are out of bag for each tree, and when the ensemble has to predict those universes it falls back to the global mean, which shrinks the importance of the interaction.

The analysis does not stop there. For a surrogate to recover the right importances in an interaction-heavy setting, the forest has to be configured so that every tree uses the full data and considers every variable at each split. Under that configuration the forest interpolates the multiverse perfectly, and PFI and fANOVA total recover the correct ranking. Under those same conditions, though, the two methods are no longer distinct from DECIDE. They collapse directly into it. Interpreting a surrogate model is therefore a lose-lose proposition, as the authors put it. Either the surrogate fails to capture the signal, or it pays for an expensive computation to reach the answer DECIDE hands over directly.

Skipping the surrogate pays off in compute time as well. The authors timed each method on five multiverses that had already been run. DECIDE produced total importance and main effects together in well under a second on every one of them. On a multiverse of 20,004 universes, fANOVA main effects took 13.9 seconds where DECIDE took 181 milliseconds, and fANOVA total, which decomposes every interaction, could not be evaluated at all on the two largest multiverses. Regression coefficients are the cheapest importances to obtain, which the paper grants. They also cannot express interaction structure without knowing in advance which interactions matter. Since no model gets fitted, the cost of DECIDE tracks the size of the multiverse rather than the complexity of a fitted proxy.

Jung and colleagues claimed that hurricanes with feminine names are deadlier than ones with masculine names, and a long debate followed over how to answer that question properly. Simonsohn and colleagues turned the analysis into a multiverse while describing specification curve analysis. There are seven decisions and 1,728 universes. The seven are which regression model to use, how to handle outliers in deaths, whether to log-transform damages, how to handle outliers in damages, which covariates to include, how to specify the predictor, and how to define the femininity of the name.

Below are the importances DECIDE assigned to the seven. Total includes interactions, main is the decision acting alone, and ratio is the main effect's share of the total.

Which regression model to use sits at the top of the table. Every estimator in the paper puts that decision first, so there is no disagreement this far. The disagreement starts immediately after. The two methods that account for interactions rank the damage transformation third, while every main-effect method drops that decision to the bottom. DECIDE main gives it 0.001, fANOVA main gives it 0.000, and the regression coefficient gives it 0.059. Whether damages get log-transformed changes nothing by itself. It splits the conclusion only when it is paired with another decision. To a method that does not count interactions, this decision is invisible.

And the variable this study set out to ask about sits at the bottom. How to define the femininity of a hurricane's name carries a total importance of 0.261, last of the seven. The authors read that as evidence that the reported effect is shaped far more by the surrounding modeling choices than by how the key predictor itself is defined.

The ratio column reads the same way on other multiverses. The fertility multiverse that Steegen and colleagues used as their example when they introduced multiverse analysis has five decisions. Relationship status ranks first, with 0.078 of its 0.089 total coming from the main effect, a ratio of 88.3%. It carries the conclusion largely under its own power. Menstrual calculation ranks third at 0.041 total with only 0.011 from the main effect, a ratio of 27.1%. On the ranking it looks like an ordinary middle-of-the-table decision, yet most of its influence arrives from where it meets other decisions. On that same multiverse the linear regression assigned the bottom three decisions 0.152, 0.152, and 0.146, pressing them into a near-tie and failing to tell them apart.

One case shows the methods diverging more sharply. Orben and Przybylski's analysis of adolescent technology use and well-being builds a separate multiverse from each of three surveys, the US YRBS and MTF and the UK MCS. On the first two the methods mostly agree. On MCS they split. DECIDE and fANOVA both rank the technology-use measure first, but the outcome variable that DECIDE places a close second gets almost nothing from fANOVA, a score of 0.003, while the regression ranks that same decision first. Importance on this multiverse comes mostly from interactions, with main-effect ratios of 21.1% and 15.5%, which leaves the main-effect methods unable to apportion the variation.

How much these rankings lean on the decision list is something the authors measured directly. The procedure fixes one decision to a single option, shrinks the multiverse, ranks the remaining decisions again, and compares that ranking with the one from before the shrink. Averaged over seven such runs on the hurricane multiverse, DECIDE total held up best at 0.86 and the linear regression held up worst at 0.69. The ranking wobbles most when model type is the decision held fixed. There the regression's ranking essentially decorrelates from the original one, averaging 0.11 and falling to −0.26 for one of the two options, and the main-effect methods drop to 0.29, while the two interaction-aware methods keep 0.66 to 0.69. The order came out the same on the fertility multiverse, 0.91 for DECIDE total and 0.65 for the regression.

DECIDE leaves the operator that summarizes outcomes and the operator that measures spread for the analyst to choose. The sections above plugged in the mean and the variance, which yields each decision's average influence. Putting in the maximum and the range turns it into a different question. How far can this decision move the outcome at its worst? That version suits the job of finding a decision that behaves most of the time and occasionally triggers a catastrophic failure, and a surrogate model, whose aggregation is fixed by construction, cannot offer the reading at all.

To show this the authors built 4,096 universes out of twelve binary decisions. Five are routine decisions that raise accuracy through their main effect alone. Six are trigger decisions, each adding 0.02 to accuracy. The last one does nothing unless all six triggers are switched on, in which case it drops accuracy by 0.30. That happens in 32 of the 4,096 universes, which is 0.78% of the multiverse.

Scored on the average, this last decision ranks lowest of the twelve. It has no effect in the overwhelming majority of cases, and the score gets diluted accordingly. Scored on the worst case, it ranks first on its own. Neither instantiation is incorrect in the authors' reading, since the two answer different questions. The trigger decisions are the part worth looking at. Taken one at a time each looks like a sensible way to add 0.02 to accuracy, and together they build the trap the last decision springs.

When we talk about data quality, what we usually count is missing values, duplicates, and label errors. That list aims at flaws in the data itself. The place this paper points to sits one step behind it. Even with perfectly clean data, preprocessing and variable definitions and model selection all remain, and as the hurricane case shows, those choices can swing a conclusion further than the study's own key variable does.

When two teams produce different reports from the same data, what we can usually say stops at the observation that the analysis methods differed. DECIDE supplies the next step. Which decision made the difference, and whether that decision acted alone or in combination with others, comes out as a number. The main-effect ratio is the single column that carries the distinction. A decision with a ratio of 0.1% yields nothing if you pull it out and examine it by itself, and has to be looked at alongside whichever decision it pairs with. A decision with a high ratio shows its influence even when you look at it alone. What a team should settle first changes with that number.

Of the four items the paper lists as future work, the third is integrating DECIDE scores into agentic data science frameworks, so that automated systems can self-audit their analytical decision-making and flag fragile choices to the user. The fourth goes one step further. Treating the reasoning choices in a language model's chain-of-thought as decisions would let DECIDE identify the concrete steps that lead to specific behaviors, which in turn informs how the model gets steered. The systems that have started making the decisions in analysis pipelines are exactly those systems. The warning in Section 1 and this piece of future work are two sides of one problem. The more judgment an agent takes on, the more it matters that each of those judgments can be traced back.

The limitations the paper records for itself carry over to practice unchanged. DECIDE's importance scores are conditional on the user-specified decision space, so omitted decisions or options directly alter the resulting rankings. The ablation experiment in Section 4 is where that alteration was actually measured, and what it showed is that DECIDE preserved the ranking better than the other methods, not that the ranking stayed put. There is also the constraint that the framework uses the complete set of universe outputs, which makes it infeasible wherever a single universe is expensive to obtain. So answering which choice changed the conclusion requires first writing down, without gaps, which choices were available. Writing that list is still a person's job, and in the field it is usually the harder half.

Thank you for reading this far. The authors and the submission date, the claims in the abstract and the introduction, the 34% and 26% from the Aczel study, the adjusted R² values of 0.816 and 0.490 for the surrogate regression, the numbers in the toy multiverse table and the hurricane table, the ratio column of the fertility multiverse, the correlations from the ablation experiment and the execution time table, the results from the three adolescent well-being surveys, and the design of the worst-case experiment along with the limitations and the future work were all checked against the original text of arXiv:2610.38797. The proposal to add analysis decisions as one entry on a data quality list, and Section 6's reading of decision documentation as quality work, belong to this article. If you have ever sat through a long meeting because two teams produced different numbers from the same data, we would be glad to hear about it.

Pebblous Data Communication Team
October 31, 2026

Primary Source

• 1.Rewolinski, Z. T., Simson, J., Kreuter, F., Kern, C., & Yu, B. (2026). "Pinpointing Pivot Points: The DECIDE Framework for Data Analysis Decision Importances." arXiv:2610.38797.

Multiverse & Reproducibility Methodology

• 2.Aczel, B., Szaszi, B., Clelland, H. T., et al. (2026). "Investigating the analytical robustness of the social and behavioural sciences." Nature, 652(8108), 135–142. DOI
• 3.Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016). "Increasing Transparency Through a Multiverse Analysis." Perspectives on Psychological Science, 11(5), 702–712. DOI
• 4.Simonsohn, U., Simmons, J. P., & Nelson, L. D. (2020). "Specification curve analysis." Nature Human Behaviour, 4, 1208–1214. DOI
• 5.Yu, B., & Kumbier, K. (2020). "Veridical data science." Proceedings of the National Academy of Sciences, 117(8), 3920–3929. DOI

Studies Re-Analyzed by DECIDE

• 6.Durante, K. M., Rae, A., & Griskevicius, V. (2013). "The Fluctuating Female Vote: Politics, Religion, and the Ovulatory Cycle." Psychological Science, 24(6), 1007–1016. DOI
• 7.Jung, K., Shavitt, S., Viswanathan, M., & Hilbe, J. M. (2014). "Female hurricanes are deadlier than male hurricanes." Proceedings of the National Academy of Sciences, 111(24), 8782–8787. DOI
• 8.Orben, A., & Przybylski, A. K. (2019). "The association between adolescent well-being and digital technology use." Nature Human Behaviour, 3, 173–182. DOI