Executive Summary

Key Figures

Source: Żatuchin, The Dice Roll Method: A Standardized Protocol for Repeated-Query Auditing of Large Language Model Brand Recommendations, arXiv:2609.04047 (2026), Tables 7, 8 and 11 and Section 6.2

How much the answer shifts each time the same question is asked again is left in raw form in the same author's earlier work. That was a study of Christmas gift recommendations, putting the same question to three models up to forty times. Revisiting the data, the methodology paper records brand-count standard deviations running from 0.49 to 3.67 across cells. In the most volatile cells, three or four brands came and went every time the question was asked again.

Change the wording of the question a little and the answer changes. Switching the prompt from husband to wife dropped Gemini's mean from 11.03 brands to 6.28 and Grok's from 10.60 to 4.13. GPT-5.2 mentioned no brand at all in 28 of 40 husband-prompt responses, a pattern the author calls brand abstinence and reads as a deliberate design choice to avoid commercial recommendations rather than as bias.

The models do not agree with each other either. On the husband prompt, the overlap between Gemini and Grok came to a Jaccard similarity of 0.68, while Gemini and GPT sat at 0.14 and Grok and GPT at 0.18. When the question dropped gender and simply asked about 2025 gifts, the only brand all three models named was LEGO. A recommendation share on a dashboard therefore shows a different world depending on which model it measures.

The methodology paper splits the source of that instability four ways. There is token-level sampling variance created by temperature and nucleus sampling, variance from prompt phrasing within a semantic equivalence class, run-to-run variance from executing the same prompt at a different time, and variance from model versions changing underneath. Repeated queries press down on the first of those only; the other three have to be fixed by design or measured separately. The paper's analysis was carried out at temperature 0.3 across three models.

So how much of the total movement is the signal we are trying to measure. The author pooled two of the studies and split the variance over 450 iteration-level observations across 75 model by prompt cells. The object of measurement is the true value for a given prompt put to a given model, and the differences between those cells accounted for 21.8% of total variance. The other 78.2% arose inside the cells, which is to say while the same question was going back to the same model.

A number obtained by asking once rides on top of that 78.2%. Iteration means averaging the value that rides on top until the 21.8% underneath comes into view, and how many times you have to average is the paper's question.

Educational measurement already has a tool for this calculation. Generalizability theory first splits variance into components, then projects how far reliability climbs as the number of iterations rises. The value it produces is the generalizability coefficient, and the convention is that anything above 0.80 is usable for group-level decisions, while judging a single object on its own calls for 0.90.

The inferential machinery is not newly invented either. Fitting a mixed model to data whose observations are not independent and obtaining power by simulation is exactly the pairing the US National Institute of Standards and Technology recommends in its AI 800-3 report on AI benchmark evaluation, and neither approach had previously been deployed in the LLM auditing literature. Existing audits that used repeated queries each chose an iteration count for pragmatic reasons, and that is the paper's starting point.

Feed the two variance components from the previous section into that framework and a value per iteration count falls out.

The author draws three tiers out of this. Five to seven iterations are exploratory, ten to twelve are confirmatory, and fifteen to twenty are rigorous. Five iterations, the count the industry uses most often, will support a description of cell-level means, but the table says it falls short of a verdict that one model recommends a given brand more than another does. Effect sizes point the same way. A large effect is first detected in a single cell at 80% power at fifteen iterations. Analyses that pool several prompts clear the bar at lower counts, and the paper is emphatic that those values are an upper bound for cell-level audits. It lists the failure of the two calculations to line up numerically as the first item of its own follow-up work.

The three tiers themselves also rest on the conditions of temperature 0.3 and three models. Raising the temperature increases token-level sampling variance and shifts the iteration requirement upward; lowering it, or auditing a single model, shifts it down at the cost of how far the result generalizes.

At twenty iterations the coefficient stops at 0.85 and never touches 0.90. Under this variance structure, reaching 0.90 would take more than forty iterations, and even then the precision gained does not keep pace. The other route is to change the design rather than the count. Pinning prompt phrasing more narrowly or lowering the decoding temperature reduces the within-cell noise itself, and the paper offers both as alternatives. So fifteen iterations are enough only for comparisons at the brand or model level. Whether the answer to this one prompt can be trusted as it stands is a question the table cannot yet answer.

2.1Cost Is Not the Reason

Cost is the usual reason given for stopping at five iterations, and the paper checked that assumption against figures. At $0.005 per API call and three models, a study of 250 queries costs $18.75 at five iterations, $37.50 at ten and $56.25 at fifteen. Going from five to ten therefore costs an extra $18.75. In exchange, reliability rises from 0.58 to 0.74 and per-cell power for a large effect rises from 0.44 to 0.74. The knee where precision per dollar starts to fall away sits around seven iterations. The real constraint, the author writes, is not money but API rate limits and the time collection takes.

2.2What Happens to the Numbers Already Published

This yardstick is not held only against audits still to come. The author turned it back on the audits he had already published. Measured on a single cell, the power of five iterations was 0.44 for large effects and 0.23 for medium ones. Resampling the actual 100 rounds of a third-party corpus produced a near-identical 0.43. So the numbers those five-iteration studies reported, the 14% cross-model agreement or the 41.6% figure, are not mis-estimations but conservative descriptions. Accurate as description, thin as evidence for a confirmatory verdict.

The ordering flipped where the preliminary version had reported the gender gap in brand counts as largest on Gemini. The new analysis found Gemini's gap indistinguishable from zero and only GPT and Grok distinguishable. Grok's effect size also came down from the large effect of the earlier version to somewhere between small and medium. The paper does not put the reversal down to the change of method alone. It gives two reasons together: the new subset is smaller and therefore lower-powered, and framing the recipient as a husband or a wife elicited model-specific response behaviour that the generic framing did not.

The lookup table in the appendix attaches one more value here. Catching a difference of this size in a single cell for confirmatory purposes takes about twenty iterations, and more than forty at the size observed on GPT.

The author's own advance hypotheses came in only partly supported as well. The prediction that five iterations would clear 0.80 power for large effects stopped at 0.44, and the prediction that ten would reach a reliability of 0.80 slipped to fifteen. The instinct for how many repetitions a measurement needs, in other words, was more optimistic than the variance structure turned out to be, and the author wrote that into his own table.

Everything to this point was computed on data the author collected himself. So the paper picked three sets of repeated-query data other teams had gathered for other purposes and released publicly, and attached a pre-registered validation. They are Motoki et al.'s political-orientation test administered over 100 rounds, Rozado's sweep of orientation tests run ten times each across 24 conversational models, and the llm-stability repository, which ran benchmark tasks ten times each at temperature 0. The analysis plan holding the hypotheses, the decision rules and the random seeds was pinned in version control before any confirmatory statistic was computed, and the failed items are reported at the same prominence as the successful ones.

What carried over and what did not divides cleanly.

The prediction machinery worked on other people's data too. On the item that estimates variance components from ten iterations and predicts reliability at higher counts, 37 of 39 cells replicated, 2 were partial and none failed. On the Motoki data, components from rounds one through ten predicted the reliability of 20-round and 50-round means, a five-fold extrapolation beyond the fitting horizon, with a maximum absolute error of 0.038. On the Rozado data the median error across 24 models was 0.008, and on llm-stability the errors were at or below 0.002. The premise that values differ from run to run even at temperature 0 replicated as well.

What failed to hold up is the numbers themselves. The prediction that five iterations fall short of 0.80 missed on all three datasets. The extreme-persona conditions in the Motoki data reached 0.83 to 0.95 at five iterations, 22 of Rozado's 24 models were already above 0.80 at five, and every llm-stability model exceeded 0.99. The objects being measured differed so widely from one another that the noise was relatively small. The opposite extreme sits inside the same data. qwen-14b-chat in the Rozado corpus had a reliability of 0.07 at five iterations. An audit that measures that model five times is measuring noise.

The convergence item missed for the same kind of reason. The prediction that 80% of asymptotic precision arrives within ten iterations held at eight on the two corpora with short grids, but slid to sixteen on the Motoki data, which observed out to 100 rounds. The 80% measured against a ceiling of ten and the 80% measured against a ceiling of fifty were never the same value. The guidance is therefore horizon-relative, and it now carries the proviso that any audit planning beyond about fifteen iterations should recompute the curve against a longer horizon.

The number of iterations needed also shifts by condition inside the paper's own data. In the study that ran the same audit across six European languages, systematic model variance dominated language variance by a factor of 7.4. That small language effect was not uniform across models, though, and Gemini was noticeably less stable on Estonian and Finnish. Reaching the same reliability on that combination takes twelve iterations rather than eight, the D-study found. Adding one language to an audit means recomputing the iteration count.

The third-party data also exposed a defect in the paper itself. The drift diagnostic that checks whether a model changes over time flagged far too many cells. On a placebo arm of 180 cells built from politically neutral questions, one regression window test flagged 142, and on the primary arm the same component accounted for 305 of 310 flags. Fitted within a single cell, the test has one cluster, so its variance estimate degenerates as within-cell dispersion approaches zero. On llm-stability the actual accuracy difference among flagged cells averaged 0.007.

Three amendments followed: replace the within-cell regression window test with a permutation test on the half-mean difference or fit it across cells with proper clustering, gate the Population Stability Index on at least twenty iterations per half, and attach a practical-significance margin to every flag. External data found a defect the author's own data could not expose.

That is where the paper ends. Seen from a team standing up an internal AI visibility metric or evaluating an outside tool, this research is not about adding one more indicator. It is about writing down the conditions attached to the indicator you already have.

Which metric you choose carries the same weight. When the paper computed five structural metrics together, the correlation between the coefficient of variation and the Gini coefficient was 0.98. Put both on one screen and it is easy to believe you are looking at two different things when the information is the same. Change the metric, on the other hand, and the ranking can invert. A Gini coefficient computed on raw brand-mention counts puts concentration at OpenAI 0.31, Perplexity 0.27 and Gemini 0.20, while a Gini computed on the Prompt-Adjusted Share of Recommendation, which gives each prompt's share distribution equal weight, produces Gemini 0.69, OpenAI 0.68 and Perplexity 0.62. Gemini's responses fragment across many brands within each prompt, which drives the raw Gini down, while it systematically favours a narrow set of top brands across prompts, which drives the adjusted one up. Two rankings out of one set of raw data.

The paper's own answer is to cut the metrics back. Keep three axes, one for structural stability, one for semantic consistency and one for fairness-adjusted visibility, and add anything else only after a pilot has shown how it correlates with them. On the semantic side the author's hypothesis was rejected. Contrary to the expectation that cosine similarity from a single embedding would serve as an adequate summary, running an ensemble of three embeddings and looking at both the minimum similarity and the disagreement between models showed that 40% of the response pairs previously classified as stable failed the criterion.

That leaves about three things to ask in front of a dashboard.

• How many times was the same question asked to produce this number, and was that count set by a pilot measurement or by habit?
• Does a confidence interval sit next to the share? Without one there is no way to tell whether the gap between this week and last week is change or wobble.
• Were the temperature and the model snapshot pinned for the whole collection period? In the paper's variance decomposition, iteration presses down on sampling variance alone, and no amount of repetition tidies up a stretch in which the model version changed.

These questions appear as requirements in the paper's protocol too. Pin and log the model identifier, the API endpoint and the snapshot date for every call. If drift flags exceed a pre-registered threshold, either add the time window to the model as a fixed effect or restrict the analysis to the pre-drift window. Register the analysis plan before the data is collected. The scale the author sketches is a study of 100 queries across three models at ten iterations, which comes to 3,000 API calls, roughly $15 and two to four hours.

The point that comparison requires a record comes back around to this paper as well. The methodology paper states in two places that all five studies were collected at temperature 0.3 and treats that value as a defining condition of the protocol. Yet open the same author's February preprint, which holds the raw data for the first of those studies, and its own first study used the API default while the other two used a temperature of 0.7, with temperature inconsistency listed among the limitations.

Since the paper states itself that the iteration thresholds were calibrated at temperature 0.3 and would need recalibrating elsewhere, the mismatch bears directly on how narrowly the recommendation should be read. Both papers recorded their collection conditions, which is the only reason a reader can make the comparison at all.

One more thing belongs in view while reading. The author who proposes this standard is the chief executive of Rankfor.AI, which builds AI brand visibility tools; all five reanalysed datasets are ones he collected earlier; and the open-source package implementing the protocol sits in the company's repository. The paper states this directly under competing interests. The raw data is available on request from the corresponding author rather than from a public repository, and the author records the remaining limitation: no other team has collected and released comparable data in the brand domain, so circularity at the level of the domain persists. The same author's February brand-bias preprint, for its part, is marked as reporting no competing interests.

The grounds for consulting the methodology anyway sit on the other side of that. The analysis plan was fixed before the confirmatory statistics were computed, the items where his own predictions missed were reported alongside the ones that held, and the conclusion that fixed tiers built on his own data collapse on other people's data is in there too. The correction that lowers Grok's effect from the large one reported in the preliminary version is in the body as well. Read knowing where the interests lie, and read what has been left open to checking at the same time.

Thank you for reading this far. The full paper is at arXiv:2609.04047. The reliability values and variance components in this article come from its Sections 5.5 and 6.2, and the brand counts and cross-model agreement figures were checked directly on 6 September 2026 against the same author's study of brand recommendation bias. If your team has already fixed an iteration count for an internal AI visibility metric, we would be glad to hear how you settled on it.

• 1.Żatuchin, D. (2026). "The Dice Roll Method: A Standardized Protocol for Repeated-Query Auditing of Large Language Model Brand Recommendations." arXiv:2609.04047.
• 2.Żatuchin, D. (2026). "Gender Bias in Large Language Model Brand Recommendations: A Three-Study Analysis of Prompt-Induced Disparities Across Seasonal and Recipient Contexts." Research Square (preprint). DOI: 10.21203/rs.3.rs-8883056/v1.

Both papers are by the same author (Dmitrij Żatuchin, chief executive of Rankfor.AI) — the competing interest is discussed in Section 4.