Executive Summary

Four numbers carry this report. The first two say how much of an identity the models actually use; the second two say where the shortfall lands.

The primary source here is a single paper posted on 24 August 2026, but the reading moves its results toward a practitioner's question: how do you audit synthetic data. When Pebblous covered a Korean synthetic persona dataset in April 2026, the question was why such a dataset is needed. This report asks what you would check that dataset's diversity with.

Running a survey means recruiting people. Seating a large language model in those chairs has been a shipping product for a few years now. You give the model a demographic profile, have it answer the questionnaire, and put its answers where a real respondent's would go. The academic thread starts in 2023, when Argyle and colleagues named the practice silicon sampling and proposed algorithmic fidelity as the test it should pass. The terms used throughout this report, silicon sampling and synthetic respondents, both descend from that work.

The fastest way in is to read what the products promise. The two lines below are lifted verbatim from the public pages of two different vendors. One sells how finely you can specify a profile; the other says the rare groups you specify can be analyzed reliably.

Cost is the second pillar under that promise. Synthetic Users puts an interview at $2 to $60 on its own pages, against $100 and up through a traditional fieldwork agency. Add the claim that a study lands in hours rather than weeks, and the temptation follows on its own: fill in cheaply the segments you used to cut because the sample was too thin. That is usually the first place synthetic respondents get called in.

Both vendors post their own limits. It is only fair to say so plainly. Fairgen calls its output directional, not definitive. Synthetic Users describes itself as a discovery co-pilot rather than a replacement for real research. So the target here is not either company's copy. It is that nothing tells you how to verify the caveat.

The industry's temperature needs two numbers read together. In the Greenbook GRIT report, 87% of practitioners who had used synthetic data said they were satisfied with it. In the same report, worry about data quality rose 40% year over year, with synthetic respondents named as the main cause. Adoption and suspicion are accelerating at the same time. For scale, the insights industry where this argument is happening was worth roughly $153 billion in 2024.

The professional norms are moving too. In a May 2026 task force report, the American Association for Public Opinion Research laid out the various ways AI enters survey work and assigned synthetic responses the highest risk tier of the lot. Then it named, in one sentence, the condition under which that risk grows.

What the products sell and what the guidance warns about are the same spot: filling rare combinations cheaply. Which leaves one question. How do you check whether that risk actually materialized? The paper this report is built on is close to the first attempt at that check, and the answer key it chose is the American Trends Panel.

The answer key's credentials are worth a moment. The Pew Research Center's ATP is a standing panel created in 2014 with roughly 10,000 adults, recruited since 2018 by address-based probability sampling, which draws households at random from postal delivery addresses and invites them in. Unlike an online opt-in panel, that means you can calculate who the sample represents. The 15 waves the paper used ran between 2017 and 2021, with between 2,524 and 10,221 respondents per wave. Because respondent-level microdata is public, you can compute the actual answer distribution for cells where race, religion, and income overlap. Grading synthetic respondents requires a grading sheet, and datasets that qualify are not common.

In April 2026 Pebblous covered a 7-million-strong Korean synthetic persona dataset, built by attaching seven kinds of narrative to one million records. The question then was why Korean training and evaluation data needs to exist in that form. This report looks at the same object from the other side: what would you use to confirm that the diversity of those personas is real?

The design explains itself without equations. Start with profiles carrying a single identity. Take a profile told only that the respondent is a woman, and measure how far and in which direction its answer distribution sits from the average across all respondents. Call that displacement a bias vector, and treat it as the fingerprint that identity leaves. Then take a profile carrying two identities and try to predict its actual displacement two ways: once by adding the two fingerprints, once by using only one of them. Count which prediction lands closer for every profile-by-question pairing, and you can see how many identities the model really used.

The scale runs like this. Closed-ended questions were pulled from 15 waves of the Pew American Trends Panel, and seven axes were combined into 1,791 profiles per question: age, gender, race, income, party, religion, and education. Of those profiles, 31 carry a single identity, 405 carry two, and 1,355 carry three. Eight models from five organizations were run against them, for 15.7 million simulated responses in the main pipeline and 21.1 million once the auxiliary conditions are included. The instruction to each model was to distribute 1,000 virtual respondents matching the profile across the answer options and return only the integer counts. The gap between two answer distributions is measured with total variation distance, which reads as the share of people who would have to change their answer to turn one distribution into the other.

The result points one way. In GPT-4o-mini, the reference model, 78.4% of two-identity cases were explained better by a single feature than by the sum, with a confidence interval of 77.8% to 79.0%. Across all eight models the range was 75.3% to 81.2%. The unit matters here and is easy to get wrong: this is not a share of subgroups. It is a share of cells, where each cell is one profile meeting one question.

2.178% Means Nothing Without a Ruler

Read 78% straight and you conclude the other 22% is fine. It is not. This win rate has a chance level, and the chance level is not one half. The paper built a ruler by simulating both extremes. In a world where addition holds perfectly, the single-feature prediction still wins 40.3% of the time; in a world where only one identity is ever used, it wins 84.3%. Rescaling an observed value between those two endpoints gives the collapse index.

The eight models land between 0.83 and 0.95 on that index, roughly six sevenths of the way to full collapse. That is a long way from reading the remaining 22% as healthy.

2.2The Identity Budget in One Picture

There is a better picture of the same result. Take the actual displacement of a two-identity profile and decompose it by least squares into some multiple of the first fingerprint plus some multiple of the second. Call those two multipliers alpha and beta, and every cell becomes a single point on a plane with alpha and beta as its coordinates. The upper-right corner, (1, 1), is the state where both identities were used in full.

The human median sits at (0.95, 0.95), and the two coordinates sum to 1.83. People use nearly both identities in full. The model points scatter along the diagonal ridge where alpha and beta sum to one, with a median total of 0.98, ranging from 0.91 to 1.03 across models. Given two identities, they spend the weight of one.

The sharper detail is that the one unit is not split evenly. The dominant component takes a median share of 0.88, and in 63% of cells one side takes more than 0.75. The models are not compromising between two identities. They pick one and largely drop the other.

2.3Give It Three and It Behaves Like Two at Best

With three identities the gap widens. Predictions that add all three fingerprints win in only 7.8% to 10.0% of cases across seven models. Predictions that pick two of the three and add those take 52% to 66%. The share of total displacement explained by addition also falls, from 19-25% at two identities to 7-10% at three. In other words, in more than nine cases out of ten a three-identity persona behaves like a two-identity one at best.

This depth needs its own ruler. Simulating the same two extremes at three identities, the all-three prediction wins 67.2% of the time in a world where addition holds perfectly, and it still wins 6.6% in a world where only one identity is ever used, purely by beating the three single predictions and three partial pairs by chance. The observed 7.8% to 10.0% sits almost on that lower endpoint, which pushes the depth-3 collapse index up to 0.92-0.98.

A last check confirms that none of these numbers depend on processing choices. Applying Pew's wave weights gives a collapse rate of 78.4%; leaving them off gives 78.3%. The three-identity addition win rate is 7.8% and 8.3% respectively. The distance between weighted and unweighted answer distributions is itself only 0.029 to 0.053, four to six times smaller than any model's error. Cell size behaves the same way. Raising the threshold from the default 20 valid responses to 200 moves the collapse rate from 78.4% to 77.3% and the three-identity win rate from 7.8% to 8.6%. Cleaner ground truth makes the models look marginally less collapsed, which is exactly the direction noise predicts. The number of answer options moves the raw win rate but not the calibrated index: 0.84 for questions with three or four options, 0.82 for those with six to eight.

An obvious objection arrives here. Holding two identities at once is hard. Would people manage it cleanly either? The authors close that door in advance by running the same rule on the same cells, this time against the real respondents.

Among real respondents, the summed prediction wins 44.4% of the time, measured over 236,752 cells with at least 100 valid responses. That 44.4% cannot be read straight either. The ground truth is itself estimated from a sample, so it carries noise, and the ceiling for this metric in a world where addition holds perfectly is 49.3%. People reach 90% of that ceiling. Run the same calculation on GPT-4o-mini and you get a 9.3% win rate against a ceiling of 29.7%, which is 31% of the ceiling.

Even the five-point shortfall on the human side looks like noise rather than failure. Narrowing to cells with 200 or more respondents cleans up the ground truth, and the win rate climbs to 47.8% with the ceiling rising alongside it. On the coefficient-based metrics, which noise does not touch, people are indistinguishable from perfect additivity. Whatever makes humans look slightly short of additive comes from estimating the answer key from a sample, not from anything about humans.

The real data also confirms that opinions sharpen as identities overlap. Measure how far each cell's answer distribution sits from the overall average, noise-corrected, and you get 0.0082 at one identity, 0.0147 at two, and 0.0204 at three. That is roughly 2.5 times more distinctive from one identity to three. Respondents who are Black and evangelical Protestant and low-income hold opinions that are clearly more particular than those of any group sharing only one of those axes.

This looks like it should help the models. It does the opposite. Had real people shown strong interaction effects, the models' failure could be defended as being asked for something humans cannot do either. What was asked here is the simplest form of combination there is. Add two slopes, and they slipped. When a two-identity profile answers much like a one-identity profile, the story is not that a hard interaction was missed. It is that the second identity was barely read.

This additive structure on the human side lines up with findings elsewhere. In social epidemiology, the MAIHDA line of work that decomposes health disparities across intersectional strata in multilevel models has consistently reported that main effects of each axis explain most between-group variation, with little left over that is specific to the combination. And multilevel regression with poststratification, the standard technique for estimating small-subgroup opinion from thin samples, assumes exactly this structure. It estimates each axis effect and reassembles them using population composition, which works well when combination-specific effects are small. The human control results in this paper are also evidence that the assumption holds in real opinion data. MRP returns later in this report, because the thing synthetic respondents should be benchmarked against is that technique, not the option of doing nothing.

If only one of two identities gets used, the next question is which one. The paper answers in two steps: first that the choice is close to random, then that a one-directional tilt sits on top of that randomness. Keeping the order matters, because it is what stops the result from being overstated.

The axis a model keeps matches the stronger axis among real respondents 53.3% to 57.9% of the time. Above half, which looks meaningful until you see the null baseline built by shuffling which axes are paired together: 50.4% to 52.2%. The models beat true chance by three to seven points. And most of that thin margin belongs to party alone. Where party genuinely dominates, 54% to 62% of cases preserve it; where party does not dominate, preservation drops to 42-47%. Put plainly, models pick which identity to keep more or less arbitrarily.

What the model is failing to see shows up more clearly when the axes are laid out against their opponents. Put the rate at which an axis should win among real respondents on the horizontal, and the rate at which the model actually kept it on the vertical, and faithful preservation would trace the diagonal. The real points do not follow the diagonal at all. They lie nearly flat inside a narrow band between 43% and 53%, even though the human-side relevance spreads from 22% to 78%. Across a more than threefold swing in how much an axis matters, the models' choices barely move away from a coin flip.

4.1One Direction on Top of the Randomness

Split the survival rates axis by axis and the picture changes. The benchmark is how dominant each axis is among real respondents: an axis that strong in the human data should survive that often in the model output. In practice, each axis misses in a fixed direction.

Gender survives 12.5 points more often than its weight in the human data warrants, and income 3.0 points more. On the other side religion survives 5.0 points less and race 9.5 points less. The under-preservation of race was statistically significant in every one of the six previous-generation models where race combinations appeared often enough to test, ranging from -8.3 to -10.6 points. Broken out by wave topic, the direction held in all 15 topics, and in waves that dealt with race head-on the gap was larger still at -11.4 points. Even when the subject was race, race survived less.

Splitting the axes into individual values shows where the deviation lives. The table below reproduces, as the paper lists them, how far preservation departed from actual dominance in the flagship model. This report adds no interpretation to it.

4.2This Is Not a Story About Party Being Strong

By this point an explanation suggests itself: the models grip political identity harder than anything else, and everything else gets pushed aside. The data does not support it. Pit the axes against each other inside the models' own representation space, with no ground truth involved, and party beats race 73% of the time and religion 72%, but loses to age, education, and income. Rank the axes by overall win rate and age leads at 63%, while race sits at 39% and religion at 35%, the two lowest. Race and religion lose against every axis they meet. The data fits "two specific axes are unusually suppressed" better than "party is unusually strong."

4.3It Was Already There Before Post-Training

The easiest explanation is alignment policy: models trained to tread carefully around sensitive attributes end up using race and religion less. That hypothesis is testable too. For any model whose weights you can open, you can split the state right after pretraining from the state after instruction tuning and run the same measurement on each. The authors compared Llama-3.1-8B under three conditions: pretrained weights with a plain prompt, the instruction-tuned model with the same prompt, and the instruction-tuned model with the chat template applied.

The deficit was already all but complete at pretraining, and instruction tuning and the chat template added almost nothing on top. The experiment could detect a change of 5.9 points for race and 2.7 points for religion, so anything large would have shown up. Mistral-7B reproduced the same direction, with -7.3 points on race and -3.6 on religion in its pretrained state.

This conclusion holds only for the two open models whose weights can be decomposed. Whether the same thing happens inside closed models is beyond what this experiment can say. The implication is still substantial. It adds one more case where a property of the training data persists in how the model conditions on inputs, and resists whatever is layered on afterwards. For anyone working on data quality, this is the most usable finding in the paper.

The questions a practitioner will raise at this point are predictable. Isn't this an artifact of how you measured? Isn't the model just regurgitating published tables? Wouldn't better prompting solve it? Won't the next model handle it? The paper blocks all four with experiments.

5.1Neither the Readout Nor the Prompt Is the Culprit

There is more than one way to pull an answer distribution out of a model. You can have it allocate 1,000 respondents at once, as the main experiment does. You can sample individual personas at temperature 1.0, a hundred per cell, and aggregate. Or you can read the log probabilities of the option tokens, the OpinionQA approach. Crossing all three methods with five model families, the collapse rate stayed between 75.3% and 83.4% in every combination. The readout method is not what produced the result.

Prompting behaves the same way. Rewriting the prompt in neutral language or switching to first person moved the number by no more than two points. Explicitly instructing the model to integrate both identities did not move it, and neither did chain-of-thought reasoning: 79.4%, 79.5%, and 79.0% in the flagship model, which is standing still. In one model it got worse. Claude Haiku 4.5 went from a baseline 79.7% to 83.1% with the explicit instruction and 85.5% with chain-of-thought added. Told to think more carefully, it used fewer identities.

The memorization objection is the heavier one. Pew's aggregate results and the OpinionQA dataset derived from them are public, and single-identity distributions are in there. Tables for overlapping combinations are not. So a model that simply picks one memorized single-identity distribution and repeats it would leave exactly the trace being called collapse here. The thing under test and the test itself can produce the same shape, which is not an objection to wave off.

The authors close the possibility four ways. First, even single-identity answers sit 1.8 to 4 times above the noise floor, which is hard to square with copying a published table. Second, which axis gets suppressed is a property of axes competing inside a prompt, so it cannot be generated from published marginals at all. Third, the same suppression appears at the same strength in pretrained checkpoints prompted with document completion, whose surface form has nothing in common with OpinionQA. Fourth, and most directly: if memorization were the cause, cells where the single distribution is closer to the truth should collapse more. Split roughly 330,000 combination comparisons into thirds by that accuracy and the collapse rate is flat at 78.5%, 78.0%, and 78.7%. Memorization, where present, pulled accuracy toward the truth. It did not manufacture the combination failure.

5.2Changing Generations Does Not Help

The most common hope is the next model. The counterexample the paper offers is a sharp one. GPT-5.5 cut prediction error by 34% to 38% relative to GPT-4o-mini, landing much closer to the real answer distributions. Its collapse rate is 79.1%, no different from the previous generation. Claude Sonnet 5 comes in at 77.8%. Getting more accurate and using more identities are two different abilities.

The same split shows up within a single generation. Prediction error largely tracks a model's general capability. GPT-4o runs between 0.171 and 0.188 across identity depths; Mistral-7B runs between 0.329 and 0.356, nearly a twofold difference. Yet how the two behave as identities go from one to three is effectively identical. Scaling improves calibration on single axes without bringing the ability to combine them along.

One thing did change in the newer generation. The under-preservation of race eased in GPT-5.5. Claude Sonnet 5, on the other hand, held the same strength as the older models. So no generalization about newer models improving this holds either. It varies by model, and the collapse rate itself came down in neither.

5.3Not Caricature, Hedging

One item in the existing literature needs correcting here. Wang and colleagues, writing in Nature Machine Intelligence in 2025, reported that language models replacing human participants flatten identity groups, and that framing now serves as the standard line in criticism of synthetic respondents. This paper produced what looks like the opposite. Simulated groups are internally more dispersed than real ones. The entropy gap in the answer distributions is +0.180 nats at one identity, +0.206 at two, and +0.234 at three, and it stayed positive under all five estimators that varied sample-size limits and normalization.

A trap the authors themselves fell into is recorded alongside this. They expected model error to be especially large for counter-stereotypical profiles, combinations like Republican atheists or Black Republicans that co-occur rarely in reality. On raw data the expectation held: mean error in those cells was a conspicuous 0.277. But those cells have few real respondents, so the ground truth itself is noisy. Peel off the noise share and 0.277 splits into 0.178 of real error and 0.099 of ground-truth noise, and after correction the correlation actually flips slightly the other way. Restrict to cells with 100 or more respondents and the relationship disappears entirely. The paper's own data demonstrated first that skipping the noise floor manufactures findings that are not there.

5.4Accuracy Turned Out to Be a Different Problem

Collapse confirmed, the natural next step is to conclude that synthetic answers are inaccurate. The paper cuts that link itself. Start with the distances. Split the real respondents in a cell in half and compare the two distributions and you get 0.079, 0.120, and 0.147 by identity depth. That is the floor no perfect simulator can go below. GPT-4o-mini's actual error is 0.212, 0.238, and 0.269, which is 1.8 to 2.7 times that floor, and the weakest model runs about 4 times above it. Refusal options do not close the gap either. Models put 5.1% of their mass on refusal against 0.7% among real respondents, and stripping that share out only brings GPT-4o-mini's single-identity error from 0.212 down to 0.168, still more than double the 0.079 floor. Adding identities does not improve the ratio: error rises exactly as much as the noise floor rises. The extra identity information was effectively no information at all.

Reproducibility deserves a note. Running the same wave four times, the answer distribution moved 0.119 between runs, and that figure held constant regardless of identity depth. The human split-half distances compared in the same table are 0.096, 0.134, and 0.159. Simply running the model again produces variation that matches or exceeds the error of the real sample it is meant to replace. What moves, though, is individual cells. Aggregate error across all cells did not shift by even 0.001 between runs, so the figures cited in this report are not themselves at the mercy of run luck. Read it as a warning about pulling out a single combination and using it.

Then one last experiment turns the picture over. To test whether collapse really causes the error, take the model's own predictions for single-identity profiles and do the addition for it. If collapse were the source of the inaccuracy, that intervention should reduce error. It did the opposite.

Doing the addition with the model's own single predictions makes the error slightly worse. Do the same addition with the real respondents' single-identity distributions and the error falls to 0.062. At the cell median that is a drop from 0.22 to 0.05, below even the human noise floor described above. The more striking part is how close the three models land: 0.062, 0.062, and 0.061.

So how are the ingredients wrong? The researchers measured single identities separately before looking at combinations, and the mismatch has two layers. The first is ordering. Which axis splits opinion most in real data varies by subject: religion is strongest in most waves, race leads on crime and economic questions, and party takes first place only in the political waves. The models' ordering does not track subject at all. GPT-4o-mini puts party first in 12 of 15 subjects, and its rank correlation with the human ordering is only 0.36. Other models do somewhat better at 0.46 to 0.55, and still put party first.

The second is magnitude. Across 429 points combining axis, value, and wave, the models inflate the axes that actually split opinion weakly. Gender goes from a real mean of 0.039 to 0.074, party from 0.071 to 0.130. The two axes that really do the most splitting are the ones that do not inflate. Race stays at 0.080 against a real 0.080, and religion barely moves, from 0.088 to 0.093. The models hand every axis a similar-sized difference, while people split differently on every axis. So the suppression at the combination stage in section 4 is not newly created there. It sits on top of two axes that were already flattened at the single-identity stage.

The practical output of this paper is not the collapse rate. It is the procedure built to measure the collapse rate, which the conclusion explicitly proposes as a standard for auditing silicon sampling. Seen from the side of the work that measures synthetic data quality, those three pieces are not survey-only tools.

6.1An Absolute Score Cannot Be Interpreted

Start with the null floor. The authors paired bias vectors from axes with no relationship to each other and measured their similarity. The median cosine similarity came back at 0.84. Unrelated identities already score 0.84. On the same scale, the summed prediction scores 0.94 and the single-fingerprint prediction 0.96. So 0.94 is not a good score. Without knowing that the floor sits at 0.84, 0.94 means nothing.

And it is not only direction. Magnitude too. The actual displacement of a two-identity profile is 0.59 times the size of the summed prediction. Plausibly right direction, roughly half the size. A quality report that prints a single similarity score hides both of these. It is hard to find a cleaner illustration of why exposing raw absolute scores in data quality metrics is dangerous.

6.2The Standards Stop at Two Dimensions

So where do the written standards for auditing synthetic data diversity currently stand? Three of them were checked, and all three stop short of the combination level. What matters here is not what these documents got wrong, but what they do not yet address.

The AAPOR report is a good document. It does not tell anyone to stop using synthetic responses. It permits clearly labeled pretesting, pilot work, and exploratory diagnostics, and says that validity and exposure risk grow once you step outside that. It names multilevel regression with poststratification as the baseline for small-subgroup estimation, and it is precise in distinguishing LLM-based synthetic responses from transparent statistical models, calling them a prompted generative process that can extrapolate opaquely beyond the observed range. What is missing is a way to check the very spot it flags as riskiest: sparse demographic clusters. Mandatory disclosure asks whether AI played the respondent. It does not ask how many identities that respondent actually used.

The Korean guide needs the same care in wording. It was written for personal data protection and handles two axes: a safety axis of re-identification risk and a utility axis of resemblance to the source. Diversity and representativeness do not stand as a third axis. Given the purpose, that is expected, and calling it an omission would be wrong. The accurate statement is that this layer has now become necessary. Internationally, the revision debate around the ICC/ESOMAR code sits on the disclosure side as well. Tell us you used it, not here is what to audit it with.

6.3The Same Shape Already Turned Up in Korean

The case for combination-level auditing does not have to rest on one American opinion dataset. An audit study released in May 2026 already caught the same failure shape in Korean synthetic personas. Bae Jun-hyung's paper proposes an audit method called the Independence-Assumption Footprint (IAF). You take the attribute combinations that a dataset's own data card says were treated as independent, and compare the synthetic joint distribution of those combinations against external official statistics.

The subject was NVIDIA's Nemotron-Personas-Korea, one million records, checked against KOSIS (the Korean Statistical Information Service), the Graduates Occupational Mobility Survey, Supreme Court given-name statistics, and the Military Manpower Administration yearbook. The result summarizes in one sentence: aligned with the KOSIS marginals, broken in three joint structures. The conditional distribution of major given occupation is off, the age profile around military service is institutionally impossible, and the share of women in heavily male occupations is flattened far toward parity relative to reality. Extended to six other locales, the pattern proved not to be universal but locale-specific.

The two papers audited different things. The study covered here looked at opinion distributions; the Korean audit looked at demographic joint structure. So they cannot be called the same conclusion. They are the same failure shape: correct axis by axis, broken in combination, and broken in the direction that pushes minority values toward parity. The first two of the four checks the Korean guide specifies are precisely where this failure passes through.

6.4Seven Checks for Synthetic Persona Diversity

The table below carries the paper's three methods, plus the baseline argument from its conclusion, over into checks a synthetic data auditor can run. Items 1 through 3 and items 6 and 7 are grounded in the paper. Items 4 and 5 are ones this report derived from the paper's methods and set up as checks. The paper does not propose those two as audit items, and it is more honest to say so.

Item 5 already has concrete numbers in this paper. If a two-identity profile spends the weight of one identity, that dataset's identity budget is close to 1, and placing it next to the human 1.83 shows the shortfall immediately. A quality report that only reports per-axis distribution match never surfaces that number at all.

The paper also pins down the direction of the fix. The response to race and religion surviving less is not to push those signals harder. It is to calibrate against measured group differences. Amplifying a particular axis without evidence lands you in the opposite failure mode, which is caricature: drawing the group as an exaggeration of itself.

6.5Why It Does Not Transfer Directly to Korea

It is better to be honest about what running the same audit in Korea would require. There are two constraints, and the second is the more fundamental one.

The first is sample size. Each combination cell needs 20 valid responses at minimum and 100 to be stable, while the Korean General Social Survey, the country's flagship opinion survey, runs roughly 1,000 to 1,600 respondents per wave. The 2025 round had 1,211. That is less than half the 2,524 of the smallest American Trends Panel wave. Statistics Korea's Social Survey has about 36,000 respondents, plenty of sample, but its questions ask more about conditions and behavior than about opinion. The Korea Welfare Panel Study covers about 7,500 households.

The second is the axes themselves. The most heavily suppressed axis in this paper is race, and Korean survey data has no direct equivalent for the role race plays in American data as the axis that splits opinion most. Religion, region, or generation could be candidates, but none maps one to one. So this report does not claim the same result would appear in Korea. It reverses the order instead: build the audit procedure before predicting the result. Which axis gets suppressed is something you have to measure, and the method for measuring it already exists.

The scope the authors drew for themselves belongs here too. These results are limited to the United States and English, closed-ended Trends Panel questions, the years 2017 through 2021, and group-level answer distributions. The authors write directly that portability to other countries, languages, and survey instruments has not been tested. They also state that their operationalization of intersectionality and additivity is deliberately conservative, and neither captures nor refutes the structural conception of intersectionality in the Crenshaw tradition. They acknowledge as well that additivity holding at the group level does not mean deep interactions are unimportant for individual-level prediction. Two further caveats attach to the accuracy statistics. The prompts did not pin down a survey date, so real opinion drift can enter the error; and because Trends Panel aggregates are public, any memorization would have pushed error down. That second caveat means the reported failure is a conservative estimate.

Pebblous has covered other failure modes of synthetic data before. The way capability diverges under repeated training on synthetic data is here, and the problem of measuring synthetic benchmark quality is here. How dataset representativeness enters as a regulatory requirement is set out in the report on Article 10 of the AI Act.

Here is why a company whose work is measuring data quality and putting that verdict on the record has been holding onto this paper. Four reasons.

7.1The Three Audit Standards Are What Data Validation Tools Already Do

The noise floor, null calibration, and split-half confirmation that the paper proposes as standards for auditing silicon sampling are not survey-specific techniques. They are the work of deciding which number has to be compared against what before it means anything, which is the same category of work a tool that attaches a quality verdict to a dataset performs. The finding that unrelated axes already score 0.84 is directly usable in quality report design. Expose an absolute score on its own and the reader interprets it against their own yardstick, and that yardstick is usually out of 100. A metric shown without its floor and ceiling misleads whoever reads it.

7.2Properties of Training Data Persist in How a Model Conditions

The three-way comparison in section 4.3 is the most usable passage in this report. The under-preservation of race and religion was not produced by alignment policy. It was already in place at pretraining, and the instruction tuning and chat template that followed changed almost nothing. The result covers only the two open models whose weights can be decomposed, but a case that separates stage by stage how a property of training data persists in the way a model conditions on inputs is not common. It adds one more piece of evidence to the claim that data quality carries through into model behavior.

7.3One Line for the Checklist, One for the Contract

Organizations that use synthetic data for training, evaluation, or research usually stop their checks at per-axis distribution match: does the gender split hold, does the age distribution resemble the source. Even Korea's official guide caps utility validation at one-dimensional distributions and two-dimensional pairwise correlation. What this paper shows is that every axis can match while information disappears at the combination level, which yields the recommendation directly: add combination-cell comparison and excess error over the noise floor to the checklist.

There is a clause for the contract as well. The finding that simply rerunning the model produces variation matching or exceeding the error of the real sample it replaces is grounds for the buyer of synthetic data to specify reproducibility requirements. Without receiving how many times the same condition was run and how far results moved between runs, there is no way to know how unstable the delivered numbers are.

7.4Neither Selling It nor Opposing It

In April 2026 Pebblous wrote about why a Korean synthetic persona dataset is needed. In August of the same year it is writing about what you would audit that dataset's diversity with. This is not for and against; it is two sides of one subject. The April report already listed the limits of the independence assumption among its caveats. In May a researcher actually audited that assumption. In August the same failure shape was confirmed in American opinion data. Given that sequence, the next question sets itself: who in Korea can actually perform combination-level validation on Korean synthetic personas?

This report is not meant to close on a moral. It does not say stop using synthetic respondents, and it does not blame the people who use them. That two identities went in and one came out is a fact you can measure, and the way to measure it is already in the paper. What is missing is a standard that establishes the measurement as a checklist item. Once the item stands, the rest is practitioners' work.

Academic

• 1.Rennard, V. & Xypolopoulos, C. (2026). Large language models simulate intersectional synthetic identities with a budget of one to two dimensions. arXiv:2608.23005v1 [cs.CY], 24 Aug 2026. Licensed CC BY-SA 4.0. Source of every paper-internal figure in this report. arXiv
• 2.Bae, J. (2026). Marginal Alignment Does Not Guarantee Joint-Distribution Fidelity: An Official-Reference Audit of Nemotron-Personas-Korea with Cross-Locale Replication. arXiv:2606.12433, 15 May 2026. The Independence-Assumption Footprint (IAF) and the audit of one million Korean synthetic persona records. arXiv
• 3.Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C. & Wingate, D. (2023). Out of one, many: using language models to simulate human samples. Political Analysis 31(3), 337–351. The starting point of silicon sampling.
• 4.Santurkar, S., Durmus, E., Ladhak, F., Lee, C., Liang, P. & Hashimoto, T. (2023). Whose opinions do language models reflect? ICML 40th. OpinionQA. The reference work for single-axis evaluation, reproduced here as the log-probability readout paradigm.
• 5.Wang, A., Morgenstern, J. & Dickerson, J. P. (2025). Large language models that replace human participants can harmfully misportray and flatten identity groups. Nature Machine Intelligence 7, 400–411. The canonical flattening account; the within-group over-dispersion result in section 5.3 pins that observation to a level.
• 6.Bisbee, J., Clinton, J. D., Dorff, C., Kenkel, B. & Larson, J. M. (2024). Synthetic replacements for human survey data? The perils of large language models. Political Analysis 32(4), 401–416. Variance reduction in synthetic responses.
• 7.Cheng, M., Piccardi, T. & Yang, D. (2023). CoMPosT: characterizing and evaluating caricature in LLM simulations. EMNLP. Measuring persona caricature.
• 8.Park, J. S. et al. (2026). LLM agents grounded in self-reports enable general-purpose simulation of individuals. arXiv:2411.10109 (v3, revised June 2026; the v1 title was Generative Agent Simulations of 1,000 People). The exception showing that self-report-based conditioning may behave differently. arXiv
• 9.Evans, C. R., Williams, D. R., Onnela, J.-P. & Subramanian, S. V. (2018). A multilevel approach to modeling health inequalities at the intersection of multiple social identities. Social Science & Medicine 203, 64–73. MAIHDA. / Merlo, J. (2018). Multilevel analysis of individual heterogeneity and discriminatory accuracy (MAIHDA) within an intersectional framework. Social Science & Medicine 203, 74–80.
• 10.Ghitza, Y. & Gelman, A. (2013). Deep interactions with MRP: election turnout and voting patterns among small electoral subgroups. AJPS 57(3), 762–776. The baseline for small-subgroup estimation.
• 11.Alaa, A., van Breugel, B., Saveliev, E. & van der Schaar, M. (2022). How faithful is your synthetic data? Sample-level metrics for evaluating and auditing generative models. arXiv:2102.08921. Alpha-precision and beta-recall. / Naeem, M. F. et al. (2020). Reliable fidelity and diversity metrics for generative models. Density and coverage. arXiv
• 12.Sun, S. et al. (2024). Random silicon sampling. arXiv:2402.18144. / Crenshaw, K. (1989). Demarginalizing the intersection of race and sex. University of Chicago Legal Forum 1989(1), 139–167. Cited as the origin of intersectionality.

Policy, statistics, industry

• 13.Rothschild, D., Marlar, J., Amaya, A. et al. (2026). Responsible AI Integration in Survey Research. American Association for Public Opinion Research, May 2026. Risk tiering of synthetic responses, the sparse demographic cluster warning, and the mandatory disclosure framework. Co-chaired by David Rothschild (Microsoft Research) and Jenny Marlar (Gallup), with Pew researchers participating. PDF
• 14.Personal Information Protection Commission of Korea (Dec 2024). Guide to Generating and Using Synthetic Data for Safe Data Utilization. Publication registration no. 11-1790377-100001-01. The four utility checks for structured synthetic data in Appendix 3. Korea Public Data Portal
• 15.Pew Research Center. The American Trends Panel. Panel size, founded 2014, address-based probability sampling since 2018. Pew Research Center
• 16.Greenbook GRIT. Smarter Insights, Faster Pace: AI's Breakthrough in Market Research. 87% satisfaction with synthetic data and a 40% year-over-year rise in data quality concern. Greenbook
• 17.ESOMAR / Research World. Inside the $153bn Insights Industry. Size of the insights industry as of 2024. Research World / Why the ICC/ESOMAR Code will matter more than ever in 2026 (commentary on the disclosure axis). Research World
• 18.Synthetic Users, official site / Fairgen, official site. The sales copy and pricing in section 1 are quoted directly from each company's public pages.
• 19.Korean General Social Survey (KGSS) cumulative data, Korea Social Science Data Archive, Seoul National University. Sample size per wave. KOSSDA

Related Pebblous reporting

• 20.Pebblous, a 7-million-record Korean synthetic persona dataset. report/nemotron-personas-korea-2026-04
• 21.Pebblous, repeated training on synthetic data and competence polarization. blog/synthetic-data-competence-polarization
• 22.Pebblous, the problem of measuring synthetic benchmark quality. blog/synae-synthetic-benchmark-quality
• 23.Pebblous, the dataset representativeness evidence required by Article 10 of the AI Act. report/eu-ai-act-article10-labeling-audit-evidence