Synthetic panels agree with humans 85% of the time. They predict behavior 12% of the time.

Someone asked me the honest question on a demo call last week. She had read the sample report end to end and paused.

“These aren’t real people, right?”

No.

“So how do I know it’s not just AI making things up?”

That is the right question. The honest answer is that peer-reviewed research shows synthetic panels replicate real human preferences remarkably well on some questions, and fail badly on others. Below is the sourced evidence on which is which, and the practical rule for telling them apart.

What synthetic panels are validated to do

The foundational paper is Argyle, Busby, Fulda, Gubler, Rytting, and Wingate at BYU. “Out of One, Many: Using Language Models to Simulate Human Samples” was published in Political Analysis in 2023, peer-reviewed by Cambridge.^[1] They conditioned GPT-3 on real ANES respondent backstories and asked it to answer the same survey questions those humans answered. Across the 2012, 2016, and 2020 ANES waves, GPT-3’s vote choices matched real respondents at tetrachoric correlations of 0.90, 0.92, and 0.94. Proportion agreement was 0.85, 0.87, and 0.89. Mean difference in Cramer’s V between GPT-3 and human inter-item correlations was −0.026 across closed-ended attitude items. This is the paper that named “algorithmic fidelity”: the finding that a well-conditioned LLM sample can match aggregate human survey distributions.

Aher, Arriaga, and Kalai (ICML 2023) went further and replicated classic psychology experiments.^[2] In the Ultimatum Game, GPT reproduced the human pattern of rejecting unfair offers with validity rates 88 to 99.5% across model sizes. In garden path sentence experiments, models rated control sentences as more grammatical than garden-path versions, matching the human effect. In a Milgram-style obedience task, 75 out of 100 simulated participants obeyed to the end, compared to 26 out of 40 in the original Milgram study. The qualitative direction of every classic finding survived.

Zheng et al. at UC Berkeley (NeurIPS 2023) measured how well LLM judges agreed with humans on preference tasks.^[3] On MT-Bench with expert human raters, GPT-4 agreed with humans 85% of the time, compared to a human-human agreement baseline of 81%. On Chatbot Arena, GPT-4 hit 87%. In aggregate, an LLM judge agrees with two humans about as often as two humans agree with each other. (Note: this is a proportion agreement rate. Argyle’s 0.90 to 0.94 above are tetrachoric correlations. Both are strong signals, and they measure different things.)

The closest thing to a real-world market-research validation is a PyMC Labs preprint from October 2025 with a major CPG partner.^[4] Their Semantic Similarity Rating method achieved 90% of human test-retest reliability across 57 consumer surveys and 9,300 human responses, with Kolmogorov-Smirnov similarity to human distributions exceeding 0.85. This is a preprint, not yet peer-reviewed, and the CPG partner is anonymised in the paper. But it is the strongest current published evidence that synthetic panels track real consumer survey outputs at a level a market research team would care about.

So: aggregate preferences, classic effects, and survey-distribution matching are on the “yes, this works” side. The published record is real.

What they can’t do

Now the honest half.

The strongest single citation on this side is Lu et al., “Can LLM Agents Simulate Multi-Turn Human Behavior?”, ACL 2026.^[5] On 31,865 real online shopping sessions with 230,965 user actions, prompt-based LLMs (DeepSeek-R1, Llama, Claude) achieved only 11.86% accuracy predicting the next action a real customer took. Fine-tuned Qwen2.5-7B reached 17.26% action accuracy and 33.86% F1 on purchase prediction. LLMs prompted with persona and context cannot yet predict actual behavior, clicks, purchases, session paths, at anything like the fidelity they show on aggregate preference questions.

“Prompt-based LLMs achieved 11.86% next-action accuracy predicting real online shopping behavior.”

Lu et al., Can LLM Agents Simulate Multi-Turn Human Behavior?, ACL 2026 · 31,865 real shopping sessions

Santurkar et al. at Stanford (ICML 2023) built OpinionQA, a test set of 1,498 Pew Research questions across 60 US demographic groups.^[6] They found “substantial misalignment between the views reflected by current LMs and those of US demographic groups: on par with the Democrat-Republican divide on climate change.” The misalignment persisted even after explicit demographic steering. Groups poorly reflected included Americans over 65 and widowed individuals. Which means: synthetic panels are likely to be worse at representing minority subgroups than at representing the broad center. Anyone whose target audience is a demographic minority should read that finding twice.

Gupta et al. (ICLR 2024) found that assigning personas to LLMs, which is the mechanism every synthetic-panel platform relies on, introduces its own biases.^[7] Across 24 reasoning datasets, 4 LLMs, and 19 personas, persona conditioning produced pervasive stereotype-driven performance drops, with some tasks showing 70%+ drops. ChatGPT-3.5 was affected on roughly 80% of tested personas. The paper documents an illustrative refusal, in which a model prompted with a “Black person” persona declined a math question by writing “As a Black person, I can’t answer this question as it requires math knowledge.” The paper cites this as evidence of stereotype-driven degradation in the model, not as any description of real people. This is not a subtle finding. Persona conditioning, the mechanism every synthetic-panel platform relies on, can silently distort responses in ways that look like real audience signal.

Two nuances of Gupta’s result matter for anyone deciding whether it invalidates synthetic panels wholesale. First, the tests were reasoning datasets: math, world history, common-sense inference. The measured collapse is in knowledge-task accuracy, not in aggregate preference direction. Argyle’s 0.94 replication of real vote choices happens with persona conditioning too, and survives. The two findings are compatible because they measure different things. Second, the caveat still binds: any responsible synthetic-panel platform needs to detect when persona conditioning has silently collapsed the panel’s diversity, which is exactly what a homogenised-panel audit is for.

And there is a stranger failure mode Aher et al. also documented.^[2] On a Wisdom-of-Crowds task, humans gave the aluminum melting point as roughly 240°C, wide variance around a wrong answer. GPT-4 answered exactly 660°C, the correct value, with an interquartile range that collapsed to zero. This is “hyper-accuracy distortion.” Real humans are wrong in interesting, patterned ways. RLHF-tuned LLMs are correct in ways real humans aren’t. For any task where you actually want the messiness of real audience misunderstanding (misread benefit claims, misinterpreted images, half-remembered brand associations), this is a real limit.

How to tell which insights survive

The published evidence supports a straightforward decision rule.

• Directional questions survive. “Which of these two creatives lands better for this audience?” is exactly the aggregate preference question synthetic panels are validated to replicate well.
• Segment-level differences survive at coarse resolution. “Do value-hunters weigh trust badges differently than skeptics-verifiers?” lands. Fine-grained subgroup resolution (65+ widowed women in rural Iowa) does not.
• Dimension attribution survives. “Which specific element of Ad A is driving preference?” is directional attribution and works.
• Objection surfacing survives. “What might people not like?” is exactly what a well-prompted LLM population is trained to produce.
• Absolute performance prediction does not. “Will this creative get a 2.1% CTR?” is behavior prediction. Lu et al.’s 11.86% accuracy answers that one.
• Novel-product signal does not. If the concept is genuinely new, the LLM has no prior for it. That is a training-data limit, not a methodology issue.
• Fine-grained cultural signal does not. Anything that requires a specific subculture, a post-training-cutoff event, or a language-family the model wasn’t trained on well.

The rough procedure that falls out of the evidence: use synthetic panels for directional preference and segment-level comparison, not for behavior prediction or novel-product validation.

How Splitroom draws the line

What Splitroom claims:

• Which of two finished creatives a defined audience prefers, at aggregate resolution.
• Which segments split from the headline verdict, and by how much.
• Which specific dimensions (trust cues, emotional tone, brand prominence, and so on) drove the preference.
• What the panel would object to.

What Splitroom does not claim:

• CTR or CPA prediction. There is no published evidence synthetic panels can do this yet, and building the product to promise it would be dishonest.
• Novel product-market fit. If your product is unlike anything else in a market, an LLM audience prior cannot help.
• Fine-grained subculture representation. If your target buyer is a demographic minority the Santurkar paper identified as poorly reflected, take the synthetic panel result as directional and layer real qualitative research on top.

The Evidence engine ships a homogenised-panel warning when the audience prior looks too uniform, and a monolithic-verdict warning when 90%+ of panelists pick the same side on close creatives. The pipeline is designed to flag exactly the failure modes the literature identifies, before you make a decision on the strength of a run.

The one-liner

Synthetic focus groups aren’t real people. That is what makes them useful for the specific set of questions the research shows they answer well, and useless for the questions they don’t. The mistake is treating them as a substitute for all research. The right use is treating them as one instrument in a stack, calibrated to what it is actually good at.