ChatGPT will pick your winning ad. It just picked both.

The first time I noticed it, I was rushing. Two ads, thirty minutes to a review call, I pasted them into ChatGPT and asked which one was stronger. It picked Ad A. Four hundred words of reasoning. Trust cues, emotional resonance, purchase intent, the whole vocabulary.

I opened a new chat to double-check the reasoning. Same two ads, same question.

It picked Ad B. Same confidence. Different reasoning. Both answers sounded like they were written by someone who knows.

That is not a bug. It is exactly what ChatGPT is trained to do. Below is what the published research says about why, and what it means for anyone still using an LLM to evaluate finished creative.

Why this keeps happening

Four documented behaviors converge on the wrong tool doing the wrong job.

Order flips the verdict. In the canonical LLM-as-judge study, Zheng et al. (NeurIPS 2023) took a set of pairwise comparisons, showed them to top models, then swapped which candidate was labeled A versus B. Same content, opposite label. GPT-4 gave the same verdict only 65.0% of the time. GPT-3.5, 46.2%. Claude-v1, 23.8%.^[1] On a random pair of ads, a single GPT-4 verdict is close to a coin flip on whether label position moved it.

Sycophancy. Anthropic’s Sharma et al. measured what happens when a user gently pushes back on the model’s answer. On difficult misconceptions, GPT-4 changed a correct answer to a wrong one 32% of the time.^[2] Claude 2’s own preference model favored sycophantic responses over truthful ones about 95% of the time.^[2] For ad evaluation the implication is direct: whichever creative you paste first, or subtly frame as your favorite, ChatGPT will drift toward agreeing with you.

Non-determinism. OpenAI’s own developer documentation calls the Chat Completions API “non-deterministic by default.” Even with the seed parameter and a matching system fingerprint, “determinism is not guaranteed.”^[3] The verdict you got yesterday is not the verdict your teammate gets today running the same prompt. There is no reproducibility on which to make a $10,000 media decision.

RLHF made it more confident, not more correct. OpenAI’s GPT-4 Technical Report includes a calibration plot comparing the pre-trained base model against the RLHF-tuned model on MMLU. The base model’s confidence tracks its accuracy closely. The tuned model, in OpenAI’s own words: “after the post-training process, the calibration is reduced.”^[4] The version you use is more confident than it should be. Confidence is manufactured, not earned.

The study built for this exact argument

If you want one citation to hand to anyone still using ChatGPT to pick between ads, it exists. Bhat, Browne, and Bingemann released the Creativity Benchmark in September 2025: 100 brands across 12 categories, 11,012 anonymised pairwise comparisons, judged by 678 practising creative professionals.^[5] They tested three LLM-as-judge setups against those human rankings.

“Comparing three LLM-as-judge setups with human rankings reveals weak, inconsistent correlations and judge-specific biases, underscoring that automated judges cannot substitute for human evaluation.”

Bhat, Browne, Bingemann · Creativity Benchmark, arXiv preprint, Sept 2025

The paper also reports that across the 100 brands the highest-rated LLM generator beat the lowest-rated one only 61% of the time.^[5]LLMs cluster tightly on creative output, because there is no ground truth to converge on. That is the environment in which the judges then fail: no independent right answer, no calibration, weak inconsistent correlations with the human raters who actually have to answer to a P&L.

What LLMs are actually great at

The rest of this piece is not a straw man. LLMs are excellent at plenty of ad-adjacent work.

• Generating copy variations at scale.
• Brainstorming angles you had not considered.
• Rewriting long copy into short copy at multiple lengths.
• Summarising customer reviews, ad reactions, focus-group transcripts.
• Pointing out obvious mistakes: grammar errors, tone drift, inconsistent claims.

These are language production and language manipulation tasks. The training data is a good prior for what “sounds like a good ad” as an average of the internet. That is where every LLM in your creative stack earns its keep.

What a single naive prompt cannot do

The moment the task shifts from “produce this text” to “predict which of these two texts will perform better with a specific audience,” every failure mode above kicks in at once. Zhuo et al. at EMNLP 2024 measured how prompt paraphrase sensitivity moves LLM answers on classification and preference tasks, and flagged subjective tasks as the ones hit hardest.^[6] Small changes in phrasing produce different verdicts. Nothing about your ad is objective.

And then there is the deeper problem, which nobody around the office quite says out loud. An LLM has no model of your audience. It has a compressed statistical prior over all of the internet’s text. Your D2C fragrance shopper is not the average of the internet. Neither is your B2B buyer, your grocery CPG shopper, your first-time app installer. When you ask ChatGPT which of two ads will perform better, it is not modeling your audience. It is modeling the average of everyone’s audience, filtered through whatever heuristics its trainers preferred.

Which is where the practical question lands: what would a real creative test actually need to be true.

What real testing methodology needs

Any evaluation you trust with a media budget has to survive every failure mode above:

• Symmetric measurement. Both creatives scored on the same dimensions, in the same order, by the same rubric. No A/B position effects.
• Independent evaluators. Each rater does not see other raters’ judgments. No agreement cascades.
• Audience-specific priors. The evaluators represent your buyer, not a global average.
• Aggregate stability. Any single sample can be noisy. That is fine when you have a thousand of them. The aggregate verdict should be reproducible the way a well-designed poll is, even though no single respondent is.

These are structural constraints, not preferences. You can meet them with a large real-panel survey. You can meet them with moderated focus groups. What you cannot meet them with is a chat window.

How Splitroom runs a simulation

The Inventory engine reads each creative independently, then cross-verifies the two against each other, so nothing one ad has and the other lacks goes unnoticed. The Alignment engine then scores both ads on the same canonical dimensions for your product category, in the same order, by the same rubric. That is where the 35% position-flip hole closes: there is no A versus B label left to swap, because every panelist sees the same symmetric scorecard. The Audience engine generates up to a thousand synthetic consumers with genuinely different priorities. A deal-hunter and a skeptic-verifier weigh the same trust badge differently, and the variance is enforced, not accidental. Which addresses the deepest single-prompt failure: an LLM has no model of your audience, but a thousand independent panelists shaped by an audience brief is a distribution, not a personality.

The Decision engine has each panelist vote independently, with every reason forced to trace back to something actually scored on both ads. No panelist sees another panelist’s verdict, which starves the sycophancy cascade before it can start. The Insight engine aggregates the votes into the verdict, the segment splits, and the disagreements worth money. This is where non-determinism gets neutralised: any single call is noisy, but a thousand independent samples produce a stable aggregate the way a well-designed poll does, even though no individual respondent is definitive. And the Evidence engine audits the whole report before you see it, flagging a too-uniform panel or an overconfident verdict. It grades its own homework in the open, which is the calibration check RLHF-tuned models don’t get.

One articulate opinion versus a measured, adversarial, self-auditing process. That is the difference between asking a friend and running a simulation, and it is what Splitroom is built to do.

The next time you paste two ads into ChatGPT and ask which will win, notice what actually happens. It answers. Confidently. And now you have already read the papers that measured what that answer is worth.