Why Splitroom judges the scroll-stop first.

What made us build a gate

A synthetic panel duel returned a 93-7 landslide for one creative.

The losing creative was, separately, the one more likely to actually get looked at in a feed. Its headline read faster. Its focal point resolved sooner. A repeated first-glance check on both images called it the clear winner of the race to be noticed at all.

And in a live campaign running the same two creatives, the real click data was tracking the "loser."

That's not a contradiction the panel's reasoning could talk its way out of. Give a synthetic consumer ten seconds to study two ads side by side, and it will find real, defensible reasons to prefer the one with better trust badges, tighter copy, a more premium mood.

A real scroller never grants those ten seconds to the ad that didn't stop the thumb in the first place. The panel was answering "which ad wins an argument." The question that actually decides ad performance is "which ad gets heard at all."

What the research says the order should be

The foundational eye-tracking work here is Pieters and Wedel's 2004 study in the Journal of Marketing. They tracked gaze across 1,363 print ads and more than 3,600 consumers.^[1]

Their finding: the pictorial element captures attention first, largely independent of its size. Text captures attention in proportion to how much space it takes up. The brand element's job is to transfer attention it already received to everything else on the page.

Translated to a gate: something has to win the first fraction of a second before copy, trust badges, or brand get read at all.

That study is 20 years old and about print. The closest thing to a direct feed-scrolling replication is Mayer, Ohme, Maslowska, and Segijn's 2024 eye-tracking study in Social Media + Society. They put 201 participants through a real, scrollable Facebook newsfeed and measured gaze on desktop versus mobile, and in a quiet lab versus a busy cafeteria.^[2]

It complicates the simple "pictures win" story. On mobile, participants spent less dwell time and fewer fixations on the picture than on desktop (d = .44 to .59), and significantly more on text. In the public, distraction-heavy condition — the one that actually resembles scrolling a feed on the go — attention to text rose again, not fell.

Kantar's own attention research draws the same two-stage line from a different angle. Their 2023 framework splits attention into passive attention (eyes on screen) and active attention (measurable emotional engagement), built on a database of over 50,000 ads.^[3]

Their data on feed and short-form formats — Facebook and Instagram feed, Stories, TikTok — shows passive attention diminishing fastest of any format in the first seconds of exposure. Their own conclusion: these formats need clear communication within the initial seconds, because that window is shorter than in formats people can't skip past.

A synthetic panel scoring an ad has none of that time pressure. It reads calmly. A gate is how we put the time pressure back in.

Then creative quality and messaging, together

Once a creative clears the scroll-stop gate, the next and largest factor is creative quality and messaging. We score it as one bundle, not split into separate "visuals" and "copy" scores.

NCSolutions' 2023 update to their Five Keys to Advertising Effectiveness study, based on nearly 450 CPG campaigns, found creative responsible for 49% of incremental sales. The single largest of the five factors they measure, unchanged from their original 2017 study.^[4] Restricted to social media specifically, creative's share was 46%.

Why not split it into a "visuals score" and "copy score"? Because NCSolutions' own methodology doesn't split it. Their creative-quality figure is one bundled measure. No published study we found decomposes it cleanly into a visual-only and copy-only share without conflating the two. A striking headline is often what makes a viewer look back at the image, and vice versa.

Scoring them independently would manufacture a precision the research doesn't have.

Then persona-specific weighting

The last tier is one Splitroom already had built. Each synthetic panelist carries its own priority weights across trust cues, price sensitivity, emotional tone, and more, generated from an archetype brief with enforced variance so the panel doesn't collapse into one homogenized voter.

This tier answers "which specific thing about this creative moved this specific kind of buyer." It's the tier most responsible for segment-level dissent in a report — the finding that a headline wins the aggregate vote but loses inside your highest-value segment.

The honest caveat: persona conditioning is real but bounded. Hu and Collier's 2024 ACL paper found persona conditioning explains less than 10% of variance in subjective LLM annotations, even with explicit prompting.^[5]

That doesn't make persona-level dissent noise. It means a single persona's stated reasoning should carry less confidence than an aggregate finding across the whole panel. That's exactly why this tier sits third, and why segment dissent is reported as a lean, not a separate verdict.

Making the gate statistically real

A single vision-model call judging "which creative wins the scroll-stop" has no more statistical standing than asking one person.

Zheng et al.'s NeurIPS 2023 benchmark, the canonical paper on using LLMs as judges, found GPT-4 flips its verdict 35% of the time on the same pair of candidates when their A/B labels are swapped.^[6] One call is genuinely unreliable at the single-instance level.

GPT-4 flips its verdict on identical candidates 35% of the time when the A/B labels are swapped.

Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023

The same paper is also the strongest evidence that aggregating those noisy calls works. GPT-4's judgments agreed with human raters 85% of the time on MT-Bench, against an 81% human-human baseline.

Argyle et al. found the same pattern for full synthetic panels. A well-conditioned LLM sample tracked real ANES respondents' vote choices at tetrachoric correlations of 0.90 to 0.94 across three election waves, even though no single simulated respondent's answer was trustworthy on its own.^[7]

Noisy at the single-call level, stable in aggregate. Same principle a polling sample runs on.

So the scroll-stop read is no longer a single call. It runs independently multiple times — the count scales with panel size, 5% of the panel, floored at 2, capped at 15 — and the gate fires off the majority vote. The vote split itself is surfaced on the report.

A near-even split, say 7 of 13, is now visible as exactly what it is: these two creatives don't have a clean scroll-stop winner. That's a real, honest finding. Not a confident-sounding single opinion papering over genuine ambiguity.

When the gate and the panel disagree

When the scroll-stop majority disagrees with the duel's actual winner, the report carries a named warning. Which creative won the scroll-stop race, and by what margin. Which creative won the head-to-head comparison. A plain-language note that a comparison win means little if the losing side is the one that actually gets looked at in a real feed.

It's not a silent adjustment to the score. The panel's verdict stands as reported. The mismatch sits next to it, in the open, on the report page — not buried in a log file.

That distinction matters. The gate is designed to make disagreement visible, not to override a verdict quietly on the customer's behalf.

What's still open

Two things we're not pretending to have solved.

First, exactly how hard the gate should bite. Right now a scroll-stop loss produces a warning next to the verdict, not an automatic score adjustment. Whether a weak or tied result should discount the verdict proportionally, or only a clear loss should trigger any adjustment at all, is a question we're leaving to calibrate against real outcome data over time. Not to guess at from research that wasn't run on ad creative specifically.

Second, none of the studies above measured actual downstream conversion. They measured attention and aggregate sales lift, not click-through on the exact ad formats Splitroom tests. The gate is built on the best published evidence for the underlying mechanism — attention has to happen before persuasion can — not on a study that ran this exact experiment.

The one-liner: a synthetic panel that's allowed to study an ad calmly for ten seconds will always find something to like about the ad that lost the real feed. The scroll-stop gate is the part of the pipeline that refuses to let it forget the ten seconds were never real.