Why Splitroom judges the scroll-stop first.

What made us build a gate instead of a footnote

A synthetic panel duel had just returned a 93-7 landslide for one creative. The losing creative was, separately, the one independently identified as more likely to actually get looked at in a feed — its headline read faster, its focal point resolved sooner, and a repeated first-glance check on both images called it the clear winner of the race to be noticed at all. And in a live campaign running the same two creatives, the real click data was tracking the "loser."

That's not a contradiction the panel's reasoning could talk its way out of. A synthetic consumer given ten seconds to study two ads side by side will find real, defensible reasons to prefer the one with better trust badges, tighter copy, or a more premium mood. A real scroller never grants those ten seconds to the ad that didn't stop the thumb in the first place. The panel was answering "which ad wins an argument," and the question that actually determines ad performance is "which ad gets heard at all."

The order isn't arbitrary: what the research says

The foundational eye-tracking work on this is Pieters and Wedel's 2004 study in the Journal of Marketing, which tracked gaze across 1,363 print ads and more than 3,600 consumers.^[1] Their finding: the pictorial element captures attention first and largely independent of its size, text captures attention in proportion to how much space it takes up, and the brand element's job is to transfer attention it's already received to everything else on the page. Translated to a gate: something has to win the first fraction of a second before anything else — copy, trust badges, brand — gets read at all.

That finding is 20 years old and about print. The closest thing to a direct feed-scrolling replication is Mayer, Ohme, Maslowska, and Segijn's 2024 eye-tracking study in Social Media + Society, which put 201 participants through a real, scrollable Facebook newsfeed and measured gaze on desktop versus mobile, and in a quiet lab versus a busy cafeteria.^[2] It complicates the simple "pictures win" story in a useful way: on mobile, participants spent significantly less dwell time and fewer fixations on the picture than on desktop (d = .44 to .59), and significantly more on text elements. In the public, distraction-heavy condition — the one that actually resembles someone scrolling a feed on the go — attention to text-carrying elements rose again, not fell.

Kantar's own attention research draws the same two-stage line from a different angle. Their 2023 framework splits attention into passive attention (eyes on screen) and active attention (measurable emotional engagement), built on a normative database of over 50,000 ads.^[3] Their data on feed and short-form formats specifically (Facebook and Instagram feed, Stories, TikTok) shows passive attention diminishing fastest of any format in the first seconds of exposure — their stated implication is that these formats need clear communication "within the initial seconds," because that window is shorter than in formats people can't skip past. A synthetic panel scoring an ad has none of that time pressure. It reads calmly. A gate is how we put the time pressure back in.

Then creative quality and messaging, together

Once a creative clears the scroll-stop gate, the next and largest factor is creative quality and messaging — scored as one bundle, not split into separate "visuals" and "copy" scores. NCSolutions' 2023 update to their Five Keys to Advertising Effectiveness study, based on nearly 450 CPG campaigns, found creative responsible for 49% of incremental sales — the single largest of the five factors they measure, unchanged from their original 2017 study.^[4] Restricted to social media specifically, creative's share was 46%.

The reason we don't split this into a separate "visuals score" and "copy score" is that NCSolutions' own methodology doesn't split it — their creative-quality figure is a single bundled measure of "creative message," and no published study we found decomposes it cleanly into a visual-only and copy-only share without conflating the two (a striking headline is often the thing that makes a viewer look back at the image, and vice versa). Pretending to score them independently would manufacture a precision the underlying research doesn't have.

Then persona-specific weighting

The last tier is the one Splitroom already had built before this change: each synthetic panelist carries its own priority weights across trust cues, price sensitivity, emotional tone, and so on, generated from an archetype brief with enforced variance so the panel doesn't collapse into one homogenized voter. This tier answers "which specific thing about this creative moved this specific kind of buyer," and it's the one most responsible for segment-level dissent in a report — the finding that a headline wins the aggregate vote but loses inside your highest-value segment.

The honest caveat on this tier: persona conditioning is real but bounded. Hu and Collier's 2024 ACL paper found that persona conditioning explains less than 10% of variance in subjective LLM annotations even with explicit prompting.^[5] That doesn't mean persona-level dissent in a Splitroom report is noise — it means the confidence we put on any single persona's stated reasoning should be lower than the confidence we put on an aggregate finding across the whole panel, which is exactly why this tier sits third and why segment dissent is reported as a lean, not a separate verdict.

Making the gate statistically real, not one model's opinion

A single vision-model call judging "which creative wins the scroll-stop" has no more statistical standing than asking one person. Zheng et al.'s NeurIPS 2023 benchmark, the canonical paper on using LLMs as judges, found GPT-4 flips its verdict 35% of the time on the same pair of candidates when their A/B labels are swapped.^[6] One call is genuinely unreliable at the single-instance level.

GPT-4 flips its verdict on identical candidates 35% of the time when the A/B labels are swapped.

Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023

The same paper is also the strongest evidence that aggregating those noisy single calls works: GPT-4's judgments agreed with human raters 85% of the time on MT-Bench, against an 81% human-human baseline. That's the same logic Argyle et al. established for full synthetic panels — a well-conditioned LLM sample tracked real ANES respondents' vote choices at tetrachoric correlations of 0.90 to 0.94 across three election waves, even though no individual simulated respondent's answer was trustworthy on its own.^[7] Noisy at the single-call level, stable in aggregate — that's the same principle a polling sample runs on.

So the scroll-stop read is no longer a single call. It now runs repeated, independent times — the count scales with panel size (5% of the panel, floored at 2, capped at 15) — and the gate fires off the majority vote, with the vote split itself surfaced on the report. A near-even split (say 7 of 13) is now visible as exactly what it is: these two creatives don't have a clean scroll-stop winner, which is a real, honest finding, not a confident-sounding single opinion papering over genuine ambiguity.

What happens when the gate and the panel disagree

When the scroll-stop majority disagrees with the duel's actual winner, the report now carries a named warning: which creative won the scroll-stop race and by what margin, which creative won the head-to-head comparison, and a plain-language note that a comparison win means little if the losing side of that comparison is the one that actually gets looked at in a real feed. It's not a silent adjustment to the score — the panel's verdict stands as reported, and the mismatch sits next to it in the open, on the report page, not buried in a log file. That distinction matters: the gate is designed to make disagreement visible, not to override a verdict quietly on the customer's behalf.

What's still open

Two things we're not pretending to have solved. First, exactly how hard the gate should bite — right now a scroll-stop loss produces a warning next to the verdict, not an automatic score adjustment. Whether a weak or tied scroll-stop result should discount the verdict proportionally, or whether only a clear loss should trigger any adjustment at all, is a question we're deliberately leaving to calibrate against real outcome data over time, not to guess at from research that wasn't run on ad creative specifically. Second, none of the studies above measured actual downstream conversion — they measured attention and aggregate sales lift, not click-through on the specific ad format Splitroom tests. The gate is built on the best published evidence that exists for the underlying mechanism (attention has to happen before persuasion can), not on a study that ran this exact experiment.

The one-liner: a synthetic panel that's allowed to study an ad calmly for ten seconds will always find something to like about the ad that lost the real feed. The scroll-stop gate is the part of the pipeline that refuses to let it forget the ten seconds were never real.