Why an audience states its split instead of answering 40 times

Ask a hundred people which of three slogans they prefer and you get a split: perhaps 45, 35 and 20. The natural way to imitate that with an AI is to ask it the question again and again, pretending to be a different member of the audience each time, and count the answers. It is also the way that goes wrong most quietly. This post is about the change we made instead, what it bought us, and the experiment we ran and threw away.

The old way, and why it flattens

Sampling means asking the model for one answer, many times. A study could draw up to 40 answers per audience. The trouble is that a language model, asked the same kind of question repeatedly, tends to give its single most likely answer over and over. The honest spread of opinion collapses into “everyone picks option A”. You get the look of a result, with far less variety than a real audience has.

The new way: ask for the split directly

Instead of asking the model to act out a crowd, we ask it to describe the audience: “out of the people in this group, what percentage would pick each option?” We ask that a small number of times, with the question worded in more than one way, and turn the percentages into the same kind of tally the rest of the system already understands.

The idea comes from recent research showing that language models describe the distribution of a group better than they can reproduce it by acting out members one at a time. We tested whether that holds for our audiences, on our data. The deciding test ran on 24 September 2026, on 342 survey questions with real answers that neither method had been run on before. The rule for a pass was written down first.

closer to the real splits on 342 fresh survey questions

3x

questions where the stated split landed closer

323 of 342

points of accuracy on DeepSeek V4 Flash, our default model in August, on held-out audiences

+6.2

On those 342 questions, the stated split missed the real one by 0.070 on average, on a scale where 0 is a perfect match. Sampling missed by 0.224, about 3 times as far. That result is why every study now asks for the split, including studies that compare two or more versions of a message, and why the sampling path is gone from the product.

What calibration fixes

There is a catch, and we found it the hard way, in an earlier test on 12 August 2026. Asked for percentages, models understate. Their raw numbers rarely climb high enough to commit to anything. On its own, the stated split was slightly worse than sampling.

The fix was calibration: the stated numbers were corrected against what real audiences did, and checked on audiences the correction never saw. With it, the stated split beat sampling on most held-out audiences, by 6.2 points on average on DeepSeek V4 Flash, our default model then. The numbers also started to mean what they said: the gap between how sure a stated number sounded and how often it came true all but closed.

That correction is specific to the question it was fitted on, whether an audience engages with something, and to that model. So we do not apply it to every split. A split is adjusted only where a test on that kind of question supports it, and is left as stated otherwise.

Where it does not win

Stating a split helps most where sampling collapses the most, as the research predicts. With some larger models sampling already holds up well, and there the gain is small or gone.

The A/B verdict, the call on which of two versions wins, is its own check with its own held-out test, which our other post describes. We did not swap a new method in under a result we had already measured.

The range, and how we compute it

Every share comes with a range, because a split of 45% from a few dozen answers is not the same as one from thousands. We use a standard interval for proportions, chosen because it behaves well with small counts. There was a fair worry that it would be too confident, since research on simulated surveys warns that such intervals can shrink as the simulated sample grows.

So we checked, on 13 August 2026, on the same held-out audiences. A range that claims to catch the truth 90% of the time caught it 96.3% of the time, and one that claims 95% caught it 98.2% of the time. At the draw counts we use, the interval is cautious, not broken. We also built a tighter version that tracks the nominal promise more closely (87.8% and 92.1%) and is 12 to 14% narrower. We kept the cautious one as the headline range, because a product that makes claims should err toward wider, and we store the tighter one beside every result in case that changes.

The test we ran and rejected

Models have a known quirk: ask “would they engage?” and then “would they never engage?” and the two answers should add to 100% but often do not. A published idea says to average the two framings to cancel the bias. We tried it on held-out items on 13 August 2026.

The gap was real but small: the two answers fell short of adding up by 7.8 points on average. And averaging made things worse: accuracy fell and the probability error rose. The reversed question turned out noisier than the bias it was meant to fix. So we do not use it, and we wrote the result down so nobody retries it without knowing.

We keep a rejected idea on record for the same reason we write the pass rule down first: so a result cannot be argued into a win afterwards.