A/B headlines, 1,500 real tests
Holds upWhich of two headlines wins?
- Named a winner on
- 660 of 1,500
- Right when it did
- 87%
- Wrong
- 13 in 100
Accuracy
scored October 2026
Right 87 times in 100 when it names a winner, on 1,500 news-headline A/B tests it never saw.
That 87 counts only the tests it named a winner on: 660 of 1,500, right on 575, wrong on 85. It called the other 840 too close rather than guess. Made to pick on 1,477, it was right 76 times in 100.
Called before it happened
Amazon, October 2026
Forecast before Amazon's October sale. 1.5 points off.
On 1 to 2 October 2026, Planetarium forecast how US shoppers would behave at Prime Big Deal Days (6 and 7 October 2026), and the forecast was locked. After the sale, Numerator published what shoppers really did.
| What shoppers were asked | Real | Forecast | Off by |
|---|---|---|---|
| Knew about the sale before buying | 91% | 91.3% | 0.3 pts |
| Placed 2 or more orders | 54% | 53.2% | 0.8 pts |
| Very or extremely satisfied with the deals | 55% | 56.0% | 1.0 pts |
| Compared prices at other stores | 55% | 53.8% | 1.2 pts |
| The sale was the main reason to shop | 35% | 32.3% | 2.7 pts |
| Bought holiday gifts | 20% | 22.8% | 2.8 pts |
| Household spend during the sale | $114.01 | $109.15 | 4.3% low |
| Typical order size | $48.74 | $46.82 | 3.9% low |
| Average miss | Percentages | Dollars |
|---|---|---|
| Planetarium | 1.5 pts | 4.1% |
| Copy last October's sale | 1.5 pts | 7.5% |
| Copy June's Prime Day | 4.8 pts | 14.0% |
| The same AI, asked directly | 16.0 pts | 42.8% |
The limit: one sale, eight figures. Copying last October's numbers did as well on the percentages; the forecast was closer on spending. It was given Numerator's results from earlier Amazon sales, all published before this one. Real figures: Numerator's Prime Big Deal Days tracker, 8 October 2026.
Against the best AI models
421 real YouTube videos
Against the best AI models, prompted directly. 15 to 27% less error than each of six.
Planetarium and six models each predicted what kind of comments 421 recent YouTube videos would get, with the real comments sealed. Then we opened the 31,400 real comments, once. Error is the share of a video's comments that would have to change type for the prediction to match what people really wrote. Lower is better.
- less error than each AI model prompted directly
- 15 to 27%
- models beaten, each by a clear margin (its 95% interval excludes zero)
- 6 of 6
- of videos where Planetarium was the single best of the 7 (next best: 18%)
- 40%
Why these six: widely used models from five AI companies, each given the same post and one identical prompt. One of them, Gemini 3.8 Flash, is the model that runs inside Planetarium, so that row shows what the audience adds to the very same model. It is also the comparison fixed before the answers were opened.
How much less error Planetarium has than each model
How much less error Planetarium has than each model prompted directly: 27% less than DeepSeek V4 Pro, 26% less than Grok 4.7, 22% less than GPT-5.6 Terra, 17% less than Claude Opus 5.5, 15% less than Gemini 3.1 Pro, 15% less than Gemini 3.8 Flash. Every gap's 95% interval excludes zero.
The full scoreboard
Error is the share of a video’s comments that would have to move to another type for the prediction to match what people really wrote; tone error does the same for positive, neutral and negative. Lower is better.
| Setup | Comment-type error | Planetarium has | Planetarium closer on | Tone error |
|---|---|---|---|---|
| Full Planetarium (Gemini 3.8 Flash inside) | 23.5% | - | - | 18.5% |
| Gemini 3.1 Pro | 27.5% | 15% less error | 64% of videos | 22.2% |
| Gemini 3.8 Flash | 27.6% | 15% less error | 65% of videos | 20.5% |
| Claude Opus 5.5 | 28.1% | 17% less error | 64% of videos | 21.2% |
| GPT-5.6 Terra | 30.2% | 22% less error | 72% of videos | 21.0% |
| Grok 4.7 | 31.8% | 26% less error | 76% of videos | 23.2% |
| DeepSeek V4 Pro | 32.3% | 27% less error | 75% of videos | 24.0% |
The average comment section, real against predicted
Plain models underestimate praise and invent questions real viewers do not ask. Planetarium knows how each channel’s own audience comments.
43%5%3%19%22%8%
42%6%2%17%28%4%
38%12%3%16%25%6%
35%9%4%15%29%8%
How the test worked
Step 1
Pick videos
421 YouTube videos published 5 Oct to 13 Oct, after Planetarium's audiences were built on 26 Sep.
Step 2
Seal the answers
Their 31,400 real comments sorted into 6 types, then sealed, with a fingerprint recorded first.
Step 3
Predict
Planetarium and 6 models each predict every video's comment mix, with the answers unseen.
Step 4
Lock
Every prediction locked and timestamped before anything is opened.
Step 5
Open once, score
The answers opened a single time, and every setup scored the same way.
The value is in Planetarium’s audience data and method, not the model: the premium models, prompted directly, did no better than the low-cost model Planetarium runs on.
How the test was run
The plain models were given each video’s title and description, a description of its thumbnail, and the same question: what mix of comments will this get? One identical prompt for all 6 models. Planetarium was given exactly the same post, plus the channel’s real audience as a client would pick it in the app, and ran exactly as it runs for customers.
500 videos were sealed and 421 were scored. 79 were left out of every setup, not only Planetarium’s: 77 because their channel’s audience is not offered in the app (too few commenters, or a label that failed our checks), and 2 because the model provider declined to answer about the video’s content.
The comparison with Gemini 3.8 Flash was fixed before the answers were opened. The comparisons with the other models were computed afterwards, from the same locked predictions. Comparisons are made per channel, so one big channel cannot swing the result.
This test measures comment reactions on YouTube. Earlier sealed tests of the same kind found the same direction against plain Gemini: YouTube 27%, TikTok 12%, Reddit 11% less error. An earlier test on X showed no clear difference.
Predicted, then checked
5 checks
What Planetarium said. What happened.
Survey question, 4,464 real answers
Close callDo the words “socially awkward” describe most research scientists well?
- Planetarium: describes them well
- 43.5%
- Real answers
- 44.3%
- Gap
- 0.8 points
Survey question, 6,127 real answers
TypicalHow much of a challenge are political divides in addressing made-up news and information?
- Planetarium: a very big challenge
- 56.5%
- Real answers
- 64.9%
- Gap
- 8.4 points
Survey question, 3,442 real answers
A missHow much do real estate values in your area affect your household’s finances?
- Planetarium: a great deal
- 14%
- Real answers
- 28.7%
- Gap
- 14.7 points
Known channels, 1,780 fresh cases
Holds upWill this audience engage with that channel?
- Forecasts checked
- 1,780
- Right
- 86.5%
- Wrong
- 13.5%
The scorecard
3 results
Three results, each scored once on data it never saw.
Read the big number, then its limit at the foot of the card.
87%
When it says one version will win, it is right about 87 times in 100.
How it was measured
1,500 held-out news-headline A/B tests from the Upworthy Research Archive (2013 to 2015), each with a real winner. It named a winner on 44% of them. Scored in October 2026 on a simpler setup than a full study.
What to make of it
It calls the rest too close instead of guessing, and it picks the direction, not the size of the lift. Other copy is not scored on its own yet, and no winner is called for images or video.
94%
How real US adults split on a survey question, it lands close.
How it was measured
772 questions in 7 survey waves never used to build or tune it. Average distance from the real split: 0.060.
What to make of it
It is checked on how a population splits, not on what any one person answers.
87%
It knows whether an audience engages with a channel it already follows.
How it was measured
1,780 fresh cases, forecast from recorded co-engagement: 86.5% accurate.
What to make of it
It covers channels an audience already follows, not brand-new content.
Benchmark
the same 1,500 A/B tests
The same tests, called other ways.
| Called by | Right when it names a winner | Names a winner on |
|---|---|---|
| Planetarium | 87% | 44% |
| Coin flip | 50% | 100% |
On your own data.
Bring a campaign you already ran. Planetarium predicts it blind; we show you the gap.
The A/B verdict, in full
1,500 tests
Right 87 times in 100 when it picks. And it does not always pick.
Each square is one real headline test it had never seen. Step through it, or change how sure it must be.
575 right, 85 wrong, 840 held back as too close.
- Named a winner
- 660
- Right
- 575
- Wrong
- 85
- Too close, held back
- 840
names morenames fewer, surer
Right when it names one87.1%
50% coin flip100%
Survey splits
772 questions, 7 waves
94% similar to how real Americans split. Wave by wave, question by question.
Pick a question and lay Planetarium's split over the real one.
- W4293.2%
- W4395.2%
- W4594.1%
- W4993.9%
- W5093.7%
- W5493.3%
- W8295.1%
88%100% the same split
All 772 questions, closest on the left
same splitfurther off
How much of a challenge are political divides in addressing made-up news and information?
Pew asked 6,127 Americans. Planetarium's split landed within 9 points on every answer.
A very big challengePlanetarium 8.4 pts lower
Real people64.9%Planetarium56.5%Moderately bigPlanetarium 4.9 pts higher
Real people26.1%Planetarium31.0%A small challengePlanetarium 3 pts higher
Real people6.5%Planetarium9.5%Not a challengePlanetarium 0.6 pts higher
Real people2.4%Planetarium3.0%
Overall the two splits overlap 92%. By the distance measure behind the 94% above, this question scores 95.8%.
How an audience answers
at least 100 real accounts each
Each audience answers from its own members. The engine stays the same.
Every audience is answered by the same engine, reading a brief built from what that audience's members posted and engage with. Pick a label to follow one through.
Pick an audience label (examples)
The audience
“Women”
At least 100 real accounts, grouped by what they engage with.
One of 6,005 audiences, 62M people
Built from its members
- What they posted: their own public comments, names, handles and links removed.
- What they engage with: the topics and accounts that grouped them.
- Who they are: gender, as a mix, from what members say about themselves.
735M recorded comments across the catalog
The same engine reads it
Every audience in a study is answered by the same engine. Only the brief changes, so two audiences answer differently because their members differ.
It answers for the group, never as one invented person.
It states the split
How the whole audience would divide across your options, each share with its range.
Checked against real people: 94% similar on 772 survey questions, closer than the same model asked plainly in 7 of 7 waves
Label check97 in 100 members who state their gender are women, against 74 in 100 across all audiences.
How we test
3 rules
Three rules, kept on every result above.
Tested on data it never saw
A test only counts if Planetarium had no look at the answers first.
Choices are made on development data. The held-out set is scored once, against a rule written before any call was made on it.
The pass mark is set first
We decide what counts as passing before we see the result.
Pass rules are written down before the test is scored, so a result cannot be argued into a win afterwards.
Every number has a range
A share on its own can look surer than it is.
Every share comes with its range. Where two options' ranges overlap, the report calls them too close rather than ranking noise.
Every study is pinned to a sealed data release, so a result can be re-read later against exactly the evidence it ran on.
The catalog today
- Audiences
- 6,005
- you can test today
- People
- 62M
- observed, not invented
- Per audience
- 100+
- real people, by construction
- Median run
- 34s
- most finish within a minute
Your question
median run 34s