How we score an A/B verdict on 1,500 headlines we had never seen

Anyone can show you a demo where a simulated audience picks the better headline. A demo proves very little, because the person running it chooses the examples. So before we let Planetarium name a winner between two versions of your message, we made it face a test where we could not choose anything: 1,500 real headline experiments it had never seen, with the real answers sitting in a drawer until the end.

real headline tests Planetarium had never seen

1,500

of them it named a winner on; the rest it called too close

44%

right when it did name one (range 84.4 to 89.5)

87.1%

What the 1,500 tests are

A publisher called Upworthy ran thousands of real experiments on its own readers, between 2013 and 2015. For each one it wrote several headlines for the same story, showed them to different readers, and recorded which headline got more clicks. A research team later released those experiments as a public archive. That archive is a rare thing: questions with answers that were measured on real people, not decided by a panel of opinions.

The archive comes in two parts. One part is for exploring, for trying ideas and getting a feel for the data. The other is set aside for confirmation. We used the confirmation part, and we used it the way it is meant to be used: choices about how Planetarium works were made on the exploring part, then Planetarium was run on these 1,500 tests. Each of them has a real winner: a headline that drew more clicks than the other.

Why held out is different from a demo

If you tune a system on the same questions you later grade it on, the grade measures how well you tuned, not how well it works. It is the same reason a teacher does not hand out the exam as homework. A held-out test means the exam questions were not in the building while the system was being built.

The second half of the discipline is writing the pass mark down first. Before we score a held-out set, the rule for what counts as a pass is recorded, so we cannot move the goalposts after seeing the result. We follow the same habit on every test we publish: the rule is written first, the data is scored once.

What “named a winner on 44%” means

Planetarium did not give a verdict on every pair of headlines. On a little under half of them, 44%, it named a winner. On the rest it said the two versions were too close to call.

That is deliberate, and we think it is the most important design choice in the whole feature. Many real headline pairs are, as far as any audience is concerned, a coin flip. A tool that always picks one is forced to guess on all of those, and the guesses dilute its real signal. A tool that is allowed to say “too close” keeps its confident calls for the cases where there is something to be confident about.

When you compare two versions of a text in Planetarium, this is what you see. Either a winner, where that kind of comparison passed a test like this one, or a plain statement that the difference is within noise. We would rather hand you “too close, test something bolder” than a confident number built on nothing.

What “right 87.1%” means, and the range around it

On the tests where Planetarium named a winner, it picked the version that really did get more clicks 87.1% of the time. Put another way, it was wrong 12.9% of the time, a little over one in eight.

We do not show that 87.1% alone, because a single percentage hides how sure we are of it. The range we publish is 84.4 to 89.5. It says: given how many tests this is based on, the true rate for this kind of question plausibly sits anywhere in that band. The range comes from the 660 tests where Planetarium named a winner, not from all 1,500, so it is wider than a headline number on 1,500 would suggest.

What the misses look like

The 12.9% matters as much as the 87.1%. We have not sorted the misses into tidy categories, and we will not pretend to. What we can say is that headlines are a hard test: small wording changes, curiosity and tone move readers in ways that are partly chance, so even a good judgment of a pair will sometimes lose to the clicks.

So we treat a named winner as strong evidence, not a guarantee. If a decision is expensive, treat the verdict as a reason to run the real test on the shortlist, not as a replacement for it.

What this test does not prove

We would rather be exact about the limits than let you assume more.

• It tests direction, not size. The question was “which headline gets more clicks?”, not “by how much?”. A winner tells you which way to lean. It does not promise a particular lift.
• It tests text headlines. The archive is news-style headlines shown to one publisher’s readers. It does not show how Planetarium does on a video, a logo or a product name. Images and video have not passed a check on real results, so Planetarium names no winner for them.
• It is a record, not a forecast of your result. It tells you how often the method was right on questions with known answers. Your message, your audience and your week are your own.

Why we publish it this way

There are two ways to make an audience simulator look good. One is to choose the examples. The other is to report only the number that looks best. We tried to close both: the questions were chosen by someone else, a long time ago, with real answers; and the number comes with its range, its coverage and its error rate in the same sentence.

If you are going to put money behind a decision, you should know how often the tool is right, how often it declines to answer, and what happens when it is wrong. Those three facts are 87.1%, 44% and 12.9%.

Two runs on the same tests

The first version of the verdict asked each audience which version it preferred. It was scored on these 1,500 tests on 23 September 2026, against a pass rule written down first, and passed: it named a winner on 46.4% and was right on 86.9% of those (84.2 to 89.2).

The verdict Planetarium gives today asks a different question: what each audience would do with each version, side by side. We built it on the exploring part of the archive, then scored it once on the same 1,500 tests, on 14 October 2026. That is the 44% and the 87.1% at the top of this post. It cleared the same bar the first version had to.

Both runs used a simpler setup than a full study: one plain audience per test and no study questions. And both were news headlines. Other kinds of copy have not been scored on their own yet, and images and video have not passed a check on real results, so for them the report names no winner at all.