How The Huffington Post uses real-time testing to write better headlines
From direct mail to web design, A/B testing is considered a gold standard of user research: Show one version to half your audience and another version to the other half; compare results, and adjust accordingly. Some very cool examples include Google’s obsessive testing of subtle design tweaks and Dustin Curtis’ experiment with direct commands and clickthrough rates. (”You should follow me on Twitter” produced dramatically better results than the less moralizing, “Follow me on Twitter.”)
So here’s something devilishly brilliant: The Huffington Post applies A/B testing to some of its headlines. Readers are randomly shown one of two headlines for the same story. After five minutes, which is enough time for such a high-traffic site, the version with the most clicks becomes the wood that everyone sees.
Headlines have always played the most promotional role in news, charged with selling readers on the articles they adorn, so it only makes sense to apply the best tools of market research to their crafting. Think of it as a more rigorous version of magazines adjusting their covers based on newsstand sales.
Paul Berry, chief technology officer at The Huffington Post, spoke briefly about their real-time headline testing on a panel at the Online News Association conference in San Francisco earlier this month. When I talked to him afterwards, Berry said the system was created inhouse, but he wouldn’t disclose much else about how or how often it’s done. He did say Huffington Post editors have found that placing the author’s name above a headline almost always leads to more clicks than omitting it.
Though it’s unrelated to this A/B testing, The Huffington Post’s new social media editor, Josh Young, has also been soliciting better headlines from readers on Twitter. That’s not as awesomely scientific, but it’s a pretty good use of the crowd.
Studying and responding to users was the theme of Berry’s panel, which also included Steve Dorsey, who conducts qualitative research on reader behavior for the Detroit Free Press, and Eric Brown, homepage planning editor for Yahoo.
Brown said Yahoo closely tracks clickthroughs to measure which content does well with which audiences, as illustrated in his slide below. Yahoo may eventually use that data to serve, say, teenage girls a different version of the front page than they serve to middle-aged men. (Brown’s entire presentation from ONA is here.)
On the topic of audience segmentation, Berry told me that The Huffington Post is considering separate East Coast and West Coast editions. He used the example of morning-after Oscars coverage, when readers flock to slideshows and blog posts about the event. But the story is old news to East Coast readers by noon, when the West Coast is only first logging on, so it would make sense to serve New Yorkers fresher headlines while Californians get their Oscars fix. Identifying readers’ location from their IP address is easy, but coordinating different editions of the site would be a challenge.
I agree that this is brilliant. It’s an effective use of learning what works for readers. Ultimately, publishers want to attract readers to their content.
I listened to Ken Lerer speak on some of the SEO and viral strategies behind the HuffPo, one thing that he pointed out was that many traditional publishers are still focusing on content, which is king, but there needs to be some great effort going into what works on the Web and how content becomes viral. Right now, it seems many news orgs are ignoring it and thinking that readers will come to them.
Real-time rating will lead to “better” headlines? I can certainly see how it will lead to more popular ones.
Perhaps they should throw in a few viscerally appealing, but wildly inaccurate or unfair headlines, just to see what happens. I’d be willing to bet a month’s supply of vowels that those headlines prove to be far more popular than the fair, accurate ones.
This is great for a high traffic site. Any ideas on how you would scale it for lower traffic sites?
That’s a great question, Claire. I think you’d have to drop the real-time aspect and test over multiple days. Keeping the headline in the same location (and trying to adjust for other variables like overall traffic), you could run one version of the headline on Monday and tweak it for Tuesday. Doing that with lots of stories over a long period might start to reveal reader preferences. Too late for adjusting the headlines you’ve already written but certainly useful for writing future headlines.
Vello, I see your concern, but headlines have always served a promotional role and readers who are burned by misleading or overhyped headlines won’t come back, so there’s still an incentive to be accurate. I think human judgment and algorithms can work in tandem.
And thanks for your post and comment, Vadim. As it happens, The Washington Post just poached a Huffington Post editor, and Michael Calderone thinks it has something to do with the editor’s viral-headline-writing abilities. —Zach
Huff Post probably does the same with headlines in email newsletters, or at least headlines within email subject lines.
That is one way a low traffic site can test – run an A/B test in your newsletter, where you have focused clicks in a short amount of time (and can measure it), then stick with the most clicked version on your website.
Yes, I share the same name as the author of the article ;-)
future of journalism on the web right now…
Vello–I can’t believe that you would suggest that the Huffington Post would actually test “viscerally appealing, but wildly inaccurate or unfair headlines!” Oh wait, they save wild inaccuracy and unfairness for the content and fake news stories they run…
Yes, but the problem with the Huffington Post is that they sometimes use headlines that my have better CTR, but less honesty as a result.
Huffingington Post makes me cringle whenever I see the headlines — just so sensational. GIGANTIC font right in the front, hitting you in the face, making it seem like you were witnessing a moon landing.
OBAMA EATS A PANCAKE
[photo of obama]
Waffle Industry Enraged… Syrup or no?… Why breakfast is underrated…
Argh.
Very cool. Kinda reminds me of the BBC having that design that learned what you liked then rearranged the front page for you automatically. Good stuff. Keep it up.
-kpau