Why Storyful is expanding its business to work with brands
ABOUT                    SUBSCRIBE
Feb. 6, 2014, 10:33 a.m.

You won’t believe Upworthy’s new way of measuring audience engagement until you read it

“Attention minutes” combine traditional metrics with measurements aimed at figuring out how much time readers really spend with content.

Upworthy wants to be known as the company that shares a massive amount of socially conscious content across the web. What Upworthy is widely known for, however, is perfecting the art of the clickbait, curiosity gap headline. (On the front page now: “The Perfect Reply A Girl Can Give To The Question ‘What’s Your Favorite Position?'” “What If Everyone Who Reacted Negatively To A Super Bowl Ad Knew The Facts? They’d Learn This.” “A 16-Year-Old Explains Why Everything You Thought You Knew About Beauty May Be Wrong. With Math.”)

The company denies that they’re the main culprit, and some initial research by The Atlantic’s Robinson Meyer suggests that may be true. But now, they’re taking steps toward measuring their performance through something deeper than clicks.

This morning, Upworthy announced it would be using a new metric — “attention minutes” — as its primary tool to determine how they’re doing. The goal is to blend traditional eyeball-counting metrics with figures that more accurately measure engagement or how much the audience actually likes the content that they’re making. And they’ll be releasing their source code so other publishers can use it.

We love thinking this way because it rewards us for sharing content that people really enjoy and find valuable — not just stuff they click on a lot. It may mean that we don’t do quite as well on uniques or pageviews, but that’s a tradeoff we’re happy to make because this is a metric focused on real user satisfaction.

Upworthy says they decided to build attention minutes in the fall out of frustration with available analytics, either because they measure the wrong things or because they measured them poorly. For example, while Google Analytics’ time-on-site metric seems like it should provide something like what Upworthy was looking for, it was often misrepresentative, calculating at one point that the average Upworthy user spent 21 minutes on site. (“We’re good, but we know we’re not that good,” they write.)

Of course, Upworthy isn’t the only publisher feeling disatisfied with the mighty pageview and its many successors. Former Knight-Mozilla OpenNews fellow Brian Abelson, who now works at Enigma, spent part of his fellowship at The New York Times working on “pageviews above replacement,” a method of looking at basic traffic results without the confounding factor of internal promotion. James Robinson, director of news analytics at the Times, had this to say about metrics in his Nieman Lab prediction for news in 2014:

But for all types of newsrooms, developing a more qualitative understanding of audiences will be increasingly important. As publishers continue to invest in longform journalism and experiment with new types of storytelling, questions will naturally arise about the ways in which people actually read and engage with these pieces, beyond the feel-good tally of how many visitors they attract.

As Upworthy acknowledges, other media companies have also begun to make moves towards more meaningful metrics. “There are definitely others looking at how to improve metrics,” writes Upworthy CEO Eli Pariser in an email, citing YouTube’s “Time Watched,” Medium’s “Total Time Reading,” and Chartbeat’s “Average Engaged Time.” (Chartbeat responded to Upworthy’s announcement on their own blog, calling the metric “another big win.”)

“They all have a similar hunch,” Pariser said. “As we’ve been building it out it’s been nice to see additional traction on this front. It gives us a sense we’re not way out in left field.”

Attention minutes are calculated using an assemblage of what Upworthy calls “signals.” Pariser says they started out by asking, “What are the ways you can know if it’s a sad, lonely, ignored tab or if someone is watching intently?”

Some of these signals include the length of time a browser tab has been open, how long a video player has been running, and the movement of the mouse on screen. “This method doesn’t store raw information — it follows a stream of events and every few seconds asks, ‘Is this person still paying attention?'” Pariser said. “If they are, that sends a signal to a data warehouse that says, ‘Chalk up a few more attention seconds for this post.'” Together with other information, including pageviews, Upworthy hopes to get a sense of what content is most delighting and engrossing viewers, not just briefly capturing their attention.

This chart is meant to illustrate how the signals that make up attention minutes present a significantly different portrait of a story than pageviews do:


Upworthy plans to release the source code “in the coming months.” Pariser says Upworthy has already begun to gather valuable information about how people consume content — like that “the relationship between the length of a video and how much people watch isn’t what we though it would be,” he says. They’ll be publishing more of that information in 2014, too.

The overarching hope is that by improving the quality of your metrics, you can improve the quality of your work. Campbell’s Law puts it well, if academically:

The more any quantitative social indicator (or even some qualitative indicator) is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.

If you measure performance in pageviews, you encourage slideshows. If you measure performance by social shares, you encourage clickbait headlines and giant Like buttons. Finding a metric that lines up with a publisher’s goals is one of the most important things it can do to encourage better work, and Upworthy thinks this is a step closer to that alignment for their needs.

“The reason that we’re focused on this metric is not just because we think it’s good for us, but because we think it would serve everybody better if we moved away from some of the metrics that people have been using,” Pariser says. “Pageviews can be bought, but this…is a truer measure of whether you’re delivering something meaningful and valuable.”

POSTED     Feb. 6, 2014, 10:33 a.m.
Show comments  
Show tags
Join the 15,000 who get the freshest future-of-journalism news in our daily email.
Why Storyful is expanding its business to work with brands
It’s one element of a broader expansion for the social news agency, which is also growing its product team and working on improving its core trend-detection technology.
An ad blocker for tragedies: How news sites handle content around sensitive stories
For stories like the Germanwings plane crash, The New York Times and many other publishers flip a switch to remove ads to avoid unwanted connections.
Newsonomics: BuzzFeed and The New York Times play Facebook’s ubiquity game
The ubiquity game has different rules for digital startups than for legacy businesses. But for both, figuring out the right relationship with Facebook is key to their audience strategies.
What to read next
Millennials say keeping up with the news is important to them — but good luck getting them to pay for it
The new report from the Media Insight Project looks at millennials’ habits and attitudes toward news consumption: “I really wouldn’t pay for any type of news because as a citizen it’s my right to know the news.”
926The next stage in the battle for our attention: Our wrists
News companies have moved from print dollars to digital dimes to mobile pennies. Now, with the highly anticipated launch of the Apple Watch, the screens are getting even smaller. How are smart publishers thinking about the right way to serve users and maintain their attention on smartwatches?
792A wave of distributed content is coming — will publishers sink or swim?
Instead of just publishing to their own websites, news organizations are being asked to publish directly to platforms they don’t control. Is the hunt for readers enough to justify losing some independence?
These stories are our most popular on Twitter over the past 30 days.
See all our most recent pieces ➚
Encyclo is our encyclopedia of the future of news, chronicling the key players in journalism’s evolution.
Here are a few of the entries you’ll find in Encyclo.   Get the full Encyclo ➚
Gawker Media
The Ann Arbor Chronicle
Knight Foundation
The Chronicle of Higher Education
The Miami Herald