Nieman Foundation at Harvard
America’s Test Kitchen, “the Consumer Reports of cooking,” wants to grow to new platforms
ABOUT                    SUBSCRIBE
Oct. 26, 2010, 10 a.m.

Getting beyond just pageviews:’s seven-part equation for measuring online engagement

As web analytics reports become a mainstay of news meetings, there’s a lot of nervousness about how attention to clicks will affect news coverage, and about the perceived incentives to produce high-trafficking junk news. Earlier this week, a web research company released a good-news study arguing that stories about substantive issues like unemployment and mortgage rates can actually bring in more revenue per pageview than celebrity crotch shots.

But two months ago,, home of the Philadelphia Inquirer and Daily News, began analyzing their web traffic with an “engagement index” — an equation that goes beyond pageviews and into the factors that differentiate a loyal, dedicated reader from a fly-by. It sums up seven different ways that users can show “engagement” with the site, and it looks like this: Σ(Ci + Di + Ri + Li + Bi + Ii + Pi)

I spoke with Chris Meares, senior data analyst at, about how the equation works, and what it’s revealed so far about the newspaper’s users on the web. The first step in measuring engagement, Meares explained, is identifying which web behaviors show that users are “engaged.”

Working off a white paper called “Measuring the Unmeasurable: Visitor Engagement” by Eric T. Peterson and Joseph Carrabis, Meares sat down with Ryan Davis, the president of, and Wendy Warren, the vice president for content, to hash out the meaning of engagement.

One possibility they considered was measuring engagement simply through how many visitors left comments or shared content on a social media platform. But that method “would lose a lot of people,” Meares said. “A lot of our users don’t comment or share stories, but we have people — 45 percent — [who] come back more than once a day, and those people are very engaged.”

They ultimately decided on seven categories, each with a particular cutoff:

Ci — Click Index: visits must have at least 6 pageviews, not counting photo galleries

Di — Duration Index: visits must have spend a minimum of 5 minutes on the site

Ri — Recency Index: visits that return daily

Li — Loyalty Index: visits that either are registered at the site or visit it at least three times a week

Bi — Brand Index: visits that come directly to the site by either bookmark or directly typing or come through search engines with keywords like “” or “inquirer”

Ii — Interaction Index: visits that interact with the site via commenting, forums, etc.

Pi — Participation Index: visits that participate on the site via sharing, uploading pics, stories, videos, etc.

Those are largely the same as in the Peterson/Carrabis white paper, although they use a Feedback Index (“captures qualitative information including propensity to solicit additional information or supply direct feedback”) in place of’s Participation Index.

The next step was to track the percentage of overall visits that satisfy each of these categories. What percent of visits to the site lasted at least five minutes? What percent included a comment or other interaction? For instance, in a recent measure of the Click Index, out of 3.9 million total visits, 698,000 were visits where a user clicked through at least six pages — which comes out to a 17.9 percent engagement rate.

As well as paying attention to the engagement rates for each category, Meares also averages the seven individual percentages to create an overall engagement score — the average percent of visits to the site that broadly qualify as “engaged.”

Because the engagement percentage typically goes down as total site pageviews go up (new visitors are, by definition, not loyal ones), Meares multiples the overall engagement percentage by the total number of pageviews to get a estimate of the total number of engaged visits.

Last week, the overall engagement was 31 percent, which translates into an estimated 1.221 million engaged visits.

Month to month, the overall engagement score for has hovered around 35 percent, Meares said. He also tracks the levels of engagement for different areas of the site, including news, sports, and living. While he said the sports score isn’t actually as high as the 73-percent figure a exec gave us last week, it is higher than the average engagement level across the site. In September, sports page visits were 46.6 percent engaged, while news page visits were only 34.4 percent engaged.

Tracking site visits with this level of specificity is time-consuming — Meares says he devotes about a third of his full-time job to analysis of the engagement equation — but it has produced some interesting information. For instance: “We’re definitely seeing the impact of social media and how it provides engaged visitors.” While Google and Yahoo provide a lot of traffic, the visits that they send to don’t tend to be engaged. Only 20.34 percent of visits that come through Google are engaged visits. In comparison, 33.64 percent of visits that come via Facebook are engaged.

More broadly, Meares said, tracking engagement allows to put traffic data in perspective. If overall traffic for the site is down, but the number of engaged users are up, that still means the site is doing well, Meares said.

Newspapers in other markets have come to for advice about measuring engagement on news sites. “It’s a really big challenge,” said Sonia Meisenheimer, digital marketing strategist for the St. Petersburg Times’ “There’s not an expressed moment of loyalty. You have to basically be a diviner with a divining rod and go around and tap on all these different things.”

Meisenheimer said she found’s engagement equation “fascinating” but impractical for her requirements. “They can do this because they have a web analytics person full time looking at ‘how do we measure success?'” she said. “We could never sustain the reporting and tracking.”

While the equation might provide interesting feedback to editors, Meisenheimer said she thought the results it produced were too complicated. In a competitive market, businesses want to compare different web outlets on apples-to-apples factors like total audience and local audiences. She said would measure engagement in a much simpler way, focusing on registration numbers and their brand index, or how many people come to the site through a bookmark or by searching for terms like “St. Pete Times” or “”

Internally, she said the most important number for is simply revenue per unique visitor. “My question to is: how does this help you make more money? Because I don’t see that in the equation.”

POSTED     Oct. 26, 2010, 10 a.m.
Show comments  
Show tags
Join the 15,000 who get the freshest future-of-journalism news in our daily email.
America’s Test Kitchen, “the Consumer Reports of cooking,” wants to grow to new platforms
“We’d like to move to other platforms, particularly as we see the changes in how people consume television.”
A program from Poynter and ONA is helping foster a community of female leaders in digital media
The Women’s Leadership Academy provides camaraderie and concrete advice beyond a bundle of platitudes.
Come talk ad blockers with Nieman Lab and a set of experts in New York
We’re having our first event in New York City with industry leaders: Wednesday, December 2 at 6 p.m.
What to read next
How one blog helped spark The New York Times’ digital evolution
“I certainly had editors tell me that I shouldn’t be wasting my time on Bird Week. But that was the best part of City Room…We were like unsupervised children.”
572News outlets left and right (and up, down, and center) are embracing virtual reality technology
Among those experimenting is The Wall Street Journal, which plans to open source its 360-degree mobile video and VR technology and hopes to turn VR into more of a mainstay of its storytelling.
502Podcasting in 2015 feels a lot like blogging circa 2004: exciting, evolving, and trouble for incumbents
The same trends we saw a decade ago — professionalization on one hand, platformization on the other — sure seem to be playing out again.
These stories are our most popular on Twitter over the past 30 days.
See all our most recent pieces ➚
Fuego is our heat-seeking Twitter bot, tracking the links the future-of-journalism crowd is talking about most on Twitter.
Here are a few of the top links Fuego’s currently watching.   Get the full Fuego ➚
Encyclo is our encyclopedia of the future of news, chronicling the key players in journalism’s evolution.
Here are a few of the entries you’ll find in Encyclo.   Get the full Encyclo ➚
The Washington Post
St. Louis Globe-Democrat
Windy Citizen
Texas Tribune
NBC News
Center for Investigative Reporting
Sacramento Press