Twitter  Take these lessons from the rubble of Thunderdome nie.mn/UxzrDE  
Nieman Journalism Lab
Pushing to the future of journalism — A project of the Nieman Foundation at Harvard

Getting beyond just pageviews: Philly.com’s seven-part equation for measuring online engagement

As web analytics reports become a mainstay of news meetings, there’s a lot of nervousness about how attention to clicks will affect news coverage, and about the perceived incentives to produce high-trafficking junk news. Earlier this week, a web research company released a good-news study arguing that stories about substantive issues like unemployment and mortgage rates can actually bring in more revenue per pageview than celebrity crotch shots.

But two months ago, philly.com, home of the Philadelphia Inquirer and Daily News, began analyzing their web traffic with an “engagement index” — an equation that goes beyond pageviews and into the factors that differentiate a loyal, dedicated reader from a fly-by. It sums up seven different ways that users can show “engagement” with the site, and it looks like this: Σ(Ci + Di + Ri + Li + Bi + Ii + Pi)

I spoke with Chris Meares, senior data analyst at Philly.com, about how the equation works, and what it’s revealed so far about the newspaper’s users on the web. The first step in measuring engagement, Meares explained, is identifying which web behaviors show that users are “engaged.”

Working off a white paper called “Measuring the Unmeasurable: Visitor Engagement” by Eric T. Peterson and Joseph Carrabis, Meares sat down with Ryan Davis, the president of Philly.com, and Wendy Warren, the vice president for content, to hash out the meaning of engagement.

One possibility they considered was measuring engagement simply through how many visitors left comments or shared philly.com content on a social media platform. But that method “would lose a lot of people,” Meares said. “A lot of our users don’t comment or share stories, but we have people — 45 percent — [who] come back more than once a day, and those people are very engaged.”

They ultimately decided on seven categories, each with a particular cutoff:

Ci — Click Index: visits must have at least 6 pageviews, not counting photo galleries

Di — Duration Index: visits must have spend a minimum of 5 minutes on the site

Ri — Recency Index: visits that return daily

Li — Loyalty Index: visits that either are registered at the site or visit it at least three times a week

Bi — Brand Index: visits that come directly to the site by either bookmark or directly typing www.philly.com or come through search engines with keywords like “philly.com” or “inquirer”

Ii — Interaction Index: visits that interact with the site via commenting, forums, etc.

Pi — Participation Index: visits that participate on the site via sharing, uploading pics, stories, videos, etc.

Those are largely the same as in the Peterson/Carrabis white paper, although they use a Feedback Index (“captures qualitative information including propensity to solicit additional information or supply direct feedback”) in place of Philly.com’s Participation Index.

The next step was to track the percentage of overall visits that satisfy each of these categories. What percent of visits to the site lasted at least five minutes? What percent included a comment or other interaction? For instance, in a recent measure of the Click Index, out of 3.9 million total visits, 698,000 were visits where a user clicked through at least six pages — which comes out to a 17.9 percent engagement rate.

As well as paying attention to the engagement rates for each category, Meares also averages the seven individual percentages to create an overall engagement score — the average percent of visits to the site that broadly qualify as “engaged.”

Because the engagement percentage typically goes down as total site pageviews go up (new visitors are, by definition, not loyal ones), Meares multiples the overall engagement percentage by the total number of pageviews to get a estimate of the total number of engaged visits.

Last week, the overall engagement was 31 percent, which translates into an estimated 1.221 million engaged visits.

Month to month, the overall engagement score for philly.com has hovered around 35 percent, Meares said. He also tracks the levels of engagement for different areas of the site, including news, sports, and living. While he said the sports score isn’t actually as high as the 73-percent figure a Philly.com exec gave us last week, it is higher than the average engagement level across the site. In September, sports page visits were 46.6 percent engaged, while news page visits were only 34.4 percent engaged.

Tracking site visits with this level of specificity is time-consuming — Meares says he devotes about a third of his full-time job to analysis of the engagement equation — but it has produced some interesting information. For instance: “We’re definitely seeing the impact of social media and how it provides engaged visitors.” While Google and Yahoo provide a lot of traffic, the visits that they send to Philly.com don’t tend to be engaged. Only 20.34 percent of visits that come through Google are engaged visits. In comparison, 33.64 percent of visits that come via Facebook are engaged.

More broadly, Meares said, tracking engagement allows Philly.com to put traffic data in perspective. If overall traffic for the site is down, but the number of engaged users are up, that still means the site is doing well, Meares said.

Newspapers in other markets have come to Philly.com for advice about measuring engagement on news sites. “It’s a really big challenge,” said Sonia Meisenheimer, digital marketing strategist for the St. Petersburg Times’ tampabay.com. “There’s not an expressed moment of loyalty. You have to basically be a diviner with a divining rod and go around and tap on all these different things.”

Meisenheimer said she found philly.com’s engagement equation “fascinating” but impractical for her requirements. “They can do this because they have a web analytics person full time looking at ‘how do we measure success?’” she said. “We could never sustain the reporting and tracking.”

While the equation might provide interesting feedback to editors, Meisenheimer said she thought the results it produced were too complicated. In a competitive market, businesses want to compare different web outlets on apples-to-apples factors like total audience and local audiences. She said tampabay.com would measure engagement in a much simpler way, focusing on registration numbers and their brand index, or how many people come to the site through a bookmark or by searching for terms like “St. Pete Times” or “tampabay.com.”

Internally, she said the most important number for tampabay.com is simply revenue per unique visitor. “My question to philly.com is: how does this help you make more money? Because I don’t see that in the equation.”

                                   
What to read next
how-why-explainer-explanatory-cc
Ken Doctor    July 25, 2014
When people talk about explanatory journalism, the focus is on new players like Vox and FiveThirtyEight, or on giants like the Times and the Post. But can connecting the dots trickle down to the local level?
  • http://philly.com Chris Meares

    First I want to thank Lois for putting together a wonderful article. I honestly don’t believe enough time is spent on web analytics in the media sector which can be seen by Sonia Meisenheimer’s comments about not having enough resources to track engagement on tampabay.com’s website. For a web-based business to not have a full time web analytics person on staff is irresponsible and is wasting money. The engagement metric is not as complicated as it seems at first blush and the only reason that it takes up so much of my time is due to the importance we are placing on the engagement of our visitors. Advertisers are looking for an engaged audience and the ability for philly.com to directly measure this audience and be able to show an advertiser their message is being seen by an engaged audience is extremely important.

  • http://www.tatvic.com/blog Ravi Pathak

    Hey Lois,

    I think this is great article. However , probably what might be a great addition is the a co-efficient of co-relation figure between the engagement metric & the revenue. The purpose of entire engagement indexing is probably to come up at one unique # at any given time that indicates visitor’s engagement. As Chris said above, having understanding of co-relation between revenue & engagement would be a great advantage. Also having corelation between new features and changes in engagement is also a great indicator for development cycle.

    All in all great work !

  • http://www.webstarresearch.com Ali Shah

    Excellent article, and thank you for sparking this wonderful discussion! Newspapers are indeed behind in fully leveraging web analytics to make informed business decisions, so it’s great to see some intelligent thought-provoking insight surrounding engagement (definitely more important than page views!).

    But at the end of the day, the engagement metric (any which way you decide to calculate it) is a reflection of page views i.e. the more engaged visitors are, the more pages they consume, creating more inventory ad revenue (if the model is ad-driven). From data that I’ve seen, local traffic is a key driver of engagement. Capitalizing on local driven strategies is key.

    Thanks again for the interesting post!

  • http://analyticsimpact.com Ophir Prusak

    I agree with Chris and Ravi that tying this back to revenue is very important.

    What about adding a metric like banner CTR or number of banner impressions to the mix?

  • http://blog.scoutanalytics.com Matt Shanahan

    Engagement is the unit of monetization in digital publishing. The amount of engagement an audience provides defines the revenue capacity for the publisher. Being able to forecast, segment, and price discriminate engagement for advertisers is the key to revenue optimization in digital publishing.

    Other industries such as airlines, hotels, etc. have used predictive analytics to optimize revenue for their perishable inventory. Predictive analytics will be critical to solving this problem for publishers as well.

  • http://www.twitter.com/liman Erin

    Click index seems problematic – the more people share content that is indexed by Google, the more targeted the view. The impact of click index might need to degrade as participation increases and content becomes reference-able in fragments.

    Perhaps we could re-purpose plagiarism tools to track the proliferation of fragments as campaigns…

  • http://www.tampabay.com Sonia Meisenheimer

    Don’t get me wrong – we do track engagement and consider it extremely important but this is a relatively new consideration and we do not have a full-time analytics resource dedicated to the task. The resource constraint simply limits the complexity that we can sustain right now. In a perfect world, that would be fantastic. Chris, you are obviously a tremendous asset to philly.com and I wish that I got to work with someone like you.

    On a different note, it is important to note that newspapers are not currently pure-play online businesses and still derive a comparably small amount of their revenue from online. We are not venture funded, but funded by our print revenues – and read the news on that :) We are invested in engagement, just using a different recipe.

  • Pingback: journalists turn to 7-part mathematical formula to gauge online readership [no, seriously] « conversations of doom.

  • http://www.pzunk.net Pattie reaves

    A great article about a topic that isn’t covered nearly enough. :)

  • http://www.webanalyticsdemystified.com Eric T. Peterson

    First, as the author of the paper cited above, I have to say that I am hugely excited to see such a great example usage of the engagement metric I first described back in 2007. This is a positively brilliant use of the metric and it’s great to read about it in the public domain.

    Second, I want to mention that I am ** more than happy ** to discuss the engagement calculation and it’s use with any of your readers and/or the foundation. I don’t get to Boston often enough but would love to come out or do a WebEx … just let me know!

    Finally, regarding the comment about tying engagement to revenue … don’t do it! Even in the media and publishing model, engagement and revenue are different aspects of consumer behavior. A consumer can be very engaged with your site but not be tremendously profitable … but you still want a way to measure their engagement independent of profit. Same for satisfaction and engagement — they are different aspects of the consumer experience.

    Don’t get me wrong, determining the correlation between engagement, satisfaction, and revenue is tremendously important, but it’s not something you’ll be able to do in the run-of-the-mill web analytics package (you’ll need more powerful tools, trust me!) Work to understand what my (or any) engagement metric can tell you about your audience first, then go looking for the relationship between engagement and money.

    Anyway, great article and thanks to Philly.com and Chris for being willing to talk about their use of the metric! Again I’m happy to discuss.

    Eric T. Peterson
    Web Analytics Demystified
    http://www.webanalyticsdemystified.com
    eric@webanalyticsdemystified.com
    @erictpeterson

  • Pingback: Our Engagement Metric in use at Philly.com | Web Analytics Demystified

  • http://www.jimnovo.com/ Jim Novo

    Those looking for simplicity and a direct tie to Revenue might try Recency by itself – one metric solely focused on likelihood to visit again. Then you are using simple prediction to make decisions rather than aggregating history.

    Another way to say this: does it matter more to you what kind of content engaged visitors in the past, or what kind of content attracts visitors who are likely to remain engaged in the future?

    Example here:

    http://blog.jimnovo.com/2007/04/03/recency-defines-engagement/

    If you’re using Google Analytics, go take a look at Recency under Visitor Loyalty, you should see similar data. Apply segments and you have it – content, campaigns, or whatever segments most likely to generate visitors who come back.

    If you want to “prove it” like I did in the blog example provided above, I’m afraid you’ll need a different tool. But the Recency effect is well known and you should have no trouble supporting it.

  • Michelle Licudine

    Pretty brilliant, and I suspect this is what most newspapers are trying to get at.

    Does the Participation index include sharing on external sites and/or social media? If not, what about the social angle?

  • http://twitter.com/mattoperry Matt Perry

    Thanks for this great article.

    Here at Grist.org we’ve begun tracking a subset of the data that goes into the index described here. We closely watch the data that I presume would be associated with C and I above — in other words any user activity other than passive consumption of content.

    While things like recency, click-depth of visit and duration of visit are surely engagement measures, they all share a common currency — the page view. We wanted our engagement index to focus on user behaviors other than just passive reading.

    So our index (which is expressed as an index, not a %) tracks behaviors like sharing, commenting, liking, registering, logging in and donating. We also weight these different behaviors according to their altitude on the engagement ladder. So, for example, commenting on a story may have a larger affect on the index than simply facebook liking it.

    Two important points:
    - the index is completely automated. We could not support it if it were not.
    - the index is only useful insofar as it is actionable. It’s early days yet, but our intent is to use this data to help us modify content authoring practices, and do make design and functionality decisions on the site.

    Thanks again for this article and discussion — it’s helpful to know that others are thinking along these lines.

  • http://blog.scoutanalytics.com Scout Analytics

    Correlating engagement and revenue can give a publisher deep insight into revenue optimization potential. This press release is an explains the benefits Elsevier BI received from tracking engagement vs. revenue.

    http://scoutanalytics.com/press_release_full.asp?pdx=49

    Which members of the audience generate the most revenue from what engagement? This is knowable by linking the ad server stats to the web analytics. Understanding engagement is good. Understanding profitable engagement is better.

  • http://www.nwsem.com Jason L.

    Neat article. The most interesting piece is the twist thrown in by Ms. Meisenheimer. At first I had an opinion similar to Mr. Meares. But, after re-reading the article 3 or 4 times, I think Meisenheimer has a point. I would not over simplify the issue the way she has, but really, engagement is just a means to an end in almost every case (unless selling impressions is really how you pay your rent). Understanding how and if engagement impacts sales is the real value. I will certainly be looking for ways to incorporate this more complete understanding of engagement into my metrics, while still understanding that sales are what pay the bills.

  • http://www.medianmarketing.blogspot.com Pramod

    This is one of the best article on engagement I have come across so far. Even though, many print newspapers ( with online presence) may find it cumbersome to practice, purely online publishers will find this method almost an enlightenment. But as we know already, web analytics is not 100% perfect. For e.g., the calculation of unique visitors is flawed. Everytime, my computers’ cache is cleared, I will become a unique visitor to the same old site I keep visiting probably everyday. So I will have to rely on other tools like my email campaign tracker, ad tracker etc. So to really arrive at this index, all the tools need to be integrated and automated.

    Even if this is not possible, this 7-point indexer is a definitive guide to deep-down online analytics and success.

    -Pramod
    @medianmarketing

  • http://www.TheBeeDance.com R. A. Burrell

    This is fascinating. We arrived at a similar formula. But our goal was to measure the performance for all marketing behavior in tourism. It’s similar in that we want to know how content is producing certains behaviors. Here, you want to know shares, etc.

    You can simplify your equation to a Behavior and its Index. It’s Σ(B * i). It’s a series for n = 1 to the end. The index is the value for each different type of Behavior.

    We’ve been working on this for years and in practice I don’t see why you have to keep all seven of those separate. I could be missing something.

  • Pingback: Engagement: Where does revenue fit in the equation? » Nieman Journalism Lab

  • Pingback: Philly.com Measures 7 Variables of Audience Engagement | Future Changes

  • Rick Conrad

    What a great idea and what an invaluable tool for news sites. It’s easier than ever before to know where our audience comes from and what they actually read, thanks to analytics, and this is a natural and welcome evolution.

    As for the revenue question, being able to measure engagement is vital for generating more revenue from existing sources and tapping into new ones. Many advertisers want to reach a general audience, but I’m willing to bet that a premium could be attached to reaching a very engaged, social-media savvy segment of your audience, for example.

    Also, the engagement metric would come in very handy when developing new features. Imagine your developers and news staff being able to focus in on what exactly keeps people on your site longer and why, as in the Google vs. Facebook example. Perhaps knowing that, beefing up social media tools and community input and interaction would be a better way to spend your developers’ and designers’ time than worrying as much about
    crafting your headlines or stories for detection by the Google bot.

    This kind of knowledge would give you an idea of what new things might work, therefore saving money, at the very least. Ultimately, it could generate gobs of revenue.

    Being able to measure engagement would also be invaluable in developing premium, user-pay features or products, without walling off your current content and alienating your already engaged, committed readers.

    Great stuff. Congratulations to Chris Meares and philly.com for such a valuable tool.

  • Pingback: Coinvolgimento Quotidiano [on line] « Il Giornalaio

  • http://www.resourcenation.com Matt K

    I think it’s really important that business owners realize that not everything in business is “measurable.” We are so weak to statistics, but we often fail to recognize that data and statistics can often be skewed. The level in which your customers are passionate and buzzing about your company is what is important. The bottom line philosophy is such a 90′s adage.

    I think this is why social media is such a phenomenon. It allows a company to tap in to their customers brains without market research, engage, and retain them.

  • Pingback: This Week in Review: WikiLeaks’ latest doc drop, the NPR backlash, and disappointing iPad magazines » Nieman Journalism Lab

  • http://www.craigmclaughlan.com/ Craig McLaughlan

    “regarding the comment about tying engagement to revenue … don’t do it! Even in the media and publishing model, engagement and revenue are different aspects of consumer behavior. A consumer can be very engaged with your site but not be tremendously profitable … but you still want a way to measure their engagement independent of profit”

    Interesting to know which visits are realy engaging, but why dedicate a third of your time to a metric that ultimately is of curiosity value.

  • Pingback: Testing an engagement metric and finding surprising results

  • Pingback: The Emergence of New Success Criteria in the Audience Marketplace | audienceevolution

  • Pingback: Lonely, Mobile, Manifesto, Realign, Index [Thursday Links] « Grist Labs

  • Pingback: culture360.org Development Blog - Web metrics for online engagement

  • Pingback: Unicorn Metrics

  • Pingback: How to measure reader engagement « MediaMaths

  • Pingback: Google +1 « Desiree LaBombard

  • Pingback: The Museum of the Future » How to measure engagement and participation? An experiment with Google Analytics

  • Pingback: A resource for newsrooms: Measuring the success of audience engagement efforts | RJI « 2010

  • Pingback: 0 hyperlocals