Is the importance of audience engagement largely anecdotal and abstract? What is it actually delivering? Do audiences want a direct connection with journalists as much as the startups that focus on engagement claim that they do? In a paper published in Digital Journalism last month (summarized on CJR on Monday) Northwestern Ph.D candidate Jacob Nelson looks at three engagement-based startups and is left skeptical about whether they are fixing a problem that real people actually have or are largely just appealing to journalists who have an unquantifiable sense that something is wrong.

Nelson embedded with Hearken, a startup that helps newsrooms create engagement strategies, for three months. In his paper, “The Elusive Engagement Metric,” he argues:

Hearken’s pitch to newsrooms relies primarily on appeals to intuition. Its employees argue that their interpretation of audience engagement will lead to a better quality of journalism, which will inevitably result in increased audience revenue as well. Though some newsrooms refuse to invest in Hearken’s offerings without proof they will yield some measurable return, others seem eager to take the chance…

As a result, Hearken’s appeal to these newsrooms is more or less, “Our way is better. Trust us.”

As things stand now, Nelson writes, there’s no standard definition of audience engagement. Does it mean “audience attentiveness,” or journalism that is “more interactive or participatory,” or journalism that “encourages audiences to participate in civic life”? Does it build loyalty? Does it change audience preferences? (“Although many in journalism assume that audiences are inherently uninterested in public affairs news, some audience engagement advocate think that audiences would eagerly tune into these stories if they felt more included in the reporting process.”)

Hearken’s method has been to invite readers into the reporting process by inviting them to submit questions that they think should be investigated and then, down the line, possibly even including them in the investigation process. An annual Hearken subscription fee is around $8,500. Nelson spent three days a week in Hearken’s offices for three months, read internal memos, monitored its Slack channel, sat in on Hearken’s pitches to newsrooms, and interviewed several employees multiple times.

Here are some of his observations from the full paper:

— “While the newsrooms participating in the pitch meetings I observed often immediately pointed to pageviews as a primary indicator for story performance, some did so more sheepishly than others. For example, one journalist from a west coast, daily newspaper responded, “We look at pageviews…but we don’t apply a measuring stick to too many things. We’re trying to look at the big picture.”

— “When Hearken staff would pitch their services, they would first try to convince potential clients that a small news audience was not their primary problem, but a symptom of their failure to adequately engage the public. Because the claim that news organizations are struggling to attract large audiences is easily verified by journalism’s currency, it goes all but unquestioned throughout the news industry. Hearken’s claim that the solution to this problem is to increase audience engagement, on the other hand, is more or less based entirely in the intuition and experiences of Hearken’s founder and staff.”

— One employee: “The only data that we have, that we, like, own, you know, is how many questions are they getting, how many votes are they getting, which I don’t actually think is that important, honestly.”

— “Hearken’s staff could not cite an engagement metric to demonstrate the impact of its tools and services because no such metric that has industry-wide backing currently exists. Instead, the company’s employees would draw on anecdotes that similarly functioned as appeals to intuition.”

— “There is so much confusion within the current news media landscape that Hearken’s pitch of ‘Journalism’s relationship with the audience is broken. We can help you fix it,’ ends up having a profoundly effective appeal. As [cofounder and CEO Jennifer Brandel] put it during one of our interviews, ‘We have to try something’…Sometimes Brandel seemed exasperated by what she saw as the news industry’s focus on metrics at the expense of common sense. In our final interview, she asked, “What things should we just say, ‘You know what? There doesn’t need to be data. We all know this to be true.'”

Reading this, I wondered why Hearken let Nelson hang with them in the first place. This paper couldn’t have been the result that they were hoping for. But how fair is this criticism? Nelson could have looked harder for studies about what audiences want from the news and incorporated them more fully into his paper; one of his big criticisms is that Hearken doesn’t present enough of these data points to potential newsroom partners. (Brandel says that Hearken does indeed have this data, posted in case studies on its website.) But a lot of audience engagement projects are difficult to quantify, which doesn’t make them less worthy.

“We enjoyed having Jacob around the office last year to study Hearken. Having been a reporter himself, he understands engagement and journalism in ways few other academics do,” Brandel told me in an email. “That said, he was only with us for a short period last year, and like all young companies, we’ve grown up, matured and evolved a lot since then…The return on investment for engagement work is something we think, talk and learn about every day, alongside many other great minds in the industry.” She pointed to case studies that Hearken has added to its website recently.

“How could we empirically measure a parent’s love?” Brandel wondered. “And in the absence of hard data, would we have to conclude that it’s not valuable? The tyranny of the quantifiable is something we at Hearken, and many others doing this work, are trying to both acknowledge and transcend.”

Here's some data for you! https://t.co/SciV9T0L53
Jacob's study is great – it's just that academic research and gathering data takes a long time. Many things have changed since he studied us, and we have results.

— Jennifer Brandel (@JenniferBrandel) April 30, 2018

Nelson ultimately bemoans the lack of a single “engagement metric…with industry-wide backing,” but it’s not Hearken’s fault that one doesn’t exist; this is something that news companies have been grappling with for a long time.

If you want to call Hearken's model unproven, fine. But to say that audience engagement may or may not be a thing would basically argue that every modern media company has just been going by "gut" for all these decades.

— Joshua Benton (@jbenton) April 30, 2018

This article is a real disappointment – it ignores all kinds of data, research and work happening in audience engagement. Hearken has lots of data (not cited here) about how their stories consistently outperform non-Hearken stories, plus a revenue study w/ @BitchMedia. https://t.co/y3zzPmHv3a

— Molly de Aguiar (@MollydeAguiar) April 30, 2018

The Bitch Media study actually is mentioned in Nelson’s full paper, though not the CJR post. In that 2017 study, Bitch Media found that:

“Looking at all the readers on its email list, Bitch Media found that just 1.35 percent of them converted to membership during the year-long study. But when it focused on those email list members who had engaged with the Hearken platform, the conversion rates surged to 7.17 percent. Readers who signed up for the email list for the first time after reading a Hearken-prompted story, meanwhile, converted to membership at a rate of 2.02 percent — nearly double the rate for users who signed up from the email list through another non-Hearken path….

[Bitch Media] expects to generate a total of $11,900 in Hearken-related extra membership support in its 2017 budget year — or nearly $6,000 in extra money after paying for the platform.

“The process by which Bitch Media’s staff made this discovery was onerous, which is likely one of the reasons it has not been replicated,” Nelson complains. But isn’t this the exact kind of data he claimed he was looking for? How else were they supposed to measure it? It’s gonna be onerous.

People who want quick answers and clear metrics to building trust that has never existed in many communities are going to be disappointed.

— Molly de Aguiar (@MollydeAguiar) April 30, 2018

ProPublica’s Dick Tofel thinks that the criticism is valid — but also acknowledges that ProPublica’s lauded engagement efforts are mostly judged on anecdotal evidence too.

I think this piece poses a real challenge, Molly. I hope Hearken will respond directly, and with the sort of data you mention. This emerging field is long on rhetoric, and could use some more rigor.

— Richard Tofel (@dicktofel) April 30, 2018

some, but mostly it's anecdotal, tbh.

— Richard Tofel (@dicktofel) April 30, 2018