Nieman Foundation at Harvard
Can an algorithm sift through piles of content to deliver top advertising dollars to top quality journalism?
The News Quality Scoring Project is relying on some humans and, well, an algorithm to separate “commodity news” from “value-added news” in an attempt to connect high quality content with advertisers who want better exposure.
It’s a tricky topic, but one that Frederic Filloux (the founder of digital media business newsletter Monday Note, a 2017 Knight Fellow, and now a Knight Senior Research Fellow) is attempting to tackle through the NQS project. Filloux is designing a mechanism to assess news articles based on quantifiable and qualitative signals — like word count, freshness versus evergreen material, quote density, contextualization, and the presence of a byline — in order to surface higher quality content for both readers and publishers.
“The digital publishing system has the big inconvenience of flattening everything,” Filloux said. It’s much more expensive to produce a six-month, heavily fact-checked investigation than “a piece of news put together by a couple of interns” — but the return on investment in terms of advertising funds is the same per click. His system, however, assigns each piece of news a score based on the value-add (determined by the news organization) and the previously mentioned signals; these signals can then be detected by an ad server to serve “pricier, premium advertising.”
Filloux is aiming for a human/machine, subjective/objective hybrid as the still-under-construction algorithm learns. With the algorithm comparing signals and “qualified professionals” manually assigning labels to news items, the scoring system should distance “commodity news” (content based on page views, churn, etc.) from “value-added news” (original reporting that involves balance, expertise, and innovation).
The resulting score, assigned on a four- or five-point scale, will remain internal to the News Quality Scoring Project to both prevent publishers from gaming the system, and to avoid shaming journalists whose jobs may require them to produce a certain kind of content. Still, Filloux said he is committed to transparency and wants to partially open-source the project at some point.
“The last thing I want to be is a censor or some sort of arbitrator of quality,” he said. “This is not my job. The last thing I want to do is ultimately judge the quality of a journalist themselves. Every journalist I know of is willing to take time to do the reporting, doing in-depth journalism,” but “sometimes we all have to” throw together less-than-glorious content.
In the next few weeks, Filloux will be opening the system to some human testers to help the machine learning process the signals. He doesn’t expect to cull any meaningful results until the end of the first quarter of 2018, and will remain at Stanford until the end of April. He received $50,000 from the Knight Foundation as part of his backing, but said he will need much more funding and support to continue hashing out the algorithm. (Disclosure: Nieman Lab also receives funding from the Knight Foundation.) “The problem is we are dealing with super dirty sets of data,” he said, complicated by the inconsistencies in news organizations’ content management systems.
Ultimately, the project could go beyond connecting advertisers and publishers. Filloux also wants to encourage news organizations to harness their ability to recommend personalized content to readers, looking to Netflix’s recommendation engine as a large-scale example. “Curation is a very important step,” he said. “It broke my heart to see that since I’ve been in this business, none of the [successful recommendation engines] have been actually produced by existing news organizations. You see a lot of applications doing that for money and so on. You don’t see any great media using their recommendation power — their credibility to put them in a position to recommend.” (Some, though, are working on it.)
Helping publishers make the most bang for the buck on a journalist-centric algorithm is an admirable goal. At the same time, algorithms have been known to be biased, and Filloux acknowledged in a recent Monday Note post that news data is “inherently complex to classify because of their fuzzy and unstable nature.” (He described how one op-ed about Harvey Weinstein was connected to another piece about President Donald Trump’s response to Hurricane Harvey.)
In the title of the same post, Filloux pointed it out himself: Scoring news stories is hard.
