Nieman Foundation at Harvard
The Society of Professional Journalists faces a “dire situation”
ABOUT                    SUBSCRIBE
Aug. 11, 2017, 10:04 a.m.
Audience & Social

Remember that Norwegian site that made readers take a quiz before commenting? Here’s an update on it

For one thing, people did really, really badly on the quizzes (although that could be due to a language barrier).

When my former colleague Joseph Lichterman wrote about a Norwegian news organization that makes readers pass a quiz on the article before they can comment on it (one of the most-trafficked stories in Nieman Lab history, by the way) the site — NRKbeta, the tech vertical of Norway’s public broadcaster — was lauded for its creativity. But NRKbeta’s editors and journalists said it was too early to tell if the program was a success.

But now, five months in? NRKbeta’s team says readers may have treated the quizzes on 14 articles more like reading comprehension games than as a gateway to the comments section.

As Lichterman described the feature, back in March:

On some stories, potential commenters are now required to answer three basic multiple-choice questions about the article before they’re allowed to post a comment. (For instance, in the digital surveillance story: ‘What does DGF stand for?’)

The goal is to ensure that the commenters have actually read the story before they discuss it.

‘We thought we should do our part to try and make sure that people are on the same page before they comment. If everyone can agree that this is what the article says, then they have a much better basis for commenting on it,’ said NRkbeta journalist Ståle Grut.

Forcing users to take a little extra time to think about the comment they’re about to post also helps them think about tone, NRKbeta editor Marius Arnesen said. ‘If you spend 15 seconds on it, those are maybe 15 seconds that take the edge off the rant mode when people are commenting,’ Arnesen said.

Now, five months since launching the comment-quiz, here’s what NRKbeta has to say about the results.

“On average, there is a lot more attempts — both correct and wrong — than actual comments.” In a post on NRKbeta, Grut explained that it appears more people took the quiz for reading comprehension than for access to the comments section. Some may treat it as a “fun little game after reading,” Grut wrote. Of the 14 articles that included quizzes, the highest number of comments on any one was 77. Meanwhile, more than 300 people took the quiz for that same article.

[Note: See that super-long red bar above? “Due to a human error, the right answer to one of the questions was not indicated. It made it impossible to pass the quiz. Hence the many logged wrong attempts.”]

“We have not established any set rules for when the quiz should be enabled.” NRKbeta opted not to require the quiz for every article’s comment section. While the quiz was built as a WordPress plug-in with questions randomized for each user, the team did not want to overwhelm commenters on quick and light-hearted subjects.

Grut noted that by requiring potential commenters to become more invested in the content, the comment-quiz favored “the most eager with the most time on their hands.” But it didn’t always result in better comprehension over time: “From time to time this has lead to a decline in quality and tone, causing [the reader] to often abandon the quiz module.”

“On average, there is a staggering error rate of 72 percent on the quiz.” This may have been because international readers invaded after our story got picked up, and tried to take a quiz written in Norwegian. The clearest finding, Grut noted, was that readers liked treating the quiz as a game. There has been an uptick in “gaming the news” or developing online games for audiences to consume news or to even learn about fake news. Games like these quizzes could potentially help reporters and editors determine if the wording and explanations in their articles are clear to readers, in addition to cooling the comments section’s hotheads.

“We’re trying to establish a common ground for the debate,” Arnesen told Lichertman earlier this year. “If you’re going to debate something, it’s important to know what’s in the article and what’s not in the article. [Otherwise], people just rant.”

Photo of multiple choice test by Alberto G. used under a Creative Commons license.

POSTED     Aug. 11, 2017, 10:04 a.m.
SEE MORE ON Audience & Social
Show tags
Join the 60,000 who get the freshest future-of-journalism news in our daily email.
The Society of Professional Journalists faces a “dire situation”
“If we don’t change our thinking, the next incoming president will be the last president.”
Four disabled journalists on how news outlets can support staffers and audience members with disabilities
“The tools that journalists are given [should be] accessible — and designed with people like me in an advisory role.”
Press freedom means controlling the language of AI
Generative AI systems act like “stochastic parrots,” using statistical models to guess word orders and pixel placements. That’s incompatible with a free press that commands its own words.