Nieman Foundation at Harvard
Former Gawker employees are crowdfunding to relaunch a that’s owned by a nonprofit and funded by readers
ABOUT                    SUBSCRIBE
Aug. 11, 2017, 10:04 a.m.
Audience & Social

Remember that Norwegian site that made readers take a quiz before commenting? Here’s an update on it

For one thing, people did really, really badly on the quizzes (although that could be due to a language barrier).

When my former colleague Joseph Lichterman wrote about a Norwegian news organization that makes readers pass a quiz on the article before they can comment on it (one of the most-trafficked stories in Nieman Lab history, by the way) the site — NRKbeta, the tech vertical of Norway’s public broadcaster — was lauded for its creativity. But NRKbeta’s editors and journalists said it was too early to tell if the program was a success.

But now, five months in? NRKbeta’s team says readers may have treated the quizzes on 14 articles more like reading comprehension games than as a gateway to the comments section.

As Lichterman described the feature, back in March:

On some stories, potential commenters are now required to answer three basic multiple-choice questions about the article before they’re allowed to post a comment. (For instance, in the digital surveillance story: ‘What does DGF stand for?’)

The goal is to ensure that the commenters have actually read the story before they discuss it.

‘We thought we should do our part to try and make sure that people are on the same page before they comment. If everyone can agree that this is what the article says, then they have a much better basis for commenting on it,’ said NRkbeta journalist Ståle Grut.

Forcing users to take a little extra time to think about the comment they’re about to post also helps them think about tone, NRKbeta editor Marius Arnesen said. ‘If you spend 15 seconds on it, those are maybe 15 seconds that take the edge off the rant mode when people are commenting,’ Arnesen said.

Now, five months since launching the comment-quiz, here’s what NRKbeta has to say about the results.

“On average, there is a lot more attempts — both correct and wrong — than actual comments.” In a post on NRKbeta, Grut explained that it appears more people took the quiz for reading comprehension than for access to the comments section. Some may treat it as a “fun little game after reading,” Grut wrote. Of the 14 articles that included quizzes, the highest number of comments on any one was 77. Meanwhile, more than 300 people took the quiz for that same article.

[Note: See that super-long red bar above? “Due to a human error, the right answer to one of the questions was not indicated. It made it impossible to pass the quiz. Hence the many logged wrong attempts.”]

“We have not established any set rules for when the quiz should be enabled.” NRKbeta opted not to require the quiz for every article’s comment section. While the quiz was built as a WordPress plug-in with questions randomized for each user, the team did not want to overwhelm commenters on quick and light-hearted subjects.

Grut noted that by requiring potential commenters to become more invested in the content, the comment-quiz favored “the most eager with the most time on their hands.” But it didn’t always result in better comprehension over time: “From time to time this has lead to a decline in quality and tone, causing [the reader] to often abandon the quiz module.”

“On average, there is a staggering error rate of 72 percent on the quiz.” This may have been because international readers invaded after our story got picked up, and tried to take a quiz written in Norwegian. The clearest finding, Grut noted, was that readers liked treating the quiz as a game. There has been an uptick in “gaming the news” or developing online games for audiences to consume news or to even learn about fake news. Games like these quizzes could potentially help reporters and editors determine if the wording and explanations in their articles are clear to readers, in addition to cooling the comments section’s hotheads.

“We’re trying to establish a common ground for the debate,” Arnesen told Lichertman earlier this year. “If you’re going to debate something, it’s important to know what’s in the article and what’s not in the article. [Otherwise], people just rant.”

Photo of multiple choice test by Alberto G. used under a Creative Commons license.

POSTED     Aug. 11, 2017, 10:04 a.m.
SEE MORE ON Audience & Social
Show comments  
Show tags
Join the 45,000 who get the freshest future-of-journalism news in our daily email.
Former Gawker employees are crowdfunding to relaunch a that’s owned by a nonprofit and funded by readers
A Kickstarter campaign aims to raise $500,000 before January 8 to try to buy back and relaunch, which will be owned by a newly established nonprofit Gawker Foundation.
A growing turn to philanthropy
“In 2018, more nonprofit newsrooms will be launched than North Korean test missiles.”
Scooped by AI
“I’m not talking about computer-generated stories about earthquakes, earnings reports, or sports scores. These will be stories on your beat, written by humans who understand how to use machine learning to aid their reporting.”