When my former colleague Joseph Lichterman wrote about a Norwegian news organization that makes readers pass a quiz on the article before they can comment on it (one of the most-trafficked stories in Nieman Lab history, by the way) the site — NRKbeta, the tech vertical of Norway’s public broadcaster — was lauded for its creativity. But NRKbeta’s editors and journalists said it was too early to tell if the program was a success.

This project from @NRKbeta and @NRKno is quite possibly one of the best new journalism ideas I've seen, ever.https://t.co/DCT3s9pDty

— Amy Webb (@amywebb) March 1, 2017

But now, five months in? NRKbeta’s team says readers may have treated the quizzes on 14 articles more like reading comprehension games than as a gateway to the comments section.

As Lichterman described the feature, back in March:

On some stories, potential commenters are now required to answer three basic multiple-choice questions about the article before they’re allowed to post a comment. (For instance, in the digital surveillance story: ‘What does DGF stand for?’)

The goal is to ensure that the commenters have actually read the story before they discuss it.

‘We thought we should do our part to try and make sure that people are on the same page before they comment. If everyone can agree that this is what the article says, then they have a much better basis for commenting on it,’ said NRkbeta journalist Ståle Grut.

Forcing users to take a little extra time to think about the comment they’re about to post also helps them think about tone, NRKbeta editor Marius Arnesen said. ‘If you spend 15 seconds on it, those are maybe 15 seconds that take the edge off the rant mode when people are commenting,’ Arnesen said.

Now, five months since launching the comment-quiz, here’s what NRKbeta has to say about the results.

“On average, there is a lot more attempts — both correct and wrong — than actual comments.” In a post on NRKbeta, Grut explained that it appears more people took the quiz for reading comprehension than for access to the comments section. Some may treat it as a “fun little game after reading,” Grut wrote. Of the 14 articles that included quizzes, the highest number of comments on any one was 77. Meanwhile, more than 300 people took the quiz for that same article.

[Note: See that super-long red bar above? “Due to a human error, the right answer to one of the questions was not indicated. It made it impossible to pass the quiz. Hence the many logged wrong attempts.”]

“We have not established any set rules for when the quiz should be enabled.” NRKbeta opted not to require the quiz for every article’s comment section. While the quiz was built as a WordPress plug-in with questions randomized for each user, the team did not want to overwhelm commenters on quick and light-hearted subjects.

Grut noted that by requiring potential commenters to become more invested in the content, the comment-quiz favored “the most eager with the most time on their hands.” But it didn’t always result in better comprehension over time: “From time to time this has lead to a decline in quality and tone, causing [the reader] to often abandon the quiz module.”

“On average, there is a staggering error rate of 72 percent on the quiz.” This may have been because international readers invaded after our story got picked up, and tried to take a quiz written in Norwegian. The clearest finding, Grut noted, was that readers liked treating the quiz as a game. There has been an uptick in “gaming the news” or developing online games for audiences to consume news or to even learn about fake news. Games like these quizzes could potentially help reporters and editors determine if the wording and explanations in their articles are clear to readers, in addition to cooling the comments section’s hotheads.

“We’re trying to establish a common ground for the debate,” Arnesen told Lichertman earlier this year. “If you’re going to debate something, it’s important to know what’s in the article and what’s not in the article. [Otherwise], people just rant.”