Facebook’s fact-checking program relies on third-party fact-checking and news organizations to help weed out fake and misleading news on the platform. That limits the program from scaling up, since there are only so many organizations that do this work.

But what if Facebook could use regular people — people whose day job is not fact-checking — to fact-check articles instead? A new working paper from researchers at MIT suggests that it would be surprisingly effective, even if those people are only reading headlines. “Crowdsourcing is a promising approach for helping to identify misinformation at scale,” the paper’s authors write.

The team used a set of 207 news articles that had been flagged for fact-checking by Facebook’s internal algorithm. (The study was done in collaboration with Facebook’s Community Review team and funding from the Hewlett Foundation.) Two groups were then asked to check the accuracy of the articles: Three professional fact-checkers, who researched the entire articles to make their verdicts, and 1,128 Americans found on Mechanical Turk who determined accuracy based online on the articles’ headlines and lede sentences. The result:

We find that the average rating of a politically balanced crowd of 10 laypeople is as correlated with the average fact-checker rating as the fact-checkers’ ratings are correlated with each other. Furthermore, the layperson ratings can predict whether the majority of fact-checkers rated a headline as “true” with high accuracy, particularly for headlines where all three fact-checkers agree. We also find that layperson cognitive reflection, political knowledge, and Democratic Party preference are positively related to agreement with fact-checker ratings; and that informing laypeople of each headline’s publisher leads to a small increase in agreement with fact-checkers.

This is significant partly because Facebook’s official fact-checking partners really don’t get through that much content: In January, The Hill reported that “Facebook’s six [U.S.] partners have 26 full-time staff and fact-checked roughly 200 pieces of content per month.” Not only is this a drop in the bucket, but “it also has the potential to increase belief in, and sharing of, misinformation that fails to get checked via the ‘implied truth effect,'” the researchers write. “People may infer that lack of warning implies that a claim has been verified. Furthermore, even when fact-check warnings are successfully applied to misinformation, their impact may be reduced by lack of trust.” (In 2019, Pew found that 70% of Republicans, and 48% of Americans overall, think fact-checkers are biased.)

This is so important, and Facebook has been told versions of this (by @brooklynmarie among others). Professional fact-checkers get overwhelmed by the sheer volume of crap on FB. https://t.co/a85ov6E4jw

— Charles Arthur (@charlesarthur) October 9, 2020

In this study, the professional fact-checkers usually — but not always — agreed with each other. “At least two out of three fact-checkers’
categorical ratings agreed for over 90% of the articles,” the authors write. Still, “this level of variation in the fact-checker ratings has important implications for fact-checking programs, such as emphasizing the importance of not relying on ratings from just a single fact-checker for certifying veracity, and highlighting that ‘truth’ is often not a simple black and white classification problem.”

Next we had 3 professional factcheckers research each article & rate its accuracy

First surprise: They disagreed more than you might expect!

The avg correlation b/t the fact-checkers' ratings was .62

On half the articles, 1 FC disagreed w other 2; on other half, all 3 agreed pic.twitter.com/QPtKCrE7fO

— David G. Rand (@DG_Rand) October 8, 2020

With that in mind, they turned to the MTurkers.

We created politically-balanced crowds & correlated their avg ratings with avg factchecker ratings

The crowd does quite well:

With as few as 10 laypeople, crowd is as correlated with average fact-checker rating as the fact-checkers’ ratings are correlated with each other!! pic.twitter.com/pr6T3aD9Uq

— David G. Rand (@DG_Rand) October 8, 2020

“A relatively small number of laypeople can produce an aggregate judgment, given only the headline and lede of an article, that approximates the individual judgments of professional fact-checkers,” the authors write. It doesn’t mean that professionals are no longer necessary: “We see crowdsourcing as just one component of a misinformation detection system that incorporates machine learning, layperson ratings, and expert
judgments.”

Caveats:
1) Individuals still fell for misinfo- but *crowds* did well
2) Need to protect against coordinated attacks (eg randomly poll users, not reddit-like)
3) Not representative sample- but point is that some laypeople can do well (FB could hire turkers!)
4) This was pre-COVID pic.twitter.com/V1iR5u2A1r

— David G. Rand (@DG_Rand) October 8, 2020