At this moment of great division in America, with tempers flaring and tensions high, let us gather together as a people, as a nation, and discuss the one thing we can all agree on: The Atlanta Falcons are hot garbage.1

In 2017, with 6:04 left in the fourth quarter of Super Bowl LI, the Falcons had a 99.8 percent chance of victory. Did they achieve that victory? No, they did not, coughing up what had been a 28-3 lead to lose to the New England Patriots in overtime, 34-28.

Two weeks ago, with 2:52 remaining against the Dallas Cowboys, the Falcons had a 99.9 percent chance of victory. Did they achieve that victory? No, they did not, turning what had been a 26-7 lead into a 40-39 loss.

Just a week ago, with 6:20 left against the Chicago Bears, the Falcons had a 99.6 percent chance of victory. Did they achieve, &c., &c.? Reader, you know how this story ends: with a 26-10 lead turning into a 30-26 loss.2

That the Falcons have this remarkable propensity for complete and total collapse is rewarding for me, as a devoted fan of a rival squad. But they are also useful as a real-world example of the limits of statistical forecasting — a lesson that, if learned at the ballot box, can have far greater implications. Yes, a team with a 99.9% chance of winning can lose. And so can a presidential candidate with a 99% chance of winning — or a 92% chance, an 85% chance, or a 71% chance.

Who knows how many Patriots, Cowboys, or Bears fans, seeing their chances dwindle to near-nothingness, turned off their TV sets in disgust, only to miss an astonishing comeback? Or how many Falcons fans developed drinking problems watching the Fates do their work? Both could be a painful mistake, but less painful than, say, [waves hands madly] all of this.

To be clear: Joe Biden’s 2020 poll performance against Donald Trump is not a carbon copy of Hillary Clinton’s in 2016; Biden’s lead has been much more consistent, his negatives are much lower, and Trump is by now a (very) known quantity, not an untraditional nominee into whom voters’ thoughts and dreams can be wished. But after 2016 — when a Clinton victory considered somewhere between “very likely” and “iron-clad cinch” failed to materialize — any degree of “certainty” feels unearned.

If you believe Yphtach Lelkes, Solomon Messing, and Sean Westwood, though, the risks of this sort of election forecasting goes beyond simple disappointment. The act of probabilistic forecasting — the sort of poll-aggregating statistical model most associated with Nate Silver and FiveThirtyEight — can actually change an election’s outcome; we are democracy’s Schrödinger’s cat.

In an op-ed for USA Today, these three political scientists (of Penn, Georgetown, and Dartmouth, respectively) argue that creating and publishing this sort of election forecasting models is actively harmful.

The 2020 election will be unprecedented, but you can already hear echoes of 2016 — in fact FiveThirtyEight’s 2020 forecast debuted with odds of a Biden win almost identical to what Clinton had on election day. That year, roughly 100 million eligible voters stayed home, some of whom did so because of the widespread consensus that a Hillary Clinton victory was inevitable.

Our research suggests that certainty was likely driven by a perfect storm of media personalities over-simplifying and misinterpreting complex predictions from election forecasters who predicted 2008 and 2012 election results with striking accuracy.

Clinton herself suggested the forecasts kept people home in a catastrophe of overconfidence. “I don’t know how we’ll ever calculate how many people thought it was in the bag, because the percentages kept being thrown at people — ‘Oh, she has an 88 percent chance to win!’” Clinton said in a 2017 interview.

In a new paper published in the Journal of Politics, we show that election forecasts greatly increase the perception that the race will be a blowout for one candidate, and that some people do not vote as a result. We also show that Democrats are more likely to see coverage of forecasts than Trump supporters.

So what’s the problem? The core of their argument is that people perceive probabilities in different ways depending on the details of the presentation.

Let’s say you’re summarizing the average polling in a pretty close race. You could say:

— “Smith leads Jones 52% to 48%, with a margin of error of ±2%.”
— “Smith has an 84% chance of victory.”
— “Smith is the 5-to-1 favorite.”

These three sentences all mean essentially the same thing! But the first emphasizes the closeness of those two numbers, 52 and 48, and notes that the level of precision in the estimate is deflated by potential errors in the process, either because of bad pollster behavior or just dumb chance.

Framing it as an 84% chance of victory, meanwhile, leans in the direction of certainty — and a non-zero share of people who see that will confuse “Smith has an 84% chance of victory” with “Wow, Smith is ahead by 84%!” or “Smith leads Jones 84% to 16%.”

And using the language of betting odds encourages people to contextualize the race in a field — sports — where uncertainty is inherent. Every sports fan is constantly reminded that upsets do happen, favorites do lose, and being a 5-to-1 favorite is no guarantee of victory.

Completely irresponsible misrepresentation of models discussed in the piece, which give Trump, respectively, a 21% chance of winning, 17.8%, and 13% chance of winning.

Far f*cking cry from "almost no chance".

And not so different from 2016 models giving him 28.4% and 15% chance https://t.co/tMf2tkMZBl

— Rasmus Kleis Nielsen (@rasmus_kleis) October 1, 2020

I find a lot of the Lelkes/Messing/Westwood argument compelling, which is why I’ve mentioned earlier versions of their research a couple of times over the years.

And it appears that at least some newsrooms are sympathetic to the underlying argument. FiveThirtyEight has taken steps to make the uncertainty inherent in its predictions more clear, both in design and in language. Checking out its forecast page today, you’ll find potential Trump Monte-Carlo-simulation-style wins highlighted prominently. Rather than lead with percentages, the top headline is the more modest “Biden is favored to win the election.” A little cartoon canine named Fivey Fox reminds: “Don’t count the underdog out! Upset wins are surprising but not impossible.”

The Economist’s new model likes Biden’s chances a lot — it gives him a 99% chance of winning the popular vote, and a 90% chance of winning the Electoral College — but it still pulls its punch a bit: “Right now, our model thinks Joe Biden is very likely to beat Donald Trump in the electoral college.” That emphasizes the now-ness of the call — things can change! — and it attributes it to what “our model thinks,” not stone tablets from on high.

And several other news sites that made forecasts in the last cycle appear to be sitting this one out entirely.

But the man most associated with the art/science of election forecasting is not a fan of the Lelkes/Messing/Westwood argument.

Close call sometimes when there's a stupid article/study/etc. making the rounds that you could persuasively debunk with some effort but you also risk Streisand Effecting it by giving it more attention.

— Nate Silver (@NateSilver538) October 1, 2020

Nate Silver has also raised objections to earlier iterations of this research, which Messing found unconvincing. (You can find some of the back and forth here, here, here, here, here, and here. I found that first link particularly helpful.)

You apparently had a hard time when you tried to debunk it before. Your raised a few objections & called on reviewers not to publish it in your podcast & now it's in a flagship journal. https://t.co/I3hFWVUzb2 https://t.co/iRUMWFvBcq

— Solomon Messing (@SolomonMg) October 2, 2020

I think Lelkes et al. are correct on their core arguments: that readers/voters are not very good at interpreting statistical data; that changing to a new way of presenting data (“52/48” to “84%” in the example above) is particularly likely to cause misinterpretations; and that believing a race will be a blowout can discourage people from voting.

But I also have some broad sympathy with Silver’s side of the argument and think that forecasting models get a bit of a bad rap. I’ll lump my reasons into two groups: one political, one journalistic.

Let’s start with the politics. The evidence for a direct causal link between probability forecasting and reduced turnout — especially its most extreme form sometimes spotted on Twitter, “It’s Nate’s fault we got stuck with Trump” — strikes me as pretty weak. (The headline on their USA Today op-ed doesn’t quite go that far — “Election forecasts helped elect Trump in 2016” — but it still goes farther than I think their evidence does.)

The Y/M/W paper includes a couple of interesting experiments that show people say they’re less likely to vote in an election after being given information that makes them think it’ll be a blowout — a situation where their vote won’t “matter.” It also shows that this willingness to sit out a race increases as the “cost” of voting goes up. (Poll taxes are illegal in America, but not in academic research!)

But a controlled online experiment and a real-world election are really, really different things. Especially a U.S. presidential election, which is the kind of race that’s gotten the overwhelming share of attention from the FiveThirtyEights of the world.

First: Presidential elections are never the only race on a person’s ballot. Every person voting for president this November will also have their local member of the House on the ballot; two-thirds of states will also have a U.S. Senate race. Governors, state legislators, mayors, ballot initiatives — there are always lots of other things on the ballot for a committed partisan to vote on.

Sure, the presidential race will typically be the one that people will be most interested in — especially this year — but it’s by no means a pure on/off switch for turnout.

And it’s hard for researchers to translate all the behavioral incentives of a national election into a Mechanical Turk job. To take one example, the researchers told the subjects of their experiments that they were dealing with a race between “candidate A,” who “supports the majority of the policies you support and is well qualified for the job,” and “candidate B,” who “does not share your views and is less qualified than candidate A.” There’s a beautiful, clinical austerity to those descriptions — but we know that partisan labels, campaign targeting, and specific issues can do a lot more than that to draw sharp distinctions and rev up voter behavior. (One suspects Biden supporters have a stronger take on Trump than “he does not share my views and is less qualified” — and vice versa.)

Second: Thanks to the Electoral College, the overwhelming majority of presidential votes don’t “matter” anyway.

Imagine this scenario: Voter X likes Hillary Clinton and wants her to be elected. Voter X clicks on HuffPost the day before the election and sees that Clinton has a “98% chance” of winning. Voter X decides their vote won’t matter and stays home on Tuesday.

The problem here is presidential elections aren’t (sadly) decided by national popular vote. They’re winner-take-all by state.3 So if Voter X lives in New York or California — or Texas, or West Virginia, or any of the other American states in which the outcome of the 2016 election was never in serious doubt — their vote never “mattered,” in the sense that there was even a modicum of a chance it might influence the outcome.

There is good evidence that being in a battleground state — one that both parties think they can win — increases your likelihood of voting. In recent cycles, battleground states have had turnout levels around 10% higher than the nation as a whole. But it’s not clear how much of that difference is tied to voters’ belief that their choice “matters” versus the result of campaigns paying a lot of attention to their state — holding big rallies, blanketing TV with ads, funding major get-out-the-vote operations, and so on.

Even if you assume a large turnout effect from someone’s vote “mattering,” the Electoral College eliminated that impact for most Americans long before statisticians had the chance to.

Third: There doesn’t seem to be any clear evidence that the FiveThirtyEight school of election forecasting was somehow uniquely powerful in driving the perception that Hillary Clinton was going to win. If Nate Silver had never been born (or just stuck to baseball), would a substantially smaller number of Americans really have thought Trump had a chance?

Forecasts are fundamentally aggregations of individual polls — but getting rid of the forecasts doesn’t get rid of the polls. There were 121 national two-way polls conducted and released between October 1, 2016 and Election Day. Only 7 of them — all from the same pollster, which used an unusual panel methodology — showed Hillary Clinton behind. (And remember, she won the popular vote by almost 3 million votes!) Is there evidence that daily headlines like “New poll: Clinton leads Trump again” weren’t at least as at fault in any false confidence that developed?

Or how about plain old conventional wisdom? Former FBI director Jim Comey famously admitted that one reason he announced a renewed investigation into Clinton’s emails — just days before the election — was that he assumed she was about to win. “I was operating in a world where Hillary Clinton was going to beat Donald Trump,” he said later.

Did Comey think that because he was refreshing the Princeton Election Consortium’s website every 15 minutes? No, he thought that because roughly everyone in his professional circles thought that, thanks to a mixture of consistent polling leads and a blinkered assumption that someone with Donald Trump’s record of behavior couldn’t possibly be elected president. That assumption was, we have since been brutally informed, wrong — but I haven’t seen any way to sift out a distinct role for probabilistic forecasting in that mix.

It should probably also be noted that 2016 was not, in fact, a low-turnout presidential election: 55.5% of the voting age population voted, up slightly from 2012 (54.9%). Of elections since 1972 (when 18-year-olds were first eligible), turnout in 2016 ranked No. 3 out of 12.

It is slightly dirty pool, in this reporter’s opinion, for Lelkes et al. to rue the fact that “100 million eligible voters” stayed home in 2016 as if that cycle’s voters were somehow uniquely turned off, or that there was evidence the alleged certainty of the election was an outsized factor in that.

Just before the 2016 election, 57% of those polled said they thought Hillary Clinton would beat Donald Trump. (Another poll put it at 58%.) That’s hardly some unprecedented level of confidence; it’s in the same ballpark of how voters saw the likely winners of other recent elections. Just before the 2012 election, 54% said they thought Barack Obama would beat Mitt Romney. In 2008, 71% said they thought Obama would beat John McCain. In 2004, 56% thought George W. Bush would beat John Kerry.

(Somehow, even though voters were significantly more confident about Obama winning in 2008 than they were about Clinton winning in 2016, Obama didn’t appear to suffer any of the “certainty” penalty being claimed for Clinton. Perhaps, just perhaps, that had more to do with Clinton being a historically unpopular candidate than with a website’s method of aggregating polls? In 2008, 35% of Americans had an unfavorable view of Obama; in 2016, 52% of Americans had an unfavorable view of Clinton.)

Fourth: There doesn’t seem to be any accounting for voters adjusting to a new form of information. One reason people might find “52/48” less confusing than “84%” is that “52/48” has been the way polling data has been presented for a century. It can take a while for people to get used to something new.

I would expect two distinct kinds of learning by voters. One is that people will simply become more familiar with what these forecasts mean as they get more used to them. (I would wager that, outside of politically active, high-information voters, only a tiny share of Americans had paid much attention to them before 2016.)

The other is that people will learn the hard lesson that these forecasts, like polls, are sometimes wrong! A nation of Clinton supporters learned about their fallibility in a very direct way four years ago. So did millions of Trump supporters who believed their guy didn’t have a chance. Whatever impact these forecasts might have had on turnout in 2016, I’d bet $100 it will be substantially less in 2020 — simply because people now have a more realistic view of them. Fool me once, shame on me, and so on.

I’m not really disputing the findings of Lelkes, Messing, and Westwood here with any of these; their experimental findings are real. I’m just not sure that they can be extrapolated to actual voter choices in the wild in any significant way — especially to the degree of claiming they were a meaningful factor in Trump’s 2016 win.

My other pushback, though, is more journalistic.

I consider how difficult the United States makes it for its citizens to vote one of our greatest crimes. A mixture of history, racial animus, and conservation of power lead to tight registration limits, polling-place closures, voter-roll purges, felony disenfranchisement, and seven-hour lines on Election Day. Journalism, as a profession whose work relies on a free and open democracy, should not be afraid to push for increased voter rights at every step of our elections.

But: It is not our job to withhold information from readers in order to maximize turnout.

Could an investigative story uncovering corruption by a local mayor impact voter turnout? Sure. In some scenarios — imagine the mayor’s party is so dominant locally that he’s likely to be reelected anyway — it could reduce voters’ interest in trodding to the polls to vote in a flawed man. In others — imagine the race is close and voters are motivated by the story to boot him from office — it could boost turnout. (There’s evidence of both sorts of impact.)

But should a newspaper, when deciding whether to publish that story, think about holding back because it might impact voter turnout? No — that’s not our job. Journalism has an effect on the world — it’s rather pointless if it doesn’t — and the goal of a more informed public will frequently come into conflict with more people taking time out to vote on a November Tuesday.

What about reporting on polling? Should a news organization ignore a poll that shows one candidate with a huge lead for fear it might reduce turnout? CNN’s latest poll in the presidential race shows Biden up over Trump by a whopping 16 points, 57% to 41%. Is there a chance seeing that huge margin convinces a Biden voter her vote isn’t needed, or discourages a Trump voter from bothering to vote for what looks like a losing cause?

Sure! People are strange — they react to information in different ways. But I would hope no one thinks that CNN should have sat on the poll results because it might make people think Biden has a substantial lead. He does! (Based on FiveThirtyEight’s polling database, the most recent national polls show Biden with leads of 9, 8, 12, 10, 11, 11, 12, 10, 6, 12, 9, 7, 11, 12, 11, 8, 7, 9, 10, 15, 16, 14, 7, 9, and 10 points.)

Critiques of horse-race reporting are ageless and often correct. But if we know that one candidate is much more likely to win than another, it’s still our duty to share that information with our readers — not to pretend things are closer than they are in order to keep people interested. That information should be as accurate and as contextualized as possible, of course; there can be good election forecasts and bad ones, just as there are good pollsters and bad ones, good journalists and bad ones. But I’m very hesitant to say that we should hold back important and useful information just because we don’t think the masses can “handle” it in the way we prefer.

Concern about people misinterpreting political information is structurally connected with the flattened news universe the Internet hath wrought. In 1988, say, if there was a fancy statistical model that predicted George Bush would beat Michael Dukakis, those results might have been shared only with readers of a paid newsletter like The Evans-Novak Political Report, or among the fax machines of a few political consultants. Complex political information could be restrained to a small circle of people prepared for the complexity.

Online, access to news and political information is more a function of interest than having a political job or subscribing to a high-priced newsletter. The people who check in on Biden’s chances repeatedly — they’re up from 82% to 83% today! — are people who have a high level of interest in politics generally and this race in particular. But of course that information lives at a URL anyone can access, which means it can leak out beyond those communities of interest with a simple click.

Which is why I want to close with the one bit of the Lelkes et al. op-ed that, frankly, bugged the hell out of me:

If 2016 taught us anything, it’s that it’s best to view forecasts and polls as the ultimate realization of what political scientist Eitan Hersh calls political hobbyism — politics as sport — but not as incontrovertible scientific truths from the oracles of statistics.

I enjoyed some of Hersh’s book Politics Is for Power: How to Move Beyond Political Hobbyism, Take Action, and Make Real Change, which came out earlier this year. Here’s the book description:

A brilliant condemnation of political hobbyism — treating politics like entertainment — and a call to arms for well-meaning, well-informed citizens who consume political news, but do not take political action.

Who is to blame for our broken politics? The uncomfortable answer to this question starts with ordinary citizens with good intentions. We vote (sometimes) and occasionally sign a petition or attend a rally. But we mainly “engage” by consuming politics as if it’s a sport or a hobby. We soak in daily political gossip and eat up statistics about who’s up and who’s down. We tweet and post and share. We crave outrage. The hours we spend on politics are used mainly as pastime.

Instead, we should be spending the same number of hours building political organizations, implementing a long-term vision for our city or town, and getting to know our neighbors, whose votes will be needed for solving hard problems. We could be accumulating power so that when there are opportunities to make a difference — to lobby, to advocate, to mobilize — we will be ready. But most of us who are spending time on politics today are focused inward, choosing roles and activities designed for our short-term pleasure. We are repelled by the slow-and-steady activities that characterize service to the common good.

Hersh is particularly focused on “condemning” this behavior among educated, middle- and upper-middle-class liberals — which is to say, the notional Hillary Clinton voter who was allegedly so interested in her victory they obsessively checked their favorite poll aggregators but didn’t bother to vote.

(Never mind that those college-educated voters went for Clinton in record numbers, and that her relative losses came among less-educated, lower-information, less-engaged voters — a group I suspect is less interested in statistical modeling of polling data.)

You will never find me suggesting that “building political organizations, implementing a long-term vision for our city or town, and getting to know our neighbors” are bad things. They’re good things! But where I think Hersh (and Lelkes/Messing/Westwood) are very wrong is to see the pursuit of information about politics and elections as mere “entertainment” or “sport” or “hobby” — and to imagine that, if only people could just stop tweeting mean things about Trump and refreshing The Upshot’s poll tracker, they’d instead spend all that time running for state senate or unionizing their workplace.

As the top of this story probably made clear, I root for the New Orleans Saints. I consume a lot of information about them! I read stories, I listen to podcasts, I watch pregame shows, I livetweet games in ways annoying to my followers. But other than my level of happiness on Sundays, not much in my life is going to be affected if the Saints win or lose in any given week. If the Falcons win the Super Bowl someday, I will likely have a very cranky evening — but I’ll wake up the next day in the same country in the same situation.

That’s just not true with a presidential election. The stakes are real! And I live a comparatively privileged life; the results of this election are much more significant for Americans who make minimum wage, who struggle to pay for healthcare, who want to have a legal abortion, who are at high risk of viral infection, or who, you know, just aren’t really into creeping authoritarianism.

It is frankly insulting to think that Americans’ increased hunger for political news and information — the thing that fueled news organizations’ “Trump bumps” — is the same as viewing it as “entertainment” or “sport.”

Look, I live in suburban Boston and work at Harvard — I know a lot of educated, middle- and upper-middle-class liberals. And I don’t think I know a single one who spends their evenings doomscrolling Twitter because they find it entertaining. Consuming political news makes people feel terrible, overwhelmed, stressed. If educated liberals are spending a lot of energy trying to figure out whether Biden is up or down in the polls, it’s not because that’s how they get their kicks — it’s because they find one potential outcome of this election terrifying.

Hersh makes some good points in his book about the ways in which Politics Twitter and its ilk heighten the idea of politics as self-expression, and he also notes correctly that the passionate consumption of news by a core group of politics junkies can create some messed-up incentives for politicians and other political actors; people who are more engaged in politics also tend to have stronger political opinions, which is a factor in increased polarization. But those are classic impacts of the simultaneously flattened and fractured universe of online news — and of digital content more broadly. (And their biggest negative impact has been to reduce the amount of political engagement among low-interest, low-ideology voters, not to increase the engagement of “hobbyists.”)

I mention the Saints again because they connect with a core metaphor of “political hobbyism”; as Hersh puts it: “Political hobbyism is to public affairs what watching SportsCenter is to playing football.” It’s a good metaphor! But the three hours I put in watching the Saints on Sundays are not three hours I would otherwise be spending doing drills with tackling dummies. They’re not substitutable goods. And the same is true in politics. The Trump era has seen a substantial increase in the amount of political news people consume. But it’s also seen a big jump in other forms of political engagement.

I don’t think “political hobbyism” explains why the share of Americans who reported giving to a political candidate doubled between 1996 and 2016 and will likely be up again in 2020. Or why small-dollar donor bases have grown substantially on the left and right. I don’t think it explains why more than 15 million Americans participated in a Black Lives Matter protest this year, driving up voter registration numbers. Or why more than 4 million women marched the day after Trump’s inauguration. I don’t think it explains why college-educated liberal workers are unionizing their workplaces in big numbers. And while the coronavirus pandemic has obviously put limits on how much physical organizing work can be done, interest in political volunteering has shot up in many quarters since the shutdowns began.

Do you really think all those people doing what Hersh and others consider the good work of politics don’t also check FiveThirtyEight, or read stories about the latest Trump outrage, or tweet about Stephen Miller — the sort of thing they scold as “cheap” “fandom” or “hobbyism”? Political information and political action are not in opposition; they are two parts of the same thing.

By all means, news outlets should work as hard as they can to make sure the information they give their audiences about politics and elections is factually accurate, responsibly framed, and contextualized for clarity. But they also have a job to do — and giving good information to citizens about things they care about is no crime.

1. There is, by the way, a 71 percent chance that a mention of the NFL in a piece I’m writing will include an attack on the Falcons. Who dat. ↩︎
2. Monday night, they lost again, but at least had the courtesy to be consistently bad from the opening kick. ↩︎
3. Unless you live in Omaha or Presque Isle. ↩︎