Twitter  Thought Catalog acknowledges that giving contributors the ability to publish and tweet offensive content is risky  
Nieman Journalism Lab
Pushing to the future of journalism — A project of the Nieman Foundation at Harvard

Alberto Cairo: Data journalism needs to up its own standards

The data visualization expert argues that FiveThirtyEight and Vox have overpromised and underdelivered — and that they need to treat their data with more scientific rigor.

Did you know that wearing a helmet when riding a bike may be bad for you? Or that it’s possible to infer the rise of kidnappings in Nigeria from news reports? Or that we can predict the year when a majority of Americans will reject the death penalty (hint: 2044)? Or that it’s possible to see that healthcare prices in the U.S. are “insane” in 15 simple charts? Or that the 2015 El Niño event may increase the percentage of Americans who accept climate change as a reality?

If you answered “yes” to at least 60 percent of the questions above, I’m 95 percent certain that you’ve been following the recent buzz around “data” and “explanatory” journalism. I’m talking about websites like Nate Silver’s FiveThirtyEight and Ezra Klein’s A few traditional news organizations have followed suit, either creating their own in-house operations (The New York Times’ The Upshot) or strengthening existing efforts.

There is a lot to praise in what all those ventures — and others that will appear in the future — are trying to achieve. Journalists are known for being allergic to math and to the scientific method; some even proudly boast about it. Many in our profession still stick to flawed practices, such as asking the same questions to two or more sources and then just reporting their answers, without weighing the evidence and then pointing out which opinion is better grounded.

At first, the current popularity of the new wave of data journalism seemed to be a good antidote to the epidemic of hardball punditry and tomfriedmanism that has plagued the news for ages. When  Silver published FiveThirtyEight’s foundational manifesto, “What the Fox Knows,” I applauded him with enthusiasm. After all, bad data is pervasive in traditional newsrooms. If you think I’m exaggerating, read this recent and infuriating Washington Post op-ed, which gets causality wrong, is oblivious of ecological fallacies, misinterprets sources, and ends with a coarse, insulting, and condescending line. Don’t blame just the authors. Blame the editors at the Post, too.

But I have to confess my disappointment with the new wave of data journalism — at least for now. All the questions in the first paragraph are malarkey. Those stories may not be representative of everything that FiveThirtyEight, Vox, or The Upshot are publishing — I haven’t gathered a proper sample — but they suggest that, when you pay close attention at what they do, it’s possible to notice worrying cracks that may undermine their own core principles.

So what’s wrong with the stories in the first paragraph?

First, the piece on bike helmets is an example of cherry-picking and carelessly connecting studies to support an idea. It’s possible to “prove” almost anything if you act like this. I can “prove” that vaccines cause autism — they don’t — just by selecting certain papers, particularly those based on tiny samples or simple correlations, while ignoring the crushing majority that refutes my intuitions. Gladwellism— deriving grand theories from a handful of undersubstantiated studies — may be popular nowadays, but it’s still dubious journalism.

Second, using news reports on kidnappings in Nigeria as a proxy variable of actual kidnappings is risky. Proxy variables need to be handled with care. You cannot assert that there are more kidnappings just because the media is running more stories about them. It might be that you’re seeing more stories simply because news publications are increasingly interested in this beat. You won’t know until you do some proper analysis.

Third, long-term linear predictions of nonlinear phenomena are nearly always wrong. Writing that a majority of Americans will reject the death penalty by 2044 “if the trend continues” yields a catchy, SEO-friendly headline, but it means very little. The article says that there’s no “reason to expect the trend to pick up speed anytime soon,” but the same could have been said about support for gay rights a decade ago, and see where we are today. Unknown unknowns, confounding variables, and black swans can kill any simplistic prediction. As xkcd explained, you cannot predict that a woman will have four dozen husbands next month just because she was unmarried yesterday (zero husbands) and married today (one husband.) That’s not just bad Math. It’s lack of common sense.


Fourth, will the 2015 El Niño event make more Americans accept that climate change is real? No idea. It could be. Or not. It’s impossible to know, as that piece by The Upshot reads like wishful thinking. It does say that “belief in warming jumps when global temperatures hit record highs; it drops in cooler years,” but the evidence to support that claim is not fully revealed, so we don’t know if those “jumps” are relevant, significant, or just pure noise. Why should I trust the writer? I’m a journalist myself. I don’t trust journalists.

Fifth, can we really assert that health care prices in the U.S. are “insane” based on 15 simple charts? I won’t bore you this time, as there’s a lot of fishy details in those graphics. If you’re interested in the nitty gritty, read this blog post. I bet you’ll be as shocked as I was.

Is data journalism in crisis (already)?

The main challenge that FiveThirtyEight and Vox (and, to a lesser extent, The Upshot) face is that they overpromised before they were launched but underdelivered after they went public. They promised journalism based on a rigorous pondering of facts and data, but they have offered some stories based on flimsy evidence — with a consequent pushback from readers. As I’m one of those readers, I will take the liberty to offer some suggestions:

1. Data and explanatory journalism cannot be done on the cheap.

It’s hard to produce a constant stream of good data journalism on the cheap and with a small team. If history can be considered as a guidance, we should remember that this is not how data journalism was done in the past.

Journalism that takes advantage of quantitative methods and visualization is not new. As the Columbia Journalism Review wrote a while ago, “[Nate] Silver’s work is arguably less revolution than evolution, one facet of a journalistic practice that has actually been around for decades.”

True. In 1973, Philip Meyer, now a retired professor at the University of North Carolina, coined the term “precision journalism” in a book with that same title. Meyer is the most popular (but not the only) advocate for a methodical application of social science to the practice of journalism, and one of the founding parents of computer-assisted reporting.

To give you an example of the power of this kind of journalism: In 1993, The Miami Herald won a Pulitzer for an investigation about why Hurricane Andrew caused such a sweeping destruction in certain neighborhoods of Miami and Homestead, while leaving others almost intact. It was related to lax zoning inspection and building quality standards. The investigation was based on databases and mapping, but also on careful on-the-ground reporting.

Data-savvy investigative reporters and visual designers at established news organizations haven’t historically worked in complete isolation. They can rely on relatively large infrastructures to provide funding, support, and legal aid, when needed. It remains to be seen if a publication that conducts just this type of journalism can survive on its own. This leads to my next point.

2. Data and explanatory journalism cannot be produced in a rush.

Both Vox and FiveThirtyEight are publishing new stories, blog posts, and explanatory pieces nonstop. As any news organization nowadays, they live in a 24/7 world. They need to feed the goat, as their business models seem to be based on attracting adequately large audiences.

There’s a big risk in that. It’s much easier to feed the goat with stories with titles like “The New York Times Editorial Shakeup As Explained By ‘Game Of Thrones’” than with thorough analyses of population trends in Ukraine. (I’ll admit I found both quite enticing, though.) It is tempting for a news startup to try to be both BuzzFeed and The Economist at the same time, no matter how chimerical that goal is. Lighthearted blahblah can be done quickly and nonchalantly. Proper analytical journalism can’t. If you have a small organization, you may have to choose between producing a lot of bad stuff or publishing just a small amount of excellent stories.

3. Part of your audience knows more than you do.

Here’s the description for a perfect storm: Data journalism is often based on publicly available databases. Besides, current standards of journalism transparency dictate that, after a journalist writes or visualizes stories based on data, she should disclose her sources and make her spreadsheets downloadable for anyone to check. Finally, due to the very nature of those stories, a good portion of the audience will likely be quite numerate. This helps explain some recent takedowns in social media. Plenty of readers out there know much more than we do about our own data, and nowadays they have the means to broadcast their outrage. They don’t hesitate to do it.

4. Data journalists cannot survive in a cocoon.

Silver likes to quote Archilochus’ phrase about foxes and hedgehogs: “The fox knows many things, but the hedgehog knows one big thing.” Hedgehog intellectuals see the world through the lens of expertise on a single area, or of a single grand idea; fox thinkers borrow tools from many fields. Journalists tend to be foxes: We have a basic understanding of different disciplines, but we don’t necessarily specialize in any of them.

This is good on one hand, as it allows for richer and more lively reporting. But it also poses problems for data journalists, as you can’t really extract meaning from data using only cookie-cutter templates. No matter how great you are at analyzing stuff with the R statistics language, you’ll be in trouble if you don’t have a deep understanding of where the data came from, of how they were gathered, filtered, and processed, of their strengths and shortcomings.

That’s the reason why, in many universities, science departments teach their own statistics courses: It’s not the same to use stats for sociological observations as for genetics, psychology, astronomy, or physics. The equations may be similar, but the outcomes of your analyses don’t depend only on those equations. Context matters.

Foxes need to partner up with hedgehogs, journalists with specialists. And it’s not just a matter of asking a couple of researchers some questions while you write a blog post — something that, by the way, neither Vox or FiveThirtyEight do often enough (therefore the lack of quoted experts in many of their stories.) It’s also a matter of doing your reporting in collaboration with those researchers, as they’re the ones that know the data really well. This isn’t a particularly groundbreaking notion: The publications who regularly conduct solid data and investigative journalism nowadays, like ProPublica, work this way on a regular basis. Their journalists are very conscious of the limits of their knowledge.

(Side note: It is dubious that Nate Silver himself is a pure fox. His biggest successes when he was still at The New York Times depended on his familiarity with sports and election data. When it comes to those topics, he’s a werehedgehog, so to speak.)

Most novel technological concepts or tools follow a Gartner Hype Curve: First, the novelty is released and enjoys a peak of inflated expectations. Then it suffers through a valley of disillusionment. After that, when the hype vanishes, it reaches maturity and is widely adopted.

Even if data journalism is by no means a new phenomenon, it has entered the mainstream quite recently, breezed over the peak of inflated expectations, and precipitously sank into a valley of gloom. Hopefully, it’ll soon enter the last phase, one of stability and productivity. There’s a need for a journalism which is more rigorous and scientific. Data skills shouldn’t be the turf of a small guild of savants — they should permeate journalism in general. Data and explanatory news organizations can help achieve those goals.

Therefore, pundits and math-challenged journalists (like the folks at The New Republic, who seem to know as much about stats as I do about quantum mechanics, but feel entitled to write about it anyway) shouldn’t feel triumphant after reading this column. First, because Silver’s manifesto is still a great read. The news data and explanatory journalism organizations just need to be up to their own standards to thrive — something that I honestly hope will happen soon.

Second, because it’s a good thing to be reminded of Fred Mosteller’s famous assertion: “It is easy to lie with statistics, but easier to lie without them.” Or Richard Feynman’s: “The first principle is that you must not fool yourself, and you are the easiest person to fool.” Indeed. Particularly when you don’t know how to weigh qualitative and quantitative evidence.

Photo of painting by Abraham Palatnik by See-ming Lee used under a Creative Commons license.

What to read next
BuzzFeed HQ
Caroline O'Donovan    Aug. 11, 2014
With a new round of investment from Andreessen Horowitz, the viral giant aims to get even bigger. One step down that path: making it more clear to readers what kinds of content BuzzFeed really offers.
  • RS

    In fairness, the Vox article doesn’t say that “wearing a bike helmet may be bad for you.” The opposite, in fact. Joseph Stromberg argues that mandatory bike helmet laws reduce the number of cyclists and, so, increase the risk and total number of crashes. It’s a distinction between an individual decision and a public health question.

    I know the debate around the Maryland law that Stromberg talks about, and the major cycling organizations in the area – all of which advocate for helmet use – opposed the law because of its net effects on safety.

    The science around bike helmets is deeply flawed, to the point that the government’s had to remove key studies under the Federal Data Quality Act. See:

  • Barry Hollander

    Of course it’s tough to take them to task when you admit, early on, “I haven’t gathered a proper sample” and then select a few instances (prunus avium picking?) to make a point. Still, I agree the sites over-promised, or perhaps we expected too much of them, and all the data crunching in the world is wasted if you use the wrong measures in the wrong way — and if you don’t include feet-on-the-ground reporting to turn data into a story.

  • JasonL

    While you’re at it Alberto, you should have a rule to never use findings derived by a corporation of it’s own data, i.e., when XYZ company says a photo is 6 times more likely to be clicked on than a headline. Perhaps an attribution to the company saying that about itself is okay, but a journalist should never use that figure as stand alone benchmark data for a story. It’s marketing. And that kind of stuff is all over the Internet posing as “research.”

  • Alberto Cairo

    Thanks for the comments, everyone. Good points.

    Barry, I didn’t gather a proper sample, indeed. It’d be hard for me to do so. I just wrote about some of the pieces that I had time to think about (I’ve seen some more recently.) I wouldn’t dare discussing other kinds of stories, like the ones based on predictive models for the World Cup, as they are way beyond my reach. Others have done it already:

    I guess that the overall point I’m trying to make is a positive one, as much of the criticism we’re seeing out there — like what’s coming from The New Republic— is nasty and unfair. I believe that what these organizations are trying to achieve is very important for the future of journalism. I do expect a lot, but that’s because I really want them to thrive.

    By the way, something that I didn’t mention in the article is that FiveThirtyEight has been quick at correcting mistakes. That speaks highly of them. I haven’t seen corrections at Vox, though. I may have missed them.

  • Alberto Cairo

    Perhaps but, to begin with, it does say: “There are two reasons to think that a helmet-wearing cyclist is inherently more accident-prone one.” Then, as “reasons”, the article uses a 2007 study with a sample size of one person and suggestions by “many people” that the author himself calls “unproven”. I would hesitate to say that those can be considered good reasons. In fact, I think that it’s irresponsible to write such a conjecture, based on the evidence shown.

  • Victoria Kamerzell

    This article makes me want to stop following Nieman Journalism Lab. Now, I need to go find some self-righteous “data” to back up every decision I ever made.

  • editorsteve

    Good article. I would, however, consider 538 and vox, despite their being somewhat uneven, as delivering on their promises. After all, the NYT is uneven as well! And Nate Silver ran the table in the last presidential election — calling the results in all 50 states. Even the one state he decided was “too close to call” took a week to resolve after the voting was over. Viewing the statistical journalism scene (what we used to call “analytic journalism”), I do notice that disturbing separation of the math from the reporting. It was inevitable as the math has gotten more sophisticated. But it does require extra training on all sides, and extra respect and extra story-creation time.

    All of us early “teaching” pioneers in the field (Tom Johnson at San Francisco State, Brent Houston at NICAR, Jim Brown at IUPUI, Steve Doig in Miami, Phil Meyer, me at Columbia University and a few others, all starting in the 1980s) stressed that 90 or 95 percent of the job was in the reporting, not the raw calculations. I used to tell my students that I spent 5% of my time at the computer and 95% on the phone or on the street. I still do. Here’s one of my old “teaching” pieces:

    The comment on bicycle helmets is revealing. The links are to supposed meta-analyses that are, despite the mathematical pretense, essentially “literature reviews with numbers.” This is also typical of such supposedly professional publications as the Journal of the American Medical Association. If the folks with PhDs can’t get it right, I’m frankly amazed that “mere” journalists still come darn close to the mark. Wear a helmet while bicycling, motorcycling, skateboarding…. Helmets are not as good as claimed (and what is?), but they are far more than trivially protective.

  • Shelly

    But but but a bike helmut would ruin my hair.