Nieman Foundation at Harvard
HOME
          
LATEST STORY
The New York Times is trying to make VR films that aren’t one-offs, and that keep readers coming back
ABOUT                    SUBSCRIBE
June 17, 2009, 2:01 p.m.

Knight News Challenge: A grant to DocumentCloud promises a data boost for investigative journalism

The Knight News Challenge‘s biggest winner, with a two-year grant of $719,500, is DocumentCloud, the primary-source index conceived by journalists and developers at ProPublica and The New York Times. Here’s why you should care: There’s good reason to believe the project will transform how some investigative journalism is conducted — and who conducts it.

Like a lot of software in the cloud, this one is complicated to explain. I wrote a long overview of DocumentCloud in November, and you can read their initial grant application in my first post about the project. Aron Pilhofer, editor of interactive news technologies at the Times and one of the project’s creators, told me on Monday, “DocumentCloud isn’t really conducive to a two-minute elevator pitch.” But later in our conversation, he ventured one: “It will turn documents into data.”

In the analog version of investigative journalism, a reporter obtains documents from sources and freedom-of-information requests, writes a story, and… that’s it. If we’re lucky, the materials are posted as unwieldy and barely searchable PDFs.

DocumentCloud’s vision is to collect, archive, and index the text and metadata of all documents used by participating news organizations, advocacy groups, bloggers, and others — “so they’re not just sitting in the corner of a newsroom collecting dust,” Pilhofer explained. That way, anyone — from other news outlets to curious readers — will be able to search across all documents in the project to find information that might not have been relevant to the original piece. If it were an animated TV series, the catchphrase might be, With our newsrooms combined — we are DocumentCloud!

Early partners in the project include the Times, ProPublica (the non-profit investigative journalism outfit) Gotham Gazette (a New York City news site published by Citizens Union Foundation, themselves winners of two Knight News Challenge grants), TPM Muckraker (the investigative arm of Talking Points Memo), and the National Security Archive (home to the largest public repository of declassified government documents). Are you salivating yet?

Anyone who has waded through the National Security Archive’s wealth of FBI files and CIA reports will immediately recognize the benefit of DocumentCloud. What if you could search across the entire archive for a particular topic of interest (Pilhofer suggested Marilyn Monroe) and get pinged whenever that topic shows up in a new document? Or what if you were a business journalist up to your neck in SEC filings? Or a local blogger keeping tabs on your congressman’s earmarks?

Details are still being worked out, but running materials through DocumentCloud will involve some sort of optical character recognition (to render those pesky image-based PDFs favored by some government agencies into searchable text) and Open Calais (to extract metadata like names, locations, and dates for more effective indexing). The news organizations that contribute documents will likely host them as well, said Scott Klein, editor of online development at ProPublica and a creator of DocumentCloud. He described the project as more of a “card catalog” than, say, a repository.

There are other aspects of the project that I’d be happy to discuss in the comments, and maybe we can get Pilhofer and Klein to weigh in here as well (as though I’m not hyping them enough). The other creators are Eric Umansky of ProPublica and Ben Koski of the Times. So if you have any questions about DocumentCloud, feel free to ask, and I’ll work on getting answers.

UPDATE, 3:30 p.m.: The Document Cloud folks introduced their project at the Future of News and Civic Media Conference at MIT today, and I shot some raw video with Qik:

POSTED     June 17, 2009, 2:01 p.m.
PART OF A SERIES     Knight News Challenge 2009
SHARE THIS STORY
   
Show comments  
Show tags
 
Join the 15,000 who get the freshest future-of-journalism news in our daily email.
The New York Times is trying to make VR films that aren’t one-offs, and that keep readers coming back
With its new Pluto-focused project, upcoming episodic series, and experiments with “meditative VR,” The Times is experimenting with different applications for the new technology.
Newsonomics: Can a Bezos buddy act help fend off Gannett’s bid for Tribune?
Tribune Publishing’s Michael Ferro says he wants to bring The Washington Post’s Arc CMS to its newspapers. Is that a grasp at credibility or a model for other news companies to outsource their tech stacks?
Die Welt’s analytics system de-emphasizes clicks and demystifies what it considers a “quality” story
Every story’s performance reduced to a single score: a reductionist take on journalism or a way to make a news organization’s values concrete?
What to read next
0
tweets
From Nieman Reports: Why your news site should be more readable for the visually impaired
Over eight million Americans have trouble with their vision. Here’s how newsrooms can (and should) design with them in mind.
0A new podcast from Mic and The Economist aims for a global perspective on the 2016 election
“Part of the appeal here is that this an unexpected combination. But I think we’d argue that we’re not that different,” said Economist deputy editor Tom Standage.
0The New Yorker Today is the magazine’s new iPhone app for its online articles (plus cartoons)
The app will be free for a time while The New Yorker figures out how many articles readers can access before hitting the paywall.
These stories are our most popular on Twitter over the past 30 days.
See all our most recent pieces ➚
Encyclo is our encyclopedia of the future of news, chronicling the key players in journalism’s evolution.
Here are a few of the entries you’ll find in Encyclo.   Get the full Encyclo ➚
Associated Press
Newsweek
Wikipedia
Suck.com
Bureau of Investigative Journalism
Yahoo
Center for Investigative Reporting
Lens
The Ann Arbor Chronicle
Voice Media Group
ProPublica
Conde Nast