Nieman Foundation at Harvard
HOME
          
LATEST STORY
Newsonomics: On end games and end times
ABOUT                    SUBSCRIBE
June 17, 2009, 2:01 p.m.

Knight News Challenge: A grant to DocumentCloud promises a data boost for investigative journalism

The Knight News Challenge‘s biggest winner, with a two-year grant of $719,500, is DocumentCloud, the primary-source index conceived by journalists and developers at ProPublica and The New York Times. Here’s why you should care: There’s good reason to believe the project will transform how some investigative journalism is conducted — and who conducts it.

Like a lot of software in the cloud, this one is complicated to explain. I wrote a long overview of DocumentCloud in November, and you can read their initial grant application in my first post about the project. Aron Pilhofer, editor of interactive news technologies at the Times and one of the project’s creators, told me on Monday, “DocumentCloud isn’t really conducive to a two-minute elevator pitch.” But later in our conversation, he ventured one: “It will turn documents into data.”

In the analog version of investigative journalism, a reporter obtains documents from sources and freedom-of-information requests, writes a story, and… that’s it. If we’re lucky, the materials are posted as unwieldy and barely searchable PDFs.

DocumentCloud’s vision is to collect, archive, and index the text and metadata of all documents used by participating news organizations, advocacy groups, bloggers, and others — “so they’re not just sitting in the corner of a newsroom collecting dust,” Pilhofer explained. That way, anyone — from other news outlets to curious readers — will be able to search across all documents in the project to find information that might not have been relevant to the original piece. If it were an animated TV series, the catchphrase might be, With our newsrooms combined — we are DocumentCloud!

Early partners in the project include the Times, ProPublica (the non-profit investigative journalism outfit) Gotham Gazette (a New York City news site published by Citizens Union Foundation, themselves winners of two Knight News Challenge grants), TPM Muckraker (the investigative arm of Talking Points Memo), and the National Security Archive (home to the largest public repository of declassified government documents). Are you salivating yet?

Anyone who has waded through the National Security Archive’s wealth of FBI files and CIA reports will immediately recognize the benefit of DocumentCloud. What if you could search across the entire archive for a particular topic of interest (Pilhofer suggested Marilyn Monroe) and get pinged whenever that topic shows up in a new document? Or what if you were a business journalist up to your neck in SEC filings? Or a local blogger keeping tabs on your congressman’s earmarks?

Details are still being worked out, but running materials through DocumentCloud will involve some sort of optical character recognition (to render those pesky image-based PDFs favored by some government agencies into searchable text) and Open Calais (to extract metadata like names, locations, and dates for more effective indexing). The news organizations that contribute documents will likely host them as well, said Scott Klein, editor of online development at ProPublica and a creator of DocumentCloud. He described the project as more of a “card catalog” than, say, a repository.

There are other aspects of the project that I’d be happy to discuss in the comments, and maybe we can get Pilhofer and Klein to weigh in here as well (as though I’m not hyping them enough). The other creators are Eric Umansky of ProPublica and Ben Koski of the Times. So if you have any questions about DocumentCloud, feel free to ask, and I’ll work on getting answers.

UPDATE, 3:30 p.m.: The Document Cloud folks introduced their project at the Future of News and Civic Media Conference at MIT today, and I shot some raw video with Qik:

POSTED     June 17, 2009, 2:01 p.m.
PART OF A SERIES     Knight News Challenge 2009
SHARE THIS STORY
   
Show comments  
Show tags
 
Join the 15,000 who get the freshest future-of-journalism news in our daily email.
Newsonomics: On end games and end times
Can publishers find a sustainable business model this new age of Facebook/Apple/Snapchat/Twitter/Google distributed content? And is local news destined to be left behind?
What Scribd’s growing pains mean for the future of digital content subscription models
It turns out that ebook subscription models don’t work very well when people read too much. So what happens next?
How research (and PowerPoints) became the backbone of National Journal’s membership program
“We no longer look at National Journal simply as a news source, but as a collection of resources, as well as a collection of experts we can turn to on occasion.”
What to read next
2843
tweets
A blow for mobile advertising: The next version of Safari will let users block ads on iPhones and iPads
Think making money on mobile advertising is hard now? Think how much more difficult it will be with a significant share of your audience is blocking all your ads — all with a simple download from the App Store.
1763For news organizations, this was the most important set of Apple announcements in years
A new Flipboard-clone with massive potential reach, R.I.P. Newsstand, and news stories embedded deeper inside iOS — it was a big day for news on iPhones and iPads.
762Newsonomics: 10 numbers that define the news business today
From video to social, from mobile to paywalls — these data points help define where we are in the “future of news” today, like it or not.
These stories are our most popular on Twitter over the past 30 days.
See all our most recent pieces ➚
Encyclo is our encyclopedia of the future of news, chronicling the key players in journalism’s evolution.
Here are a few of the entries you’ll find in Encyclo.   Get the full Encyclo ➚
Kaiser Health News
Arizona Guardian
Connecticut Mirror
The New Yorker
Media Consortium
Knight Foundation
The Ann Arbor Chronicle
Quartz
El País
DNAinfo
Ushahidi
NBCNews.com