Nieman Foundation at Harvard
HOME
          
LATEST STORY
Votebeat will cover local election administration as a permanent newsroom
ABOUT                    SUBSCRIBE
March 1, 2013, 1:19 p.m.
LINK: www.theverge.com  ➚   |   Posted by: Joshua Benton   |   March 1, 2013

Russell Brandom at The Verge has a piece on Common Crawl, “a non-profit foundation dedicated to providing an open repository of web crawl data that can be accessed and analyzed by everyone.” At one extreme, that dataset could be used to build your own local or targeted search engine; at a smaller scale, it could be a boon to data journalists:

For example, web crawl data can be used to spot trends and identify patterns in politics, economics, health, popular culture and many other aspects of life. It provides an immensely rich corpus for scientific research, technological advancement, and innovative new businesses. It is crucial for our information-based society that the web be openly accessible to anyone who desires to utilize it.

Be forewarned: If you think a Hadoop cluster is a kind of Easter candy, this isn’t the weekend hacking project for you. (Here’s an earlier piece from MIT Technology Review.)

Show tags Show comments / Leave a comment
 
Join the 60,000 who get the freshest future-of-journalism news in our daily email.
Votebeat will cover local election administration as a permanent newsroom
“How do you produce journalism that strengthens elections? That’s the question that runs through my mind every day.”
Hype is a weaponized form of optimism
Want to know the true value of AI, NFTs, and other much-touted technologies? Ignore the news and look at the harsh judgment of the market.
For print newspapers, one Florida retirement community is a better market than Atlanta, St. Louis, or Portland
For local newspapers, print circulation has collapsed for every audience except retirees. That’s why the daily paper in The Villages, Florida (metro population 129,752) prints as many copies as the one in Atlanta (metro population 6,930,423).