Nieman Foundation at Harvard
HOME
          
LATEST STORY
U.S. police have attacked journalists more than 120 times since May 28
ABOUT                    SUBSCRIBE
March 1, 2013, 1:19 p.m.
LINK: www.theverge.com  ➚   |   Posted by: Joshua Benton   |   March 1, 2013

Russell Brandom at The Verge has a piece on Common Crawl, “a non-profit foundation dedicated to providing an open repository of web crawl data that can be accessed and analyzed by everyone.” At one extreme, that dataset could be used to build your own local or targeted search engine; at a smaller scale, it could be a boon to data journalists:

For example, web crawl data can be used to spot trends and identify patterns in politics, economics, health, popular culture and many other aspects of life. It provides an immensely rich corpus for scientific research, technological advancement, and innovative new businesses. It is crucial for our information-based society that the web be openly accessible to anyone who desires to utilize it.

Be forewarned: If you think a Hadoop cluster is a kind of Easter candy, this isn’t the weekend hacking project for you. (Here’s an earlier piece from MIT Technology Review.)

Show tags Show comments / Leave a comment
 
Join the 50,000 who get the freshest future-of-journalism news in our daily email.
U.S. police have attacked journalists more than 120 times since May 28
“Although in some incidents it is possible the journalists were hit or affected accidentally, in the majority of the cases we have recorded the journalists are clearly identifiable as press, and it is clear that they are being deliberately targeted.”
Riot or resistance? The way the media frames the unrest in Minneapolis will shape the public’s view of protest
Research finds that protests about anti-black racism and indigenous people’s rights receives the least legitimizing coverage.
Unicorn Riot, a nonprofit media collective, is covering the Minneapolis protests live and close up
Unicorn Riot is just five years old, but this week’s unrest isn’t its first time covering protests against a police killing in the Twin Cities.