• Fellowships
  • Reports
  • Lab
  • Storyboard
    • Nieman Foundation at Harvard
    HOME
    About
    Subscribe
    Archives
              
    Foundation
    Reports
    Storyboard
    LATEST STORY
    Would you pay to be able to quit TikTok and Instagram? You’d be surprised how many would
    Business Models Mobile & Apps Audience & Social Aggregation & Discovery Reporting & Production
    ABOUT                    SUBSCRIBE
    March 1, 2013, 1:19 p.m.
    Build your own search engine with Common Crawl
    LINK: www.theverge.com  ➚   |   Posted by: Joshua Benton   |   March 1, 2013

    Russell Brandom at The Verge has a piece on Common Crawl, “a non-profit foundation dedicated to providing an open repository of web crawl data that can be accessed and analyzed by everyone.” At one extreme, that dataset could be used to build your own local or targeted search engine; at a smaller scale, it could be a boon to data journalists:

    For example, web crawl data can be used to spot trends and identify patterns in politics, economics, health, popular culture and many other aspects of life. It provides an immensely rich corpus for scientific research, technological advancement, and innovative new businesses. It is crucial for our information-based society that the web be openly accessible to anyone who desires to utilize it.

    Be forewarned: If you think a Hadoop cluster is a kind of Easter candy, this isn’t the weekend hacking project for you. (Here’s an earlier piece from MIT Technology Review.)

    Show tags
     
    What’s the best way to follow how the news is changing?
    Our daily email, with all the freshest future-of-journalism news.
    Prefer a once-a-week email? »
    The latest from Nieman Lab
    Would you pay to be able to quit TikTok and Instagram? You’d be surprised how many would
    Peter Martin
    BREAKING: The ways people hear about big news these days; “into a million pieces,” says source
    Joshua Benton
    In 1924, a magazine ran a contest: “Who is to pay for broadcasting and how?” A century later, we’re still asking the same question
    Julia Barton
    You’re more likely to believe fake news shared by someone you barely know than by your best friend
    Joshua Benton
    To find readers for longform investigations, Public Health Watch leans on partners and in-person work
    Sarah Scire
    Join the 60,000 who get the freshest future-of-journalism news in our daily email.
    Would you pay to be able to quit TikTok and Instagram? You’d be surprised how many would
    “The relationship he has uncovered is more like the co-dependence seen in a destructive relationship, or the way we relate to addictive products such as tobacco that we know are doing us harm.”
    BREAKING: The ways people hear about big news these days; “into a million pieces,” says source
    The New York Times and the Washington Post compete with meme accounts for the chance to be first with a big headline.
    In 1924, a magazine ran a contest: “Who is to pay for broadcasting and how?” A century later, we’re still asking the same question
    Radio Broadcast received close to a thousand entries to its contest — but ultimately rejected them all.
    Help advance the Nieman Foundation’s mission “to promote and elevate the standards of journalism” by making a donation.

    To promote and elevate the standards of journalism

    Covering thought leadership in journalism

    Pushing to the future of journalism

    Exploring the art and craft of story
    © 2024 by the President and Fellows of Harvard College  /  Some rights reserved
    Harvard Trademark Privacy Digital Accessibility
    Walter Lippmann House    One Francis Ave.    Cambridge, MA 02138    617 495 2237