Twitter  .@craigmod: "We have enough photos of coffee in Brooklyn, right?" nie.mn/1hoFePU (also, the thinking behind his startup @sayhi)  
Nieman Journalism Lab
Pushing to the future of journalism — A project of the Nieman Foundation at Harvard
fuego1

Introducing OpenFuego, your very own heat-seeking Twitter bot

We’re open-sourcing the code behind Fuego, which mines Twitter to find the links a community is talking about most. Build your own!

Editor’s note: Today we’re announcing the release of OpenFuego, an open-source version of Fuego, our popular Twitter robot. Andrew Phelps — formerly of Nieman Lab, now at The New York Times — wrote Fuego’s backend code and has been working to convert it to sharable shape in his spare time. We hope you give it a download and build your own Fuego.

I just want to say that sharing your code with the world for the first time is terrifying. Hold me.

Two years ago, when I was a staff writer at the Lab, boss Josh Benton asked me to build what would become Fuego, a robot that monitors Twitter to surface the stories that journalism nerds are sharing and talking about.

Actually, first he asked me if I knew how to build something like that; I lied and said yes. The idea interested me, and I figured the best way to finally learn how to program would be to tackle a project. I wrote some pretty clunky PHP, the only scripting language I (kind of) knew. I resorted to a lot of hacks and shortcuts, but it worked. And as long as it worked, no one had to know what was under the hood.

Fuego (and its accompanying Twitter account) became a daily habit for a lot of readers and a reliable way for us to keep up with the future-of-news world. The best way we could come up with to describe it was this: It’s like being on Twitter all day long without having to be on Twitter all day long. To this day I receive many, many emails from people asking (a) if I could make a Fuego for their organization or (b) if the Lab would ever open-source it.

fuego2

When Twitter announced it would make significant changes to its API, Fuego needed an overhaul. And now that Twitter has finally flipped the off switch on API v1.0, we decided this would be a good moment to release a version of Fuego you can install and use yourself. In keeping with the open-source spirit of our friends across town, OpenFuego is now available and distributed under an MIT license.

Which brings me to terrifying. I’ve always been a tinkerer but never a programmer. Most people I’ve met who can write code, at least in the journalism world, say they aren’t programmers either. A lot of people seem to learn by accident. But writing code is practically a daily part of my job in the newsroom, even if it’s not in my job description.

So I lay myself bare, my amateurish code made available for public scrutiny, hoping you’ll help make it better and be inspired to do cool new things.

fuego3

A bit about how it works: The setup process is like WordPress. You create a database and follow the instructions in the config file. Then it’s time to play curator. Choose a group of authorities — the Twitter users who will seed the universe you want to track. Say you’re a beat reporter covering the universe of local politics. You might select 10 authorities — a mix of your fellow beat reporters, local politicians, a gadfly or two, and anyone else who’s really engaged in the business of city hall or local government.

(What makes for a good authority? It’s not necessarily that they tweet a lot about the subject you’re interested in — although that helps. What makes an authority good or bad is how well they’ve curated the collection of people they follow. So, in this example, if your local mayor follows a bunch of local politicos and other government officials, she could be a good OpenFuego authority. But if she’s a huge Mets fan and, of the 100 people she follows, 70 are baseball-related, she’s probably not going to be particularly useful to your OpenFuego — even if she’s in the corner office at city hall.)

After you identify these 10 authorities, OpenFuego does the rest. The app follows them on Twitter, as well as all of the people they follow, up to a total of 5,000 sources. When one of those sources shares a link, the link is scored according to the source’s influence. Influence is determined by the number of authorities who follow that source.

This process happens in the background, 24 hours a day. What we’re releasing now, OpenFuego, is just the back-end part. If you want to display that data on a web page — the way we do at niemanlab.org/fuego — you’ll have to make that yourself. (Our implementation of Fuego allows us to look at only stories that have surfaced over a set period of time — the last four hours, the last 24 hours, the last week, and so on — and to rate more recently surfaced stories higher than older ones, to emphasize the newsiness of what it’s finding.) OpenFuego contains a function to make database queries easy, and we’ll be providing some front-end examples to help get you started.

For more details, and to fork the code, see the GitHub project page. Please be gentle.

                                   
What to read next
half-life
Craig Silverman    April 22, 2014
When you commit to explaining the important issues in the news, you commit to a life of updating. What’s the best way to manage a corpus of news knowledge with an uncertain half-life?
  • http://ldopa.net/ Jeff Hobbs

    Fantastic! Fuego always returns great links — can’t wait to give this a spin.

  • Natalie Sutton

    This looks fantastic thanks Nieman!

  • Laura Oliver

    Thanks for being brave and for opening this up. Looking forward to having a play with my (even more) limited knowledge of code.

  • http://www.billysuratt.com/ Billy Suratt

    I’m a little disappointed you released this four days ago and nobody has created a WordPress plugin yet. (Kidding — thanks for what looks to be a great resource.)

  • Iwan

    Thanks for this great app, just what I need. Just one (technical) question… can someone look at it? https://github.com/niemanlab/openfuego/issues/5

  • JamesDiGioia

    It’s a bit more complicated to build a WordPress plugin than that, because it has to run continuously, and most shared hosts (where I would presume most people have their blogs hosted) won’t typically allow you to run a script like this.

  • Murphy

    Hi can you please tell me why is it stopping after 24 hours of background running in my server?

  • http://www.niemanlab.org/ Joshua Benton

    Almost certainly because your webhost is killing background cron jobs after 24 hours.

  • Murphy

    thanks,
    Can you tell me How to get the last updated time ?