HOME
          
LATEST STORY
The Internet Archive hopes to boost its collections through funding from the Knight News Challenge
ABOUT                    SUBSCRIBE
Nov. 10, 2010, 1:30 p.m.

Talking Points Memo’s first developer talks startup life, jumping to ProPublica and data journalism

What’s it like being the only in-house techie at a news startup? Talking Points Memo’s first developer Al Shaw says “it’s kind of like being a reporter….you have to be a generalist,” doing everything from ad-side work to election-night interactives.

Shaw was the primary technical force behind most of the bells and whistles that cropped up at TPM over the past two years, including a redesign that lets producers switch up the layout of the homepage, and an array of slick interactives like the real-time election results tracker that made TPM look a lot less like a scrappy startup and more like an establishment outlet on Election Night earlier this month. (Shaw is quick to explain he had some help on the election map from Erik Hinton, TPM’s technical fellow.) He’s also been good about blogging about his technical endeavors in ways that could be useful to his peers at other news organizations.

Shaw announced last month he is leaving TPM to start a new gig at ProPublica, where he’ll keep working on data-driven journalism. On one of his few days off between jobs, I talked with him about what it’s like working for a news startup, what he hopes to accomplish at ProPublica, and where he thinks data journalism is headed. Below is a lightly edited transcript. (Disclosure: I used to work at TPM, before Al started there.)

Laura K. McGann: How did you approach your job at TPM? What did you see as your mission there?

Al Shaw: When I started, I came on as an intern right before the ’08 election. At that point, they didn’t have anyone in house who really knew much about programming or design or software. I came on and I saw an opportunity there because TPM is such a breaking-news site, and their whole goal is to do stuff really fast, that they needed someone to do that, but on the technology side, too.

I had a big role in how we covered the 2008 election. We became able to shift the homepage, rearrange stuff. Being able to really elevate what you can do in blogging software. That was kind of the first foray. Then I started redesigning some of the other sections. But the biggest impact I had was redesigning the homepage. That was about a year ago. I had the same goal of being able to empower the editors and nontechnical types to have a bigger palette of what they can do on the site. I created this kind of meta-CMS on top of the CMS that allowed them to rearrange where columns were and make different sections bigger and smaller without having to get into the code. That really changed the way the homepage works.

There is still Movable Type at the core, but there’s a lot of stuff built up around the sides. When we started to build bigger apps, like the Poll Tracker and election apps, we kind of moved off Movable Type all together and started building in Ruby on Rails and Sinatra. They’re hosted on Amazon EC2, which is a cloud provider.

LKM: What have you built that you’re the most proud of?

AS: Probably the Poll Tracker. It was my first project in Rails. It just had enormous success; it now has 14,000 polls in it. Daily Kos and Andrew Sullivan were using it regularly to embed examples of races they wanted to follow and it really has become a central part of TPM and the biggest poll aggregator on the web now. I worked with an amazing Flash developer, Michiko Swiggs, she did the visual parts of the graph in Flash. I think a lot of it was really new in the way you could manipulate the graph — if you wanted to take out certain pollsters, certain candidates, methods, like telephone or Internet, and then you could see the way the trend lines move. You can embed those custom versions.

I think the election tool was also a huge success [too], both technologically and on the design and journalism side. We got linked to from Daring Fireball. We also got linked to from ReadWriteWeb and a lot of more newsy sites. Andrew Sullivan said it was the best place to watch the elections. Because we took that leap and said we’re not going to use Flash, we got a lot of attention from the technology community. And we got a lot of attention from kind of the more political community because of how useable and engaging the site was. It was kind of a double whammy on that.

LKM: What was your experience working with reporters in the newsroom? TPM is turning ten years old, but it’s still got more of a startup feel than a traditional newspaper.

AS: It’s definitely a startup. I would fade in and out of the newsroom. Sometimes I’d be working on infrastructure projects that dealt with the greater site design or something with the ad side, or something beyond the day-to-day news. But then I’d work with the reporters and editors quite a bit when there was a special project that involved breaking news.

So for example, for the Colbert-Stewart rallies we put up a special Twitter wire where our reporters go out to the rallies and send in tweets and the tweets would get piped into a special wire and they’d go right onto the homepage. I worked with editors on how that wire should feel and how it should work and how reporters should interact with it. I remember one concern was, what if someone accidentally tweets somethng and it ends up on the homepage. How do we delete that? I came up with this system with command hashtags, so a reporter could send in a tweet with a special code on it which would delete a certain tweet and no one else would know about that, except for the reporter.

A lot of the job was figuring out what reporters and editors wanted to do and figuring out how to enable that with the technology we had and with the resources we had.

LKM: I remember an instance in my old newsroom where we had a tweet go up on the front page of another site and the frantic emails trying to get it taken down.

AS: Twitter is such an interesting medium because it’s so immediate, but it’s also permanent. We’re having a lot of fun with it, but we’re still learning how best to do it. We did this thing called multi-wire during the midterms, which was a combination of tweets and blog posts in one stream. There was a lot of experimentation with: When do we tweet as compared to a blog post? Should we restrict it to certain hours? That was a really interesting experiment.

LKM: What emerging trends do you see going on in data-driven or interactive journalism?

AS: It’s really good that a lot of sites are starting to experiment more with data-driven journalism, especially as web frameworks and cheap cloud hosting become more prevalent and you can learn Rails and Django, it’s really easy to get a site up that’s based around data you can collect. I do see two kind of disturbing trends that are also happening. One is the rise of infographics. They may not be as useful as they are pretty. You see that a lot just all over the place now. The other problem you see is the complete opposite of that where you’ll get just a table of data filling up your whole screen. The solution is somewhere in between that. You have a better way of getting into it.

It’s really great that there’s kind of a community forming around people that are both journalists and programmers. There’s this great group called Hacks/Hackers that brings those two cohorts together and lets them learn from each other.

LKM: How about at ProPublica? You mentioned you aren’t sure entirely what you’re going to do, but broadly, what do you hope to accomplish there?

AS: I’m most excited about working more closely with journalists on data sets and finding the best ways of presenting those and turning them into applications. That was one thing I was able to do with Poll Tracker, but it didn’t seem like TPM had as big of a commitment to individual stories that could have side applications. Poll Tracker was more of a long-running project. ProPublica is really into delving deeply onto one subject and finding data that can be turned into an application so the story isn’t just a block of text, there’s another way of getting at it.

One of the other things they’re working on is more tools for crowdsourcing and cultivating sources. I know that they want to start building an app or a series of apps around that. And they’re doing some cool stuff with Amazon Mechanical Turk for kind of normalizing and collecting data. I’m sure there’s going to be a lot more fun stuff to do like that.

POSTED     Nov. 10, 2010, 1:30 p.m.
SHARE THIS STORY
   
Show comments  
Show tags
 
Join the 15,000 who get the freshest future-of-journalism news in our daily email.
The Internet Archive hopes to boost its collections through funding from the Knight News Challenge
The home of the Wayback Machine and other efforts to preserve the Internet is among 22 projects based around libraries receiving $3 million in funding through the Knight News Challenge.
Constantly tweaking: How The Guardian continues to develop its in-house analytics system
Since its launch in 2011, The Guardian has consistently made changes to its in-house analytics tool, Ophan.
Bloomberg Business’ new look has made a splash — but don’t just call it a redesign
Bloomberg digital editor Joshua Topolsky on uncomfortable news design, new ad units, and why they killed the comments.
What to read next
2902
tweets
Don’t try too hard to please Twitter — and other lessons from The New York Times’ social media desk
The team that runs the Times’ Twitter accounts looked back on what they learned — what worked, what didn’t — from running @NYTimes in 2014.
728From explainers to sounds that make you go “Whoa!”: The 4 types of audio that people share
How can public radio make audio that breaks big on social media? A NPR experiment identified what makes a piece of audio go viral.
722Q&A: Amy O’Leary on eight years of navigating digital culture change at The New York Times
“In 2007, as digital people, we were expected to be 100 percent deferent to all traditional processes. We weren’t to bother reporters or encourage them to operate differently at all, because what they were doing was the very core of our journalism.”
These stories are our most popular on Twitter over the past 30 days.
See all our most recent pieces ➚
Encyclo is our encyclopedia of the future of news, chronicling the key players in journalism’s evolution.
Here are a few of the entries you’ll find in Encyclo.   Get the full Encyclo ➚
GateHouse Media
The Guardian
Corporation for Public Broadcasting
Medium
CBS News
The Awl
San Francisco Chronicle
The Christian Science Monitor
Patch
The Nation
La Nación
Groupon