The Internet Archive hopes to boost its collections through funding from the Knight News Challenge
ABOUT                    SUBSCRIBE
Sept. 23, 2013, 11:59 a.m.

The Hub of the Twitterverse: The Boston Globe has built a localized, tweet-powered news aggregator

61Fresh is the latest experiment from Globe Lab, the paper’s R&D department, to find ways to harness social signals to see what readers are talking about.

61freshscreenshotIn the Boston area, the passing of a Dunkin Donuts franchise is news. So are the finer points of Tom Brady’s hair. But in this instance, we know the closing of the Mass Ave. Dunkin reached a certain level of…prominence? virality? social lift? — because it made an appearance on 61Fresh, the experimental news aggregator built by the Globe Lab, the research and development team inside The Boston Globe.

This tweet, in particular, caught fire:

(In fairness, Matt Damon, a napkin, and apples also made that Dunkin newsworthy.)

What, exactly, is 61Fresh? And what’s with the name?

“Half the people who respond to it ask, ‘What’s up with the name?’ And I say, ‘Oh, it’s like 617, but six-one fresh,” said Chris Marstall, a creative technologist in the Globe Lab. (Out-of-towners: 617 is the primary Boston area code.)

The name, like the site, is an experiment. Think of 61Fresh as a close cousin to something like Techmeme, Mediagazer, or our very own heat-seeking Twitter bot, Fuego. Where those algorithmic link aggregators are topic-based, 61Fresh instead has a geographic focus: “It’s philosophically the same,” Marstall says. “It’s using a network to find the most relevant stories in a narrow domain for now.”

Every 10 minutes, 61Fresh takes a spin around a localized slice of Twitter searching for rising news items. On any given day, you’re likely to find stories on apartment fires, startup companies, or the Bruins. Unlike some aggregators that mix in some human control, 61Fresh is 100 percent algorithm-driven — which means both that it runs without the need for ongoing staff resources and that the end product can be a little raw, with different versions of the same story repeated during breaking news.

The project is still in early stages, and Marstall said they’re still figuring out the best way for users to interact with 61Fresh, which for now exists as a website and Twitter feed. “The stories that pop up are stories drawing interest on Twitter,” he said. “It’s sort of an open question if this can be a standalone site in the future or if this is something that is a compelling part of Boston.com.”

But how does 61Fresh figure out what’s news, what’s talk, and what’s noise? The algorithm built by Globe Lab looks for tweets that link to stories or updates from just over 500 pre-selected websites in the Boston area. Aside from recognizable names like Boston.com, and the Boston Herald, the universe also includes stories from radio, TV, town government sites, universities, and blogs, including the growing network of online outlets that cover the particular cross-section of education, technology, and culture unique to the Boston area.

“We got three daily newspapers, a bunch of weekly newspapers, we got TV and radio, and all these blogs. That’s just the tip of the iceburg,” Marstall said. “It shows there’s a lot of vibrancy here and a lot of smart people fighting to get attention for their stories.”

Here’s how they explain it:

We’re downloading a real time stream of the local Twitter posts (originating roughly within Route 128) that link to stories on one of those 500 sites. Every tweet represents a vote. Our algorithm mixes popularity with freshness and a couple other ingredients, discounting the value of ‘official’ accounts and bots. We group similar articles together, generating a diverse feed that changes frequently throughout the day.

What you get with 61Fresh is a cascade of headlines, but in the form 140 characters, as the links that are pulled in use the text of the most prominent tweet on a particular story. “We want this to look like a conversation, we don’t want this to look like a bunch of headlines,” Marstall said. Keeping in that theme, each story pulled into 61Fresh shows a roster of people who have tweeted the link, as well as Twitter prompts to reply, retweet, or favorite.

One interesting feature on 61Fresh is the “mute sports” button, which, as the name implies, lets users excise anything related to sports from their feed. “Early on, we realized the sports stories could really dominate,” Marstall said. “I like it — I’m a sports fan — but I know for a lot of people it’s total noise.”

Ali Hashmi, a former Knight fellow at the Globe Lab who is now at MIT, created a natural language processing system that helps the aggregator parse sports from the rest of the news. Using a base set of 1,000 sports stories, the system learns how to spot a sports article and pluck it out of the stream of stories, Marstall said, if you want neither pucks, punts, nor pitchers.

61Fresh is one of the latest projects to come out of the partnership between the Globe and MIT’s Center for Civic Media. Originally, the lab’s creative technologists and Knight fellows from MIT were working on a way to archive Globe reporters’ tweets from the Boston Marathon bombing. Lessons and code from that were put into use for 61Fresh, Marstall said.

This is not the first time the Globe Lab has experimented with social media feeds as a means of monitoring life in Boston. Previously the lab developed Snap, a tool that maps Instgram photos to their geolocated spots around the city.

Jeff Moriarty, the Globe’s vice president of digital products and general manager of Boston.com, said 61Fresh “fits in with the idea that we’re moving Boston.com to include more of what people are saying and what people are doing right now in Boston.” With the split between the two Globe sites, Boston.com has been focused on news and information, but also digital tools and online discussion, Moriarty said. The central idea is to be able to give readers a sense of what they need to know in a format that works best for them. “By combining both the news and what’s popular on Twitter, you get this pretty unique view of what’s happening right now,” Moriarty said.

At the moment, 61Fresh is a little rough around the edges, the design is not particularly eye catching, and the team at Globe Lab is catching problems as they arise. But the lessons from the project will be applied in the future, either for a stand-alone aggregator or built into parts of Boston.com. Moriarty said he thinks the project can have a lot of value for niche topics like sports or technology — pulling in tweets and headlines for a Boston audience interested in the NFL or biotech, for instance. As Moriarty sees it, the job of Boston.com isn’t just to deliver the news, but to provide context, conversation, and the other pieces to help readers see the bigger picture.

“I think it really needs to be what we do as a media organization: People want someone to help them make sense of what is going on around them,” Moriarty said.

POSTED     Sept. 23, 2013, 11:59 a.m.
Show comments  
Show tags
Join the 15,000 who get the freshest future-of-journalism news in our daily email.
The Internet Archive hopes to boost its collections through funding from the Knight News Challenge
The home of the Wayback Machine and other efforts to preserve the Internet is among 22 projects based around libraries receiving $3 million in funding through the Knight News Challenge.
Constantly tweaking: How The Guardian continues to develop its in-house analytics system
Since its launch in 2011, The Guardian has consistently made changes to its in-house analytics tool, Ophan.
Bloomberg Business’ new look has made a splash — but don’t just call it a redesign
Bloomberg digital editor Joshua Topolsky on uncomfortable news design, new ad units, and why they killed the comments.
What to read next
Don’t try too hard to please Twitter — and other lessons from The New York Times’ social media desk
The team that runs the Times’ Twitter accounts looked back on what they learned — what worked, what didn’t — from running @NYTimes in 2014.
728From explainers to sounds that make you go “Whoa!”: The 4 types of audio that people share
How can public radio make audio that breaks big on social media? A NPR experiment identified what makes a piece of audio go viral.
722Q&A: Amy O’Leary on eight years of navigating digital culture change at The New York Times
“In 2007, as digital people, we were expected to be 100 percent deferent to all traditional processes. We weren’t to bother reporters or encourage them to operate differently at all, because what they were doing was the very core of our journalism.”
These stories are our most popular on Twitter over the past 30 days.
See all our most recent pieces ➚
Encyclo is our encyclopedia of the future of news, chronicling the key players in journalism’s evolution.
Here are a few of the entries you’ll find in Encyclo.   Get the full Encyclo ➚
The Orange County Register
The Globe and Mail
Demand Media
National Review
BBC News
The Christian Science Monitor
Fox News