What happened when a college newspaper abandoned its website for Medium and Twitter
ABOUT                    SUBSCRIBE
Jan. 23, 2012, noon

Meet Deep Dive, the New York Times’ experimental context engine and story explorer

Using an article as a jumping-off point, Deep Dive can create a custom, contextual feed that will allow readers to follow topics in the news.

Thinking about the sheer volume of information — stories, images, videos, data — available from The New York Times can evoke a simultaneous glee and terror. Skimming Times Wire gives you an idea of the hundreds of pieces of content they produce each day. For readers, it’s a tip-of-the-iceberg thing: Yes, on a day-to-day basis you have access to the news and a decades-spanning archive, but you’re not seeing anything close to all of it.

The task (or, more accurately, one of the tasks) for beta620, the Times experimental projects group, is to find a better way to make the newspaper’s information more readily available — both to readers and to the Times itself. Their latest stab at the problem is something they’re calling Deep Dive, a project that aims to give readers a richer, more nuanced understanding of stories.

Deep Dive uses the Times’ massive cache of metadata from stories to go, as the name suggests, deeper into a news event by pulling together related articles. So instead of performing a search yourself within the Times and weeding out off-topic results, Deep Dive would provides readers a collection of stories relating to a topic, based on whatever person, place, event or topic of their choosing. So let’s say you’re interested in protests in Yemen, with Deep Dive you could use an article from nytimes.com as a seed and let the system collect a history of previous items relating to news from the region.

It’s a novel project, still just in demonstration phase — one that aims to let the Times put its extensive archives to better use, but also to create a different experience for consuming news. David Erwin, a software engineer with beta620, said they began building Deep Dive to address problems they were having in following the life of a single news story — and the realization that they were in a position to do something about that. “It kind of came organically from a need that we felt ourselves to better understand what was going on as well as an expression of the potential of in our data and technology,” Erwin said.

What’s interesting about Deep Dive? At least three things:

The use of the Times’ rich metadata

Deep Dive relies on the extensive tagging system the Times uses for all its stories and makes the Times Topics pages possible. As part of the editing flow tags are applied to stories by editors or producers, with suggestions provided by an internal algorithm. Deep Dive looks for connections among topics, so in the case of our Yemen story, it would likely find other stories on protests in the Middle East.

At the moment Deep Dive is limited to topic tags, which are mostly broad terms like “Middle East and North Africa Unrest (2010- )” and “Demonstrations, Protests, and Riots.” That means that that Yemen story connects with stories in on protests in Egypt and Syria, not more stories about what’s going on in Yemen. Erwin said they hope in the future the system could incorporate other factors to make connections through semantic data, editorial data, or time elements. The Times’ metadata is likely the richest of any news organization.

Part of the challenge will be to figure out what level of specificity readers want in a dive — how related they want their stories to be. In other words, does someone “diving” off a story about Italy’s debt crisis want more stories about Italian politics? The Euro crisis in places like Greece or Spain? The state of the global economy? How do you weigh an older story that’s spot-on versus one that’s breaking now but slightly more off-topic?

“What Deep Dive does it brings you some of the relevance of the topic as you’re reading the article itself,” Marc Frons, the Times CTO for digital operations told me. “So you’re immersed in a topic and you’re going further in. You don’t have to leave the page, which I think is very powerful.”

The user interface

What Frons is referring to is Deep Dive’ unique interfact, where the related articles flow into the same frame as the main story when selected. You need never leave the page; jumping backwards or forwards in articles all happens in the same space. That’s a departure from the pageview-driven way most news sites are designed. But Deep Dive’s UI matches its underlying thesis: that individual articles are really pieces of a larger story, told in pieces over time and across bylines and datelines.

Frons said the design is just one possible look for the project and not locked in. Still, the design in some ways aligns with Times Skimmer and Times Reader, which may be why Frons sees a future for it. “To me, this sort of notion you have to click and go to another page and wait for the page to load is something that is going to seem very quaint in the not too distant future,” he said. “It’s probably going to seem like using the international operator to place a call to France.”

Saving your “dives”

More interesting, Deep Dive will also allows users to save their “dives,” which would be constantly updated with new articles. While there are plenty of tools that let a person tailor what news they read online, they’re often based around broad categories (like sports or politics), keyword searches (like Times email alerts), or social networks. What Deep Dives promises is an alert more directly based around a specific developing story — story in the “Arab Spring” sense, not story in the one-specific-article sense. Erwin said the idea is to create a personal news experience that will provide contextual information and be consistently updated. “It basically allows you to keep up with your interests over time by providing custom feeds of articles as they are written,” Erwin said.

But beyond those elements, the real promise of Deep Dive, though, is that it continues to show the Times’ flexibility in providing different ways for different kinds of readers to access its content. For some, the blog-style Times Wire, with its constant reverse-chronological updates, will be perfect. For others, a quick skim of the front page of nytimes.com, with its added editorial judgment about what’s most important at the moment, will work better. Grazers and hunters come in a variety of forms, and the Times is trying to better use the enormous amount of work it generates daily to reach as many of them as they can.

“The reader who wants to dive into a topic and better understand an article in the context of a story isn’t really reading the news in the same was as a reader who is browsing current news,” Erwin said. But the audience that wants to pull back the curtain on a story exists, the Times is seeing it, Frons said. There are heavy users — often subscribers — but also others like researchers and professionals who consume a lot of content from the Times, he said. But the lines between heavy users and casual browsers can often blur, and the reader who stops by the Times for a Newt Gingrich story might wind up looking for more context, Frons said.

“What Deep Dive and some of the other things we’re thinking about do is make it almost easier to serve both people at once,” he said.

It could also be a method of converting those browsers into heavy users — making the grazer a hunter. The ability to save dives — or as Erwin says, “putting a pin in a progression of articles along a certain story” — could be useful when you’re trying to follow a story but then miss the latest developments because you didn’t check nytimes.com on a given day. Another thing beta620 would like to explore is the idea of sharing dives among friends, Erwin said.

It may be some time before Deep Dive is ready for the spotlight, both Frons and Erwin say its likely to stay in the experimental stage as they try and refine the product, especially to make its ability to find commonalities between stories more granular. Frons said it’s too soon to know how something like Deep Dive would be affected by the Times paywall, but ideally it would be available to all readers whether or not they’re subscribers. “Now that we’re a paid site we really want to build things that encourage the subscriber to use the site more and encourage people to subscribe by giving them dynamic and innovative features,” he said.

POSTED     Jan. 23, 2012, noon
Show comments  
Show tags
Join the 15,000 who get the freshest future-of-journalism news in our daily email.
What happened when a college newspaper abandoned its website for Medium and Twitter
At Mt. San Antonio College, they’ve traded in print for distributed publishing, focusing on realtime reporting and distribution: “We’re speaking the language of our generation.”
Prairie news companion: Why The Tulsa Frontier thinks it can succeed with a hard paywall and no ads
Launched by the former publisher of The Tulsa World, The Frontier is betting on a high-subscription-cost model — $30 per month! — to reach a core group of civically engaged locals.
Open-mic journalism: How The Arizona Republic found success with storytelling events
The four-year-old program has helped boost the newspaper’s events business and helped strengthen relationships with the community through nights of storytelling.
What to read next
The State of the News Media 2015: Newspapers ↓, smartphones ↑
The annual omnibus report from Pew outlines a story of continued trends more than radical change.
576The Upshot uses geolocation to push readers deeper into data
The New York Times story changes its text depending on where you’re reading it: “It’s a fine line between a smarter default and being creepy.”
424Knight Foundation invests $1 million in creator-driven podcast collective Radiotopia
The money will help PRX’s collective of public media-minded shows develop sustainable business models and expand with new shows and producers.
These stories are our most popular on Twitter over the past 30 days.
See all our most recent pieces ➚
Encyclo is our encyclopedia of the future of news, chronicling the key players in journalism’s evolution.
Here are a few of the entries you’ll find in Encyclo.   Get the full Encyclo ➚
The Tyee
St. Louis Globe-Democrat
GateHouse Media
Daily Mail
Houston Chronicle
New York