Nieman Foundation at Harvard
HOME
          
LATEST STORY
Block Club Chicago offered two versions of the same breaking news story — with and without a horrifying video
ABOUT                    SUBSCRIBE
Dec. 3, 2020, 9:23 a.m.

How The New York Times prepared for the ultimate stress test — the 2020 election

Senior vice president of product engineering Brian Hamman describes prepping for unusual results, multiple needles, and vote counting that stretched for days. “Essentially, any time after the election, we could be sending out a push notification calling the election and bringing massive traffic to our site.”

“It’s like you’re on the space shuttle and you hear what sounds like an air leak. You’re like, ‘Uh, what was that noise?'”

That’s one of the ways The New York Times’ senior vice president of product engineering Brian Hamman described preparing the organization’s systems for the 2020 election. The Times prepped by intentionally breaking the site to identify problem areas, simulating unusual (but not impossible) results, and forming a multi-team “Election Readiness” project.

All that preparation was not in vain; 237 million readers visited the site that week, including 120 million on Nov. 4th alone. By the time former Vice President Joe Biden was declared the winner, The Times had four record days — Nov. 3-6 — with more than twice the traffic of prior election days.

There’s nothing quite like election night for a newsroom. Hamman said the closest comparison would be Super Tuesday. His team uses the primary’s traffic as a baseline to help predict what they can expect to see during the general election but the final numbers wound up beating their expectations.

The contentious and closely-watched presidential race also arrived a time of increasing complexity across multiple platforms at the Times. I talked with Hamman about the preparation process that kicked off more than a year ago. The conversation has been lightly edited and condensed.

Sarah Scire: Tell me about these stress tests.

Brian Hamman: We have an internal library of weird calls that we know are possible but unusual. There’s a team in the newsroom — largely the interactive news team and graphics — that does a ton of simulations on that side. On the traffic side, we basically pushed what we expected to be our election traffic against our live site three or four times leading up to the elections. We first did it without scaling up our system in order to deliberately break our site so we knew where the breaking points were and to learn what changes we needed to make. And then did it again, scaling up the system.

The last thing was more of a paper exercise where we essentially wrote out a bunch of scenarios and put them together in what we call a “run book.” If a certain system fails, how are we going to respond to that? If we need to get the print paper out at a certain time and a system is struggling, how will we do that? It allows us to do the hard thinking ahead of time so when things are happening and we have to act quickly we can follow those steps.

Scire: What did you notice on election night and the following days, in terms of traffic and what readers were looking for? Any surprises?

Hamman: It was not surprising what people were looking at. That tends to follow very similar patterns — they’re generally on the homepage and then click on election results. This year, we did a lot of work on our live updates, which are new here, and we did see a lot of traffic to that. In terms of the amount of traffic, we targeted all of our systems for a level of scale that we thought was the maximum we could possibly achieve and didn’t actually expect to hit that. And then we did almost exactly what we had designed all of our testing around.

Scire: Not every newsroom has the same resources as the Times. What advice would you give to someone running a smaller team? What would you prioritize?

Hamman: I would definitely say start early. If your systems aren’t able to scale or if you’re worried about reliability, the work that you need to do to fix that is generally not work that you can do overnight. It’s the kind of work that can take months. So if you identify something in October, you’re not going to be able to fix it by November. And then I would focus on where you have the most risk and where you have the most leverage to fix things. For us, this year, we asked “What are the most important things we do on election night?” It’s our results, the homepage, publishing articles, and pushing out alerts. We focused on those areas only.

For smaller organizations, just try not to overcomplicate it. Try to build things — to begin with — as simple as possible. So that it is easier, on election night, to act and react quickly. And I’ll say, having been at the Times for a long time, we have certainly gotten much more complex, as our results have gotten much more complex and our products have become much more complex and multi-platform. We used to have a much simpler setup.

Scire: You sound a little wistful.

Hamman: Yeah. [Laughs.] It’s definitely gotten harder to pull this off. But the result of that is we can create the richest election coverage that we’ve been able to provide our readers by far, in terms of the integration with our push notification and real time updates across multiple platforms. We’ve never been able to pull off something that sophisticated before. Our homepage was programmed in real time on election night, meaning that we could change the stories and layouts out the homepage quickly and using the standard tools within our CMS. This allowed us to, for example, completely flip the layout of the homepage when we called the race, which is not something we have been able to do previously.

Usually when we call a race, it’s the print paper that people are taking photos of and sharing around — and this year, it was the website and the app. That’s something the team was incredibly proud of.

Scire: This election was unusual for a bunch of different reasons, including the expectation that the president would contest the results and that, because of the pandemic, an unusually high number of votes would be cast by mail. Did either of those factor into the preparedness plans?

Hamman: Yes, absolutely. Usually elections are a two-day affair. You start the day of the election and you have a large day the day after the election, too, particularly the morning after when all the people who fall asleep before anything is called wake up and are eager to see what’s happening. We design our coverage around that 24- to 48-hour period. Usually, that means all the key engineers are sitting at their computers on a Google Hangout, on Slack, in direct communication so that if anything happens, they can react very quickly — what we call “eyes on glass” coverage. With this election, that was challenging, because that period could go on for weeks. Essentially, any time after the election, we could be sending out a push notification calling the election and bringing massive traffic to our site. We couldn’t just have roughly 100 engineers sitting at their computers for weeks on end. We had to come up with a more complicated tiered plan that scaled back after a few days.

On election day too, the needles depended on data that throughout election night became more or less relevant. As the states that we built the needle to focus on became a little less central to the story, we had to adjust to focus on different parts of the map and different parts of the story and less on the needle itself. That required us to react pretty quickly on election night to move things around on our homepage and in different packages. It’s something that would have been challenging for us to do in prior years, but because we’ve built a much more sophisticated homepage curation system — that we call Curator — we were able to do it quickly.

Scire: A while back, former New York Times CEO Mark Thompson said “the single biggest reason” behind the Times’ success was the decision to give more autonomy to teams working on the publication’s various digital products. Can you tell me more about experimentation? Is an election the right time to try new things?

Hamman: I could point to a couple of things that were essential to election night that were the result of tinkering throughout the last year. The way we pull people through stories — called Storylines — is something our engagement team built and has been experimenting with in our coronavirus coverage. It’s for any story with an arc that’s larger than a single article. The homepage was the first real test of that new curation platform. All of these things we’ve been developing over the past year and then pulled together on election night.

It’s always exciting to see them all come together. Election night and the Olympics are the two places where we love to just really test our systems and push them as far as we can.

POSTED     Dec. 3, 2020, 9:23 a.m.
SHARE THIS STORY
   
 
Join the 50,000 who get the freshest future-of-journalism news in our daily email.
Block Club Chicago offered two versions of the same breaking news story — with and without a horrifying video
Readers told the nonprofit local newsroom that they appreciated the option to read an article omitting graphic video and images of 13-year-old Adam Toledo’s death.
Substack will spend $1 million to support “up to 30” local news writers
“This is not a grants program, nor is it inspired by philanthropic intent.”
Would you pay $34.99 a month to get news from Reuters.com? That’s their hope
Who deems Reuters.com so essential that they’ll pay more than two Netflixes a month for it?