Last August, The Wall Street Journal set a push alert about a story about how alcohol companies are responding to new research that shows consumption of adult beverages might increase the risk of cancer.

The notification could’ve been a fairly staid report on industry trends, but the Journal decided to try and make the story more relatable. “Moderate drinking is under fire and alcohol companies are pushing back,” the alert read, recalled Journal mobile product manager Greg Emerson.

“It is an analysis piece of trends going on in the alcohol industry with implications for alcohol and beverage companies, but we were able in that push to give the news that alcohol companies need to devise new strategies to combat social changes and behavioral changes that lead to people drinking less, and we were able to make that clear but also make it pretty human and pretty relatable,” Emerson said.

With Facebook and Google controlling much of how readers consume news, outlets such as the Journal know that push notifications can be a powerful way to reach their audience directly. To improve its understanding of those alerts, the paper started using a new weighted open rate — one that discounts devices that hadn’t opened the Journal’s app in more than two months. The Journal has also begun A/B testing its alerts, and it’s looking at ways to focus on individual users regardless of what devices they use to access the Journal’s apps.

The alcohol notification was one of the top three most-opened push alerts the Journal has ever sent, Emerson said. In fact, the Journal’s five best-performing notifications were all sent in the second half of 2016, he said. Those alerts would all still be the the top performing notifications using traditional open rates as well, he said. (The top notification? An alert sent the night before Election Day that offered latest predictions on various races. It was “six times (see update below) the average open rate.”)

Since the weighted open rate was implemented last March, the Journal’s average open rate increased by 21 percent, Emerson told me. With the traditional open rate, there was about a half a percentage point difference between the variability of the highest and lowest performing alerts. By factoring out the dormant devices, the variability jumps to about two percentage points.

The Journal wouldn’t share any other data, however, including average open rate, the number of devices with the app downloaded, or the number of dormant devices.

“I realized that if some device hasn’t accessed the Wall Street Journal app in months, then they’ve seen pushes come and go for weeks and weeks and weeks and not interacted with any of them,” Emerson said. “The next push that we send is not going to motivate that group either…Can we target our push strategy at the casual and frequent users of the app, people who regularly visit us, maybe not every day or every week, but have some kind of casual relationship with us? Can we write our pushes better, choose better content for that group, and engage them better?”

Looking at the data from the weighted metric, the Journal re-examined its approach to how it pushed stories that weren’t explicitly breaking news. The data showed that readers appreciated more analytical stories — and as a result the Journal began pushing more of them.

“Using this new metric, we saw that those were some of our most successful pushes every week,” Emerson said. “So we started doing them more and saw that our audience has a robust appetite for the non-breaking news analysis pieces.”

Meanwhile, stories that were more explicitly about business, such as the one on the alcohol industry, typically underperformed. To improve their reach, the Journal worked on rewriting them to make them more accessible. “Straight finance stories were only engaging a small portion of our audience, but because it’s so core to our content offering, we’ve made a particular effort to optimize and refine the language, and do some experiments with that type of content to make it even more engaging,” Emerson said.

Emerson said the paper has conducted upward of 150 A/B tests thus far, and has already begun to see results from tweaking the language of the alerts, though he wouldn’t share specific details.

For instance, every Sunday night the Journal pushes a story that examines what is happening in the economy in the week ahead. That alert was typically one of its lowest performing notifications. Using the testing, the Journal was able to determine that they could improve the open rate to focus on the content of the story and not the fact that it was the weekly economic preview. “Essentially, we’re pushing on the content, not on the content type,” Emerson said.

Similarly, the Journal sends a push alert every morning to its iPad app about the coverage in that day’s paper. By adding the phrase, “good morning,” to each notification, the open rate increased by “100 percent.” (see update below)

“I think it sounds a bit more human…anytime you can signal that there are people behind that push — it is our mobile team, our mobile editor,” that helps, he said.

Across devices, Emerson said the Journal has particularly seen an increase in “mildly active” devices from this strategy, though, again, he declined to provide specifics.

For now, the Journal’s testing has focused exclusively on devices, but its hope is that it will soon be able to target specific users regardless of what device they’re on.

For instance, a user might get the same notification on five different devices, but they only really need to open and read it on one of them. If you look at the open rate, the alert is only being opened one in five times, “but 100 percent of ‘me’ engaged with that push,” Emerson said.

“That will allow us to get more granular in terms of seeing how people respond rather than devices respond,” Emerson said. “We can see that being a big advantage for targeting different types of push for time of day or day of the week.”

Update: As Felix Salmon noted, Emerson initially got a number wrong when he characterized the impact of a couple changes on open rates. The most-opened notification actually had six times the average open rate, not “six orders of magnitude above the average open rate.” Adding the phrase “good morning” to alerts increased the open rate by “100 percent,” not “an order of magnitude.