For a business that not that long ago had basically zero data on how consumers used its products, the news industry has become obsessed with metrics. We’re seeing a bit of a swing back to the middle of the spectrum, where analytics are now just one piece of a larger puzzle, but we still haven’t mastered how to talk about the value of news alerts in our readers’ most intimate spaces: inboxes, lock screens, and desktop browsers.

With traditional email newsletters, we can look at open rates (how many subscribers opened a particular newsletter) and clickthrough rates (how many people clicked a link) to see how readers are engaging with our products. But that’s harder with breaking news email alerts, where the subject line might contain everything the reader needs to know. They act more like a push notification: It might not need to be opened for readers to appreciate it landing in their inboxes.

The value of an email send, push notification, or browser alert goes beyond its open rate, and talking about that value needs to become more mainstream in our thinking of how all of our notification products work together.

Research from 2017 divided push alerts into four categories: headline, teaser, round-up, and additional context. More than half of the alerts sent in that study were based on breaking news, and it found that the majority provided additional context that went beyond a straight headline.

In a news environment where the day’s fifth-most-important story would have been an all-hands-on-deck affair just a few years ago, news consumers have to choose what they’ll read deeply and what they’ll just be superficially aware of. The reader might not need more than what happened and a slight bit of additional context. So the fact that a push alert didn’t get opened doesn’t mean it wasn’t valuable to the reader.

Push alerts show up in spaces where the interruption is hard to ignore: your phone’s locked screen while you’re trying to fall asleep, your smartwatch while you’re in a meeting, a popup while you’re answering an email. They drain your phone’s battery. Long story short: If someone doesn’t want to receive a push alert, they’ll change their settings. An underrated metric in measuring an alert strategy’s success is simply the number of subscribers a push notification list has. Editors can also look at the lifecycle of an alert subscriber: How long do they stay subscribed? How often do they change their settings?

And now that more news organizations are adopting browser notifications, the industry has to be smarter about how all of these interruptions work together. It can be tempting to push the same alert through all available outlets (or to auto-subscribe new members into these various channels), but we have to be careful to not bombard readers on every platform.

Some newsrooms have advanced tools that allows readers to be explicit in how they want to receive different kinds of news. But for newsrooms without those resources, it can be helpful to be as strategic and thoughtful with sending notifications as you are with editing stories. Just as a story should be only as long as it is interesting, and just as every story should serve the reader in some way, we need to be judicious with our push alerts and think about how pushing on different platforms can be treated like a story package, where the components are complementary to one another.