If you’re an Apple Watch owner, one of the first things you probably had to do was fiddle with the notification settings on the device to avoid your wrist getting buzzed (excuse me, gently tapped) constantly. A phone vibrating in your bag might be annoying, but a buzz on the wrist is more invasive.

Wearables add a new level to the challenges that news organizations face when they decide on notification strategies for their apps. And it’s easy to mess up. Think of the progression of the way that we receive news updates online — from the portals of the 1990s, to the Twitter and Facebook feeds of the 2000s, to the desktop and mobile notifications of today. “We are much more likely to forgive some of the territorial mistakes that someone might make on a portal or a feed,” said Matt Boggie, executive director of The New York Times R&D Lab, at the NYC Media Lab’s “Future of Notifications” panel Wednesday.

So far, publishers seem to be using much the same notification strategies for wearables that they did for mobile devices, Boggie said. And that’s a recipe for irritating users. “By the time you’ve decided to allow an app to send you a notification, you are a loyal user, and every notification is an opportunity for that app to screw up and annoy you.”

What's fun about the Apple Watch is that there's so many ways to dismiss notifications and only some methods work some of the times!

— Craig Mod (@craigmod) June 8, 2015

It’s further complicated because of concerns over the privacy of personal data. “It’s incredibly unwieldy to hand-curate all the potential combinations of what you want,” said Steven Levy, editor-in-chief of Medium’s Backchannel, who recently wrote about some of these issues. “To do it for every app is incredibly burdensome. The danger is that in order to have some control over the level of what’s being notified, we’re going to turn it over to these big artificial intelligence powers [Google, Apple, and so on]. That means giving up some privacy.”

Panelists were split on how to make notifications better. “Maybe there should be a charge,” said Dana Karwas, a lecturer in integrated digital media at the NYU Polytechnic School of Engineering. “Just like we can’t call the president and get a direct line, maybe it should cost the content providers just a little bit to take our time.” But Otto Toth, CTO at The Huffington Post, argued that’s nuts when people also want and expect quality content for free.

Matt Hartman, director of seed investments at Betaworks, suggested that apps could request access to users’ calendars and promise not to send them notifications when, say, they’re in meetings. But when he asked how many people in the audience would grant such access, only one person raised her hand.

“Notifications are really annoying,” said Noah Chestnut, product lead on the just-released BuzzFeed news app. He cited a recent Urban Airship report that found that, in 2013, about 5 out of 10 people installing a news app agreed to receive notifications. In 2014, that number was down to 4 out of 10. “People are not even saying they’ll give you a chance to win them over. They’re saying no.”

News organizations figuring out the notification systems for their apps have two main problems to deal with, Chestnut said. First, “we have to deal with the baggage of all the other apps and products we had no hand in building.” People who’ve been burned by notifications on other apps are less likely to turn them on for your app, no matter how carefully you may have thought them out. And speaking of how carefully you have to think them out: “We have to be smarter about what the app actually does when we make our pitch to say, hey, do you want notifications? How does context matter?” For example, are you asked to turn on notifications immediately upon installing the app, or after you’ve used it for a while?

Publishers also need new metrics to figure out whether their notifications are working. “Clickthrough rate is a weird metric for a lot of notifications,” Chestnut pointed out. “When I get a headline from the Breaking News app, or through Circa, I don’t need to read the story. I’m good to go. Yet the only metrics we have are: Did we send it? Did somebody open it? Did somebody uninstall the app? There should be more. We need to do a better job of figuring out what an impression is for a notification, and how to ascribe value to that.”