Over the past several months, the Guardian Mobile Innovation Lab has introduced a new format that we have been working on: the Smarticle. We’ve now run three Smarticle experiments, and learned a lot about how readers like the format and what topics work well in it. Most significantly, we learned that there’s an appetite for this story format: Through a combination of written feedback from users and analytics, we learned that people enjoyed consuming news in this way, and they were comfortable only receiving elements of the story that were new to them.

First, a quick description for those unfamiliar with the project: the Smarticle is a story format designed for mobile that aims to meet readers where they are in their knowledge of a developing story by only presenting them with the elements that are most useful to them. It works like this:

• A story is broken down into its core elements, which are displayed as a series of blocks.
• When a reader first visits the page, the blocks contain only the most fundamental story details up to that point.
• As the reader returns, an algorithm determines what should appear for each user. For these first three experiments, the algorithm took into account what someone had read on their last visit, as well as the importance of each new story development.
• Users who read the story on a Chrome browser have the option to subscribe to notifications, which are sent when there is a critical mass of updates or a major development in the story.

By running repeated experiments, we’re looking to learn how we can help readers follow an evolving story, using an article format that aims to keep them informed while also respecting their time. The format also tests how comfortable readers are consuming news that isn’t written or presented as a traditional article. While we will keep adjusting and testing the format beyond these three experiments, we wanted to report on what we’ve learned so far.

A few views of the Smarticle coverage of the Republican tax bill.

Since our initial post, we’ve run Smarticles on three different topics: The first on Trump’s dispute with a Gold Star widow, the second on the sexual harassment scandal surrounding Roy Moore, then the Republican candidate for senate in the run-up to a special election in Alabama, and the third on the passage of the Republican tax bill.

These three stories shared some core characteristics that we felt made them apt candidates for experimenting with in the Smarticle format. Here are a few:

• Stories that develop over time. First and foremost, they were stories that we expected would include significant developments over at least a few days. We decided the ideal time for experimenting was three days — long enough to gain a good number of readers, but short enough so that our small team (an editor, a reporter, a designer/engineer and a product manager) could reasonably incorporate all the important updates. So we chose stories that were likely to keep going for at least that long.
• Topics covered by The Guardian. Our experiments also relied on The Guardian’s coverage, so we needed to be reasonably sure the Guardian US newsroom would be covering the chosen topic. (We also drew content from social media and other sources as well as Guardian stories) The stories we chose seemed to have momentum, and we consulted with the news desk to see if the coverage would continue before launching a Smarticle.
• Stories with a clear arc. The stories also developed in an incremental nature that suited the format well. While there were competing narratives within the topic, each story followed a distinct arc punctuated by key events, such as statements from core figures or votes in Congress.
• Stories with repeated context. Finally, each article written on these topics contained a paragraph or two of context that was repeated in each piece. In the Smarticle, reporters only have to write it once and readers who are already aware of it do not need to re-read it.

Each test ran for about three days. The Lab wrote and edited the story elements, largely drawing from Guardian US coverage, while also including other outlets’ stories as well as social media posts.

Smarticles ran alongside Guardian US coverage of the same topic, with links to the Smarticles were included within those Guardian articles offering an experimental version designed for mobile.

Although Smarticles are designed for mobile, they’re available on any web browser, on any device. The full experience is optimized for users of Chrome browsers on Android devices, who were additionally able to sign up for notifications, which we sent to subscribers to alert them to major updates in the story.

The response

As with all lab experiments, we sent out a survey to our alert subscribers after each test. These surveys give us a chance to learn a great deal from the thoughtful feedback from users. The questions we ask are meant to gauge how interesting and useful these new formats are to those who use them, and include questions about specific elements of the format (a design feature, or type of content, for example), as well as open-ended fields to invite reactions to aspects we have not thought to ask about. Pairing the feedback from these survey responses with engagement data from our analytics suite allows us to get the full picture of insights about how each Smarticle was used and received by readers.

Part of the Smarticle on the sexual harassment scandal involving Roy Moore.

The data from the three Smarticle tests consistently indicates a strong affinity for this story format, and perhaps even a need for it. Many readers who responded to the survey say they found the format a more useful way to follow stories than through reading multiple articles, and the majority of survey respondents also said they would like to follow other stories in this format.

More on survey results

About 15–25 percent of people who received the survey responded with feedback. On average, 92 percent of survey respondents told us they found Smarticles as useful or more useful than following the same topic by reading articles. Each topic that we covered in the Smarticle format ran in parallel to Guardian coverage of the same story, ensuring an obvious comparison. We worked with the Guardian US production team to include links directly to the experimental Smarticle format below the headline of most Guardian articles on the topic. These survey results indicated that readers were comfortable following the topic in this one-page format, where they would usually read a series of articles on the topic, each on its own page.

Among survey respondents who provided feedback on the Roy Moore and Republican Tax Bill Smarticles, 69 percent said they would like to follow other stories in this format; another indicator of the appetite that readers had for the format. (We had not asked this question in the survey for the first Smarticle, on Trump’s response to a Gold Star widow.)

Beyond that, an average of 63 percent of survey respondents were satisfied with what they found on the page each time they returned to the Smarticle. This was an important indicator that most users were comfortable with the layout and page design, and were also happy with both the amount of information and the quality of story updates that were shown to them on each visit.

An average of 77 percent of respondents said that only seeing what was new to them was useful. Readers who experienced the feature were comfortable with the idea that elements of the story they had already seen would disappear, which should save them time.

Readers also left comments in response to an open-ended question (“Anything else you would like to tell us about this experiment?”) that indicated their interest in this format. “So far I like the format — it’s nice to see only the new information and not what you’ve already read,” one reader wrote. “I like the new feature because it bundles together many articles on a topic, making it possible to explore from many angles,” another said in the comments.

People also stated they would like to see Smarticles on different topics, “Would like to try it again, especially with something I was very interested in.”

A few more reactions:

Great idea. Keep innovating with clarity and imagination !

If you can implement this well, then I would become a contributing supporter. This is the kind of innovation that we need to filter through an ever changing/evolving news cycle!

Limitations of the experiments

For all the positive feedback we received, which we appreciate as validation of the premise of Smarticles, the experiments we’ve conducted have had some known limitations.

For example, users only had one way to discover the Smarticles: through links on Guardian articles about the topic. Since people normally don’t revisit articles, this meant that one of the easiest ways for users to return to the Smarticle was through the notifications we sent. Since only Chrome users on Android devices or desktop were able to sign up for notifications, there was a built-in limit to the number of users who were able to experience all of that feature.

We also had conversations about the best way to handle chronology in the presentation of updates. Generally, Smarticle content is laid out in chronological order, with the oldest information at the top. Users who come into a story, then, will find the latest update buried at the bottom. We received some user feedback from our first two experiments that this felt counterintuitive, so we addressed this by ensuring that users who came in from a notification saw the update tied to the notification first at the top of the page.

Audience size

The size of the audience for each of our experiments tended to directly correlate to the readership on the Guardian site. Overall, the audience for the set of Smarticles ranged from about 1,200 readers to nearly 12,000. A smaller set of those readers visiting on Chrome browsers were eligible to sign-up for notifications, which ended up being between about 4 percent and 11 percent of readers who had the option to subscribe.

Here are some more detailed notes on the audience size for each experiment:

Pageviews:

• Trump-Gold Star: 13,013
• Roy Moore: 2,773
• Republican Tax Bill: 8,243

Users:

• Trump-Gold Star: 11,295
• Roy Moore: 1,214
• Republican Tax Bill: 6,685

Notification signups:

• Trump-Gold Star: 178 (3.5 percent of eligible Chrome users)
• Roy Moore: 120 (11 percent of eligible Chrome users)
• Republican Tax Bill: 160 (4 percent of eligible Chrome users)

Moving forward…

While we’ve learned a great deal from the experimentation we’ve done so far, there is still a lot to explore. Now that we’ve established readers are comfortable with elements of the story disappearing based on whether or not they’ve been seen, the main question that we are exploring next is: When is the right time to resurface information reader’s have already read, if at all?

To examine this, we will be collecting and analyzing a number of signals from readers of the next Smarticles, to try to determine at what point they may need reminding of, or simply want access to, certain elements of the story. Deciding what elements to resurface through the algorithm and when to bring them back will be a major focus of our future experiments.

It will work roughly like this:

1. Each element of the story is assigned a rank relative to its overall importance to the story.
2. The algorithm will take into account the assigned rank of each story detail along with elements of the user’s behavior, such as the time since their last visit, how long they viewed each detail, and the number of details that have been published.
3. A combination of these factors will decide what elements readers will see each time they return to the page.

We’re excited to learn how readers respond to this and we’ll keep you posted. Stay tuned!