Twitter  How Mashable went from being a tech blog to putting a former New York Times editor in charge of its news desk nie.mn/1nxFQLN  
Nieman Journalism Lab
Pushing to the future of journalism — A project of the Nieman Foundation at Harvard

What would it take to build a true “serendipity-maker”?

What if we created a “ChatRoulette for news” that generated content we tended to disagree with — but was also targeted toward our regular levels and sources of news consumption? How hard would it be?

For the last 24 hours or so, the Twitter-sphere has been buzzing over Daniel Vydra‘s “serendipity maker,” an off-the-cuff Python hack that draws on the APIs of the Guardian, New York Times, and Australian Broadcasting Corp. in order to create a series of “news roulettes.” In sum, hit a button and you’ll get taken to a totally random New York Times, Guardian, or ABC News story. As the Guardian noted on its technology blog, “the idea came out of a joking remark by Chris Thorpe yesterday in a Guardian presentation by Clay Shirky that what we really need is a ‘Chatroulette for news’”:

After all, we do have loads of interesting content: but the trouble with the way that one tends to trawl the net, and especially newspapers, simply puts paid to the sort of serendipitous discovery of news that the paper form enables by its juxtaposition of possibly unrelated — but potentially important — subjects.

This relates to the much-debated theoretical issue of “news serendipity,” summarized here by Mathew Ingram. In essence, the argument goes that while there is more news on the web, our perspectives on the news are narrower because we only browse the sites we already agree with, or know we already like, or care about. In newspapers, however, we “stumbled upon” (yes, pun intended) things we didn’t care about, or didn’t agree with, in the physical act of turning the page.

As Ryan Sholin has been pointing out all morning on Twitter, the idea of a “serendipity maker” for the web isn’t entirely new. And I don’t know if the current news roulettes really solve the problem journalism theorists are concerned about. So I’d like to know: What would it take to create a news serendipity maker that automatically knew and “factored in” your news consumption patters, but then showed you web content that was the opposite of what you normally consumed?

For example, I’m naturally hostile to the Tea Party as a political organization. What if someone created a roulette that automatically generated news content sympathetic to the Tea Party? And what if they found a way to key it to my news consumption patterns even more strongly, i.e., if somehow the roulette knew I was a regular New York Times reader and would pick Tea Party friendly articles written either by the Times or outlets like the Times (rather than, say, random angry blog posts?)

I think this is interesting, because it would basically hack the entire logic of the web. The beauty of the web is that it can direct you towards ever more finely grained content which is exactly what you want to read. It would somehow know what you wanted even before you did. In other words, it might be the opposite of what Mark S. Luckie called “a Pandora for news.” And it would solve a very real social problem — or at least a highly theorized social problem — what Cass Sunstein calls the drift towards a “Daily Me” or “Daily We,” where we only read news content we already agree with, and our political culture suffers as a result.

So. This is a shout out for news hackers, developers, and others to weigh in: How hard would it be to create a machine like this? How would you do it? Would you do it? I would really like to write a longer post on this, based on your replies. So feel free to chime in in the comments section, or email me directly with your thoughts. I’d like to include them in my next post.

                                   
What to read next
huge-cc
Ken Doctor    July 16, 2014
Rupert Murdoch wants to add Time Warner to his stable of media properties. Seemingly everyone’s looking for an acquisition. What is it about this moment that is pushing everyone to get bigger?
  • http://twitter.com/susannahvila Susannah Vila

    The problem is that the moment you start pushing an “eat your vegetables” type of application, in which you are encouraging people to consume that which they by definition won’t enjoy, they’ll probably stop using your tool. Better to keep it random.

  • http://ryansholin.com Ryan Sholin

    Let’s go down this path for a moment.

    Buttons on the toolbar = Thumbs Up, Thumbs Down, Random, and “Make Me Uncomfortable.” How does the system measure what makes you uncomfortable? Well, it can’t just be based on everything you dislike. It’s going to need more data than that.

    Maybe instead of “uncomfortable” the action is “show me the opposite of this,” but that still assumes more variables than just thumbs will provide.

    Maybe Facebook has enough data on us to build this? Who else?

  • http://www.niemanlab.org/ Joshua Benton

    I remain a big skeptic on the idea that the Internet is a serendipity inhibitor. I think it’s the greatest serendipity machine known to man. Newspapers and broadcast media were pretty good at representing the world from the 45th to the 55th percentile (plus an added filter of upper-middle-class whiteness). My quotient of newly learned, previously unfamiliar information each day is, I dunno, a thousand times higher than it was when I was just reading a mediocre Gannett paper and watching Dan Rather at night?

    My other concern is that serendipity debates tend to be reduced down to left-right debates. It’s “liberals should watch Fox” and “conservatives should read The Nation.” I think that artificial dualism is one of the least appealing tropes of traditional journalism — the idea that there are two sides to be represented in stories, or that the world can be nearly broken into “Tea Party friendly” and “Tea Party unfriendly” stories. I’d argue that true serendipity comes from things that don’t fall neatly into that rubric at all. The world isn’t just about politics. If you read a lot of Dem-loving blogs and a lot of GOP-loving blogs, your news diet is still radically oversaturated with white college-educated DC men with bad hair and Model UN trophies back home.

  • http://www.ramanarao.com Ramana Rao

    The “beauty of the web” may not be exactly that it “directs you to ever more finely grained content which is exactly what you want to read.” The web is based on human navigation as much as machine intelligence and by getting us to the right places with a coherence on their topics and themes we learn what our interests are, how to articulate them better, and best of all, ongoing navigation works even as our interests evolve.

    If you include a variety of sources and filter topics based on current best search technology (no where near close to “somehow knows what you wanted”), you will naturally get serendipity and plenty of context in the content that matches. So it’s not as easy as you might think to isolate yourself i.e. I agree with your adjusted statement of “highly theorized social problem.”

    We think we’ve created a reasonable Pandora for News experience in our current service at http://www.icurrent.com. And it doesn’t seem creating insularity or losing serendipity will be problems.

  • Kate Fink

    I agree with Susannah that it should be more random… better to read with an open mind rather than one that’s already thinking, “I’m going to hate this.” Also, I agree that the idea of finding an “opposite” story is artificial… and I’m worried this kind of model would only appeal to people who see stories as just two-sided.

    I like the Guardian model in that it’s random but also limited (by what’s in that paper). I think a ChatRoulette for news would be great if it would allow users to set some parameters (maybe dates, subjects, places?) that are specific enough to generate interest, but still broad enough to allow for surprise.

  • http://windycitizen.com Brad Flora

    I question the premise that ChatRoulette is offering content that people “tend to disagree with.”

    ChatRoulette is offering an experience that millions of people want, the chance to run into someone whose views or actions you cannot publicly endorse despite being very interested in them. ChatRoulette’s randomness takes all the blame off your shoulders. If you land on a naked dude, you can blame ChatRoulette, not yourself, but rest assured, you knew there were naked dudes on ChatRoulette.

    It’s also a lot of fun and something to do with friends.

    I don’t see how this translates to news. To echo Susannah, above, this sounds kind of boring and “eat your vegetables-y.”

  • http://flavors.me/howard Howard Weaver

    There may be value in a random selection of news, entertainment value perhaps. But it certainly is not the same value people describe when speaking of the serendipity of news.

    Serendipity ≠ random.

    For serendipitous news to have value it needs to delight and engage readers with information that otherwise wouldn’t have encountered. That takes more than algorithmic randomness.

  • http://www.columbia.edu/~bjp2108/blog/ Ben Peters

    Fascinating idea. Suppose this opposition news tool were to exist (the software core is no doubt close to what operates data visualization software), and that it were made available for wide distribution, what interests me would be this question: Which news consumer groups would tend to use it most? Who is most comfortable with opposition?

    My hunch is that what we would find is also my own critical comment on the project: my guess is no one political leaning would be better suited than another. Moreover that binary oppositions would start to break down through a meta-level analysis of what attitudes we harbor toward opposition news.

    To the point, namely that there are, obviously, multiple parties and perspectives at play around every news event. I am not sure an opposition news tool is really what we want. We may want a serendipity news tool that exposes the reader not only to the opposite viewpoint, which may be inferred anyways, but to wholly fresh, third, fourth, or fifth points of view. Two-party oppositions do not recognize the Archimedean point from which the opposition itself begins to make sense.

    Two cheers for deliberate serendipity, and only one for seeing only the other side. From a technical point of view, neither seem in the slightest impossible. If anything, in fact, “not what I expect” is probably much easier to produce than “the opposite of what I want.”

    Hope to hear more about this. Very promising kernel.

  • http://quietbabylon.com/ Tim Maly

    What you want in the serendipity engine is something that gives you a mixture of high quality content that you agree with and high quality content that you are likely to find challenging and new. But in both cases we want it to be high quality.

    So we want a system that builds a profile of you and what you like a-la SumbleUpon but also builds a profile of who your opposite is. And then every X links it drops something that your nemesis might really enjoy. Your nemesis is not your opposite, just someone like you in most ways but very different in a few critical (in this case political) ways.

    Another option would be a system like that described in the UltraGleeper paper: http://www.crummy.com/software/UltraGleeper/IntroPaper.html

    Specifically the Indie Rock Peter Principle that’s described where certain attributes when they pass a threshold of agreement begin to decrease the score of something. In the paper it’s used to avoid showing you obvious links but it could similarly be used to avoid preaching to the choir.

  • Pingback: What Would it Take to Build a True “Serendipity-Maker”? | C.W. Anderson | Voices | AllThingsD

  • http://iduncan.tumblr.com Ian Duncan

    What about Google Reader’s play function? I’ve been messing about using that for a couple of days and it is quite fun. I assume it works by analysing the feeds you’re already subscribed to and working out what other people who subscribe to those feeds also read and taking bits and pieces from that.

    But you can also create bundles of feeds on Google Reader to subscribe to with a single click. So what if you get your conservative/old/different in whatever way friend to make a bundle and run it through the Play filter?

    Perhaps a bit long winded but it might work quite well.

  • http://sethlewis.org/ Seth Lewis

    Great post, Chris … and interesting ideas all around!

    Perhaps one way to look at this is to pose the question: Can the Web be semantic and serendipitous at the same time?

    That is, on the one hand we want a system that knows us: our interests, our browsing history, our likes and dislikes. But on the other, the engine has to be unpredictable enough that it gives us a sprinkling of *good material* that we might not otherwise have found. (I put emphasis on “good” because, as Howard Owens pointed out above, randomness does not equal “news serendipity,” which in the traditional sense implies some kind of vetting for quality, making the newspaper, for example, a serendipitous assortment of reasonably good stuff.)

    As others have mentioned, I have problems with the false binaries associated with “give me the opposite” kind of serendipitous news. But, at the same time, what I *would* like to see is a function for getting stories that really explain where other sides are coming from on an issue. Take the Tea Party, for example. If I’m anti-Tea Party, don’t just give me a story that’s sympathetic to its anti-government rhetoric. Instead, give me an “explainer” that helps me come to understand *why* Tea Partiers feel the way they do — not simply the *what* that is usually what you find in news stories (e.g., images of protest, soundbite claims, etc.).

    As Matt Thompson’s SXSW panel talked about, the Web can do so much more in giving us the crucial context to make sense of our world. Why not build a system that does both — that is explanatory and “multiperspectival” (as Gans put it)? Rather than just get the headline version of “the other side(s)” to the story, let’s see if we can’t gain a better appreciation in the process about where they’re coming from.

    We can imagine, anyway.

  • http://jonathanstray.com Jonathan Stray

    “Eat your vegetables” is obnoxious. I’d rather simply know what other points of view are out there. Where are they, what are they, how many people are there?

    This is why I want to make maps of global news output: real-time visualizations of every article recently produced, clustered by links or co-consumption. Then mark the articles that I’ve read in a different color — a “you are here” marker.

    I bet most people would see that they never leave their home town in information space. Call it a curiosity generator, just like a map of the world.

    I developed this idea in some detail in a 2009 post, “Mapping the Daily Me.”

  • Pingback: Serendipity is unexpected relevance « BuzzMachine

  • Pingback: Terry Heaton’s PoMo Blog » Blog Archive » The matter of serendipity

  • Pingback: This Week in Review: The iPad’s skeptics, Murdoch’s first paywall move and a ‘Chatroulette for news’ » Nieman Journalism Lab

  • Pingback: ChatRoulette for News « The Media Mash-Up

  • Pingback: Pigsaw Blog » Blog Archive » Bookmarks for 7 Apr 2010

  • Pingback: Navigation headaches

  • Pingback: This Week in Review: The iPad’s skeptics, Murdoch’s first paywall move and a ‘Chatroulette for news’ | Mark Coddington

  • Pingback: Google News experiments with human control, promotes a new serendipity with Editors’ Pick » Nieman Journalism Lab

  • Pingback: “Blood libel”: How language evolves and spreads within online worlds » Nieman Journalism Lab » Pushing to the Future of Journalism

  • Pingback: Thoughts on Discourse and Technology | Josh Braun's Blog

  • Mandiquita

    Now there’s a new Chatroulette 2.0, With a lot of options to have fun.