Twitter  Facebook and Storyful are teaming together to highlight news content published on the social network nie.mn/1mDqO3V  
Nieman Journalism Lab
Pushing to the future of journalism — A project of the Nieman Foundation at Harvard

Googling serendipity: How does journalism fare in a world where algorithms trump messy chance?

Twelve years ago, when I was reporting on the pending Microsoft antitrust case, I learned that what was really at stake wasn’t immediately apparent in the legal briefs. It wasn’t the browser market (remember Netscape?) or whether Windows should be able to run somebody else’s word-processing program. Rather, it was how control was exercised over the places where we learned, created, and engaged in critical thought.

One of the best thinkers on the topic was Ben Shneiderman, founding director of the Human-Computer Interaction Lab at the University of Maryland. He told me at the time that the critical question for Microsoft was not whether the company encouraged innovation — it did — but rather how financial pressures dictated which innovations it adopted and which it let wither. The Microsoft software suite, he noted, wasn’t very accessible to people with learning disabilities or those with low incomes.

Fast forward to 2010, and now we hear from Eric Schmidt, CEO of Google, another powerful technology company that controls the tools of creativity and expression. Schmidt recently talked to The Wall Street Journal about the potential for applying artificial intelligence to search, suggesting that the search engine of the future would figure out what we meant rather than find what we actually typed.

Schmidt seems to be pushing the idea that the future — or, more accurately, each of our individual futures, interests, and passions — all can be plotted by algorithm from now until our dying day. The role of serendipity in our lives, he said, “can be calculated now. We can actually produce it electronically.”

Really?

According to Webster’s, serendipity is “the faculty or phenomenon of finding valuable or agreeable things not sought for.” So if the essence of serendipity is chance or fortune or chaos, then by definition, anything that a search engine brings to you, even on spec, isn’t serendipitous.

I don’t know whether Schmidt’s comments should be chalked up to blind ambition or to quant-nerd naivete. But it’s troubling that Schmidt seems to discount the role that human nature plays in our everyday lives and, ultimately, in guiding our relationships with technology.

It might be that Schmidt’s vision for the search engine of the future would serve us well in finding a new restaurant, movie or book. But if Google really wants to take the guesswork out of our lives, we should be asking the same question that Shneiderman put to Microsoft. How might financial pressures shape Google’s “serendipity algorithm”? What content — journalism and otherwise — will it push our way that will shape our worldview? And, to Shneiderman’s point, what limits does it impose?

I think it’s safe to say that some good ideas don’t lend themselves to being monetized online — witness the rise of nonprofit startups in bringing us investigative, public affairs, and explanatory journalism. How might they fare in Schmidt’s world order?

I caught up with Shneiderman on Monday, and he agreed that this is one of the key questions that should be debated as we depend more and more on a “recommender system” in which companies like Google or Amazon use massive databases to anticipate our needs and wants. Public interest groups and other nonprofits that can’t afford the right keywords could be most vulnerable in these systems, Shneiderman said. “How far down the list do the concerns of civic groups get pushed?” he asked.

It’s fair to ask companies what considerations and factors might be weighted in their search formulas, Shneiderman said, but it isn’t clear what level of transparency should be expected. “What is a reasonable a request to make without exposing their algorithm and their business practices?” he said.

I can’t say either. But I do think there are some lessons that Google can take from the history that Microsoft has helped write.

One lesson is that what’s good for the bottom line doesn’t always jibe with what’s best for consumers. A dozen years ago, the Netscape browser was regarded by many as more as more functional, but Microsoft saw it as a threat. So it bundled its own Internet Explorer browser in its operating system and effectively pushed Netscape out of existence.

Another lesson is that it isn’t always possible to divine what people will want in the future based on a profile of what they (or people like them) have wanted it the past. Indeed, some of the most successful technology companies — Google included — have succeeded precisely because their vision for the future was radical, new and compelling. Microsoft once played that role to a monolithic IBM. But today, as Microsoft’s market valuation has been eclipsed by that of Apple, it has become debatable whether Microsoft remains a consumer-driven company.

None of this should be interpreted as an anti-capitalistic rant. We’re all better off for Google’s search box, and it’ll be interesting to see where Schmidt’s vision takes the company.

Rather, it is a suggestion that even the most elaborate algorithms and high-touch e-marketing can’t address every human need.

One of the best vacations I ever took was when I pulled out of my driveway in Raleigh in late August 1991 with no particular destination. Two days later, I found myself in North Dakota, discovering places I never would have appreciated based on my past interests or those of my friends and peers. The experience was so compelling to me precisely because it was serendipitous.

That trip has served as an important reminder to me ever since. When we don’t know what we want, sometimes what we really need is to figure it out for ourselves.

                                   
What to read next
new-york-times-nytimes-building-cc
Ken Doctor    April 24, 2014
CEO Mark Thompson, thus far, is managing the difficult task of boosting circulation revenue despite continued print declines.
  • Pingback: Googling serendipity: How does journalism fare in a world where algorithms trump messy chance? » Nieman Journalism Lab « Mike Cane's xBlog

  • http://www.jason-stevens.com Jason Stevens

    Jim, interesting angle… ‘financial’ pressure will dictate ‘which’ of the ‘innovations’ market leaders such as Google and Microsoft will follow.

    But, If I were to come at this from another POV — especially with regards Google’s’ future ‘mastermind’ algorithm (which is based exclusively on programmatic heuristics) — would be to compare it with Amazon’s “Mechanical Turks” program which seeks to use humans to tip the scales of ‘search’ in the user’s favor.

    I also think Gartner’s recent paper “Get Ready To Swarm” points to another inflection point in which anonymous users will be join together in random and temporary groups to solve a user search issue. It points to a non-Facebook scenario where ‘mobs’ of users are directed at problems – whether they be search or energy issues — and collectively solve the issue in small amounts of time.

    In other words, the algorithm will become a hybrid model and not exclusively programmatic, whether Google realizes this yet or not.

    I think you did tacitly conclude this in one of your final paragraphs:

    “Rather, it is a suggestion that even the most elaborate algorithms and high-touch e-marketing can’t address every human need.”

    Thus, I agree, Google may well be missing out the next biggest game in town: The anonymous online “swarm” of users — That is — you, me and whoever else wants to solve an issue.

    Nice article!
    Jason

  • http://www.trevorbutterworth.com Trevor Butterworth

    The threat is not algorithmic; it’s not knowing how to search for information. The study commissioned by the British library showed that kids who had parents schooled in the old research library habits were much better at using Google as a tool, and that this was the crucial epistemic digital divide. Kids without this typical middle/upper middle class background simply didn’t know that they didn’t know how to search, mine, and rank information found through Google. This suggests that the key informational problem *is* serendipity for this, rather large, group.

  • http://sites.google.com/site/lynheadley Lyn Headley

    Like @Jason Stevens, I feel that you are emphasizing too heavily the dichotomy between humans and code. Just because some outcome or event depends on code or other technologies does not eliminate the role of serendipity (otherwise your car trip would be eliminated too). Serendipity and technology are not mutually exclusive.

  • http://jason-stevens.com jason stevens

    Lyn, point taken. Interestingly, as an example, both Google Goggles and Amazon’s Mech Turks are both generating ‘flawed’ results in terms of photo recognition search currently which I assume ‘electronic serendeptiy’ will remove in the future.
    However, for the time being, in my opinion, human interjection is required to ‘stabilize’ Google search limitations viz. Demand Media content strategy. I suppose this point is probably just stating the obvious!

  • Pingback: This Week in Review: ‘Mosques’ and SEO, Google’s search and social troubles, and a stateless WikiLeaks » Nieman Journalism Lab

  • Pingback: Serendipity Algorithm and What It Means to Marketers « PixelBits

  • Pingback: Algorithms That Rule the Web « Breaking News « Theory Report

  • Daisy Downes

    the ability to articulate the search query is important but even when a user has good search skills, the algorithms may influence the results that are presented based on factors such as previous search history, location, etc which can limit the user’s potential to learn. Serendipity is be part of the answer but greater user control of the search parameters might also be useful for some searchers.