Information is everywhere — in the world, in your home, everywhere. In today’s pair of videos from my visit to The New York Times Co.’s R&D Lab, Brian House, The Times Co.’s Creative Technologist for R&D, demonstrates the Lab’s, er, reflection of that idea — in the form of a data-bearing mirror. The device (working name: “the magic mirror”) uses Microsoft’s Kinect motion-sensing technology to read physical cues from its user; it uses voice recognition technology to detect verbal cues. (In the videos, you’ll hear House talk to the mirror, Snow White-style.) The mirror also uses the the Times’ powerful APIs to serve up information on-demand.

The device, within its notional home, would replace the standard bathroom mirror. And like the R&D Lab’s screen-topped table, it’s all about bringing a new kind of intimacy to the news experience. You can use it, say, to browse Times headlines, or watch Times videos, while you’re brushing your teeth. You can use it to schedule events on your personal calendar, or to shop online, or to exchange messages — from the classic “buy milk” on up — with other members of your household. While the mirror is capable of serving (relatively) traditional forms of content — individual articles, videos, etc. — via its screen functionality, even more striking is its experimentation with information that has, directly, very little to do with the Times itself. In exploring the realms of health and commerce alongside more standard editorial content, the Times Co. is hinting at the products we might see when news organizations expand their scope beyond the news itself.

Essentially, the mirror fuses news — and, in this case, a highly branded, New York Times experience of the news — with all the other forms of data that we encounter in our daily lives. Again, the “information shadow” idea. By building a device that is both a screen and a mirror, the R&D Lab can experiment with the ways to combine the personal and the informational in ways that (it hopes!) aren’t intrusive, but rather helpful and, in that, welcome. This is The New York Times Company acting not just as a curator of information about the wider world, but also as a curator of the information that punctuates, and complicates, and in some sense defines its customers’ personal lives.

Here are the videos’ transcripts.

Health:

So one more thing with health. In an off-the-shelf product like Claritin, here, you know, even though it’s not a fancy digital device, it can still participate in information exchanges by virtue of an RFID present in the packaging. So we put it on our shelf here; it’ll bring up directions on how to take it — it’s a lot easier to read than on the box itself.

It can also, by virtue of it knowing what my schedule is — and maybe I have a trip coming up, and it knows that I’m using this product — that creates another information intersection that we can use to query our database, and bring up an article that might be interesting.

And it’s also a retail opportunity. So, say, based on my use of this product, that a local retailer wants to promote something to me.

“Mirror: show coupon.”
“Mirror: show coupon.”

I can call up a coupon here, and then save it to my phone, and then go to a physical location to redeem it. So, again, it’s a conversation that happens in front of the mirror, but then it can drive behavior elsewhere, out in the world.

A similar thing with prescription information. There’s a lot of information behind this, and I can call it up by putting it on the shelf. So it’s personalized to my particular prescription: It shows my doctor and how many days I have left with this prescription, and when my next appointment is, etc. And you can imagine knowing what you’ve taken and what you haven’t taken — or, if you have a family, knowing which medication is yours, or for elderly people to help them take their medications on time. So there’s a lot of possibilities in this domain, too.

Retail:

So we have another mirror under development, which will have a few more bells and whistles and focus on the retail experience — which is another domain in which an interface like this might make sense for it to play a part. We do a proof of concept here, quickly:

Mirror: show retail.

So this is fun because it’ll find my face — and from my perspective, the tie lines up. It’ll attempt to match the colors to put on the tie. So, again, it’s all about context. And you can see previous outfits that I may have been wearing, and from there make recommendations about what I might like to try on next, etc. We’re working on this, and so we’ll have some more to come.

At The New York Times Company’s R&D Lab, the group’s collective of technologists, artists, and journalists talks a lot about “information shadows” — the auras of data that surround us in our daily lives. Tracking and processing the info trails we leave, the thinking goes, allows for deeper insights into ourselves — and it can also help media organizations to provide their users with news consumption experiences as intimate as they are relevant. We tend to emphasize the “self” aspects of “the quantified self“; the R&D Lab is exploring what it means to be a part of a quantified community — and, for the Times, what it means to be a provider of information to that community.

I recently had the chance to visit the R&D Lab, which is housed on the 28th floor of the Times building in New York. In the video above, Matt Boggie, The Times Co.’s Media & Technology Strategist for R&D, demonstrates the Times’ screen-top version of a kitchen table. It’s based on Microsoft’s Surface technology, modified by the R&D Lab to create a Times-oriented user experience that reimagines the old “around the breakfast table” reading of the paper. You’ll notice that, in the demo, news is both highly personal and highly social — and that the line between “consumer” and “news consumer” is a thin one. Ads look pretty much the way we’re used to them looking, but they’re also integrated into the tabletop flow of information.

And news itself, in the same way, collapses into the broader universe of information. We’re used to thinking of “the news” as its own category, as something to be consumed primarily during commutes or during post-work relaxation in the evening. But news is becoming more pervasive (there’s evidence that many people, at the moment, consume the bulk of their news during the day, integrated into their work), and the R&D platforms reflect its ubiquity. The prototypes on display at the R&D Lab consider how news can be used, in particular, in the home, woven into the intimate contexts of the morning coffee, the family dinner, the daily getting-ready routine. They explore what it means to brush your teeth with the Times.

It’s been two years since the last time the (Nieman Journalism) Lab visited the R&D Lab. In a series of videos back then, Josh and Zach offered glimpses into the group’s experiments with the next version of the newspaper, the intra-platform shuffling of news content, the living room of the future, lifestream integration for the news, and advertising innovations.

Since those videos were shot, the R&D team has been experimenting with more new platforms that project — and also guess at, and also hint at — how news will be used in the next several years. (We wrote about one of those platforms — a Twitter visualization system that can be used to track stories’ social lives on the web — in April.) The future isn’t static, Michael Zimbalist, the Times Company’s vice president of R&D, told me — so the Lab can’t be, either.

We’ll post the other video we recorded during our latest Lab visit — of a “magic mirror” that integrates information consumption into the daily routine of getting ready in the morning — tomorrow. Meanwhile, should you prefer to read about the table rather than watch it, here’s the transcript.

So the first thing you’ll notice here is that we have changed the way that the layout works. We’ve gotten rid of that sort of broadsheet design of columns and headlines in favor of a more tactile experience. Working at a table, you expect to be able to manipulate physical objects. So what we’ve represented each section as is a stack of the photographs for each of the articles that live therein.

If you’re looking for a section and you can’t find it here, you can scroll each of these — they’re a scrollable column — and the idea here is that you can share this space, as well. You might be sitting across from someone and sharing the paper with them. So you can turn these columns so that they’re facing the opposite direction.

What I can then do is open up any of these sections and pull up a carousel of articles. As you can see, we’ve left space for advertising, again, to work with our partners and continue to make this a viable business. And then what we can do, when we open them up, is page through them just like any other reader application. You can just swipe to the next page. And if you’re here by yourself, you can unfold the paper as you would with a regular paper and take up a little more room here at the table.

Here in the reader view, though, the photography tends to take a bit of a backseat as compared to the navigation. So what we’ve been able to do is tap on the photos, and for any article, the photos sort of spring out of the template. And now we can take them, move them around, scale them up, and show them to our partner across the table.

And then once we’ve done with the article, the photos themselves can continue to live on in this space, making the table a little messy and a little more playful.

In addition to that, we wanted to make sure we kept those social features of being able to share an article and send it to a friend. So what I can do there is, again, open up a carousel, pick something, I can leave a note on any of these different areas … so we can leave little notes on any of these articles. Here, just sort of a quick “did you see this?” typed hastily, with typos. And then what I can do is, I can share that. I can share that with people who work here at the Lab, or at Facebook, or on Twitter.

But then that begs the question: How can I announce my presence to the table? What’s my feature for logging in? It didn’t seem right to be able to walk up and type in a login, or have it scan your hand, necessarily. But, you know, typically, when you get home at the end of the day, you throw your keys, your phone, your bag onto the kitchen table. That gives us an opportunity to use this as an opportunity to recognize that I’m here. I put my phone down, I get these little red radials coming out of it. And now I’m presented with a list of those articles that have been shared with me. I can tap that last one that’s been left, and then it comes right back. I can take a look at that and see what’s been left for me. And that could have naturally spawned some sort of an alert on my phone, or on my laptop at work, in a couple of different ways.

And certainly we can have the table react to other objects, as well — you know, it is a table, first and foremost. So you might be eating your breakfast or having a cup of coffee. That gives us an opportunity to be a little playful with the ad experience, as well.

So then the next thing that we do here with this table surface is to talk a little bit about the way we think technology, particularly consumer technology, will be changing the experience of consuming news and creating news. So what we’ve done here is to use this application as an opportunity to learn a bit more about different devices. I can take a device — for example, this is a 3-D camera from Panasonic — and rather than describing all of its features, and providing more of a view into that, I can use the table as a tool for this portion of the presentation. I get a price tag that shows what kind of device this is, what its model number is, a range of prices that we were able to find on the Internet, as well as a range of reviews that we found.

And then we can attach content to it, as well. For example, this is an article that was written back in January that compared the Panasonic camera here to another Sony camera that was similar and came out at the same time. And it shows us, in a couple of different ways, how New York Times content will be finding its way into experiences that we don’t necessarily own or control. And that’s by design.

We’ll be doing a lot of work in tagging our articles with different locations, or people, or concepts, and opening that up to APIs and developers where they can build them into their own experiences. So this is one example of where that might happen.

And then the last thing I’ll show you, here, is, to the question of, “Great, we’ve got all these devices in here; how do we as people begin to interact, as well?” — well, we’re instrumenting ourselves increasingly. For example, this is the docking station for a FitBit. (I’ll get that off out of your way.) I’ve been wearing one of these for a while here in the Lab — it’s just a simple electronic pedometer. It’ll track how many steps you’ve taken, and it’ll use this dock to sync it up or to charge it and bring that information into a service that shows you how much activity you’ve had over the course of the last few days, done in 15-minute intervals.

The trouble, though, is that this kind of experience of sitting at the table may not be where you want to be presented with that information. But there are some places within the house where that kind of data and context makes a lot more sense. So, for example, getting ready in the morning. You might be weighing yourself, checking out your figure, seeing if your clothes fit really well. Presenting you with this kind of information might provide you with a sort of behavioral cue.

So what we wanted to do is build that experience. And unlike the table here, where we were able to use a commercial product, we actually had to build that ourselves. So we built a “magic mirror,” which we’ll show you next.

Our tour of The New York Times Co.’s research and development lab, which concludes with today’s video, represents the first time many of their projects have been seen in the wild. But before we got in there, similar tours had been given to more than 150 advertisers. The company, of course, has a huge stake in the next generation of marketing, which appears as uncertain as the future of news.

Some of the R&D group’s advertising innovations include: RFID chips that connect print ads to more dynamic content on the web, ads that can shift from one screen to another, ads that are linked to what friends are chatting about online, and targeted advertising of all sorts. They also developed the new, more-prominent, advertising units that have been adopted by members of the Online Publishers Association. Those ads are scheduled to roll out in June on major sites like the Times, ESPN, and CBS.

If the news industry’s paradox is declining revenues amid unprecedented popularity of its content, advertisers face the opposite problem: in the midst of record spending, there’s increasing evidence that their work is largely ignored. And while the fate of advertising is not necessarily tied up with the fate of news, the opposite is certainly true, so it’s no surprise that much of the R&D group’s work is focused on this area.

Loyal viewers of our first four videos from the R&D lab will notice that I’ve repurposed some footage for today’s installment, but most of it is new. And as always, a full transcript of the video is after the jump.

Nick Bilton: This is just a tikitag RFID chip, and so here is an ad for Chanel. So if I could put this on my computer, it will go off and get the appropriate ad that goes along with that. So it automatically knows because it’s RFID, it’s connected, that it’s Chanel ad that goes along with this experience. [...]

Alexis Lloyd: It used to be that you had a newspaper, and that was your sole conduit to the general public, and advertisers were buying a piece of that ability to talk to the mass audience, and that’s no longer really the case. And you see advertisers trying to create content — and sometimes successful, usually not very successful — and we’re looking at, are there new kinds of partnerships that can be formed that are beneficial for the advertiser, beneficial for the end user, hopefully, and that don’t compromise our journalistic ethics and mission. So as, so we’re exploring some of those models, whether it’s ways of helping advertisers to create content because that’s our expertise and that’s not really their expertise. Whether it’s ways of integrating advertising into some of these experiences that we’re showing you. We’ve also helped to develop some of the new online ad units that all the OPA sites are going to be launching with in June. So those are some of the areas we’re looking at in terms of advertising innovation. [...]

Michael Young: Right here I have one of our new ad units for the website that are actually going to go live this summer. It’s just one of the new OPA units. It’s just an expandable ad. So we wanted to look at this to say, what parts of advertising, you know, could we send to a TV if we wanted to, for example? So in this case, it’s a Ralph Lauren ad that we mocked up. Any component of the ad I can actually take to look on the TV on a larger screen. So if it was, in this case it’s shots from the runway. So if I saw this and wanted to have a better look at some of these clothes, again I could just take this and drag it up to the TV and see the high-res image of this on the television. [...]

Bilton: I think the question is not, it’s not finding the place [to put advertising], it’s culling it, right? It’s finding where not to put it. Because you could put it anywhere in these interfaces. You could, you know, there could be ads on the side and bluh bluh bluh, all over the place, but I think that the real challenge is, as we look to aggregate content better based on the device and look at the challenges between content and context, advertising becomes equally as important as far as where it goes. So it’s not necessarily, you know, it’s more about limiting it and figuring out the best possible solution for where you see it. And the CustomTimes experience, you know, if I watch Mark Bittman, you know what I’m watching, you know what device I’m on. You can give me a really, really great advertising experience that actually makes sense.  It’s not just a random, you know, car ad or whatever, you know?

Lloyd: And I think that one of the things that we’re seeing is that advertising doesn’t always have to be a bad experience. I never thought I’d hear myself say that. But it doesn’t always have to be a bad experience for the user. [...]

Ted Roden: Nick just tweeted. He’s a big clotheshorse, and the new Ralph Lauren line is out, he says. It looks awesome. Now we can just look at that and say, this is the new line from Ralph Lauren, and we can parse that out and figure out what’s going on and figure out if we have an ad based on that. So we fling that over and see what he’s talking about there, and the ad comes up. [..]

Lloyd: And in that case, advertising can be a value-added proposition for both the advertiser and the user, in that it’s more particularly targeted to what you’re doing, what you’re talking about. And therefore you’re more likely to actually respond to an ad and find it useful rather than intrusive.

Josh Benton: As people who look at what’s going on in online advertising, are there any sites or any advertisers who you think are doing a particularly effective job in breaking through the sort of way that the web experience, at least, teaches you to ignore the ads? Have you seen anyone who’s really doing something interesting that you think is effective?

Lloyd: Yeah, I mean, I think that the new Honda ad on Vimeo —

Bilton: Yeah, that’s fantastic.

Lloyd: I thought was a really, really fantastic use of the page.

Benton: Made me watch it.

Lloyd: It did. It made me watch it. I would never watch a car ad, I don’t, you know, really drive much. [Laughter] But —

Bilton: I think Gawker’s actually done some really innovative advertising work. They did a piece a while ago where the content looked like it was sitting on like these cartoon bookshelves. And I really like the fact that they take over the whole page, and the whole page becomes this big storytelling mechanism. Who else?

Roden: There was a great Wario Land or Wario Wii game that took over the whole YouTube, which is probably a pricey ad buy. And it was the whole page and it was kind like the Vimeo one: You pressed play, but then the whole page fell apart ’cause it was so violent or whatever. It was really good, though, ’cause it was so subtle for a very long time.

Lloyd: And I think one example of where people are using technology to target people is on Facebook, but I don’t actually think it’s — it’s an example of how targeting isn’t the whole story, that it has to be interesting or useful advertising as well. It’s almost creepy how well targeted the ads are.

Bilton: I saw an ad for a dentist that said, “New York Times employees get 20 percent off with this dentist.” And I was like, “Whoa, cool,” and I clicked on it, and then I realized that it probably just changed it for — it doesn’t matter what network you’re in, and that’s it.

Benton: Just search for your employer.

Bilton: A lot of the advertisers, you know, with the recession, they’re having a tough time trying to figure out what’s working, and the ability to experiment is not there where it used to be. The mobile space is a clear example of that. There’s not a lot of innovation in mobile advertising. You know, even on iPhones, there’s still little banner ads that click off to a regular mobile site. There’s a few that have worked, and they’ve actually created iPhone-specific ads, but there’s not that many, and I think that what we’re trying to do is say, hey, look, these are the things you can do. Maybe, you know, it’s what we call a duvet ad that covers the whole thing. You could blow on it to make it disappear or all these different things that you could do along those lines to take advantage of the experiences, the applications they’re using.

We’re back in Living Room 2.0 at The New York Times Co. today for their research and development group‘s vision of how news will fit into the armchair experience of the future. Ted Roden, a creative technologist in the group, describes two applications for Times content that might work well on your television or other large screens.

The commuter app is a mashup of publicly available traffic cameras, Google Maps, and location-specific content from the Times. Laying out news on a map is a tired concept that rarely lives up to its promise, but this app points to what feels like a truly effective use of geocoded articles and blog posts: informing me of what stands between my current location and my destination. In his keynote address at the O’Reilly Tools of Change Conference, design integration editor Nick Bilton discussed some other promising uses of geocoded news.

Roden also demonstrates what RSS feeds, Twitter, and other lifestreams might look like in the living room. The concept seems poised to go mainstream this year or next with Yahoo’s widgets for web-enabled televisions, though who knows if people really want their friends’ tweets to share the screen with Lost. (There’s evidence they do.)

It’s important to note that the R&D group doesn’t expect people will mount four screens on the wall of their living room, awesome as that would be. Their best guess is that consumers will segment their widescreen TVs in the style of cable news channels. A lot of innovation will be required to make that a pleasant experience. My living room is a two-screen operation — three, if you count a picture frame that could be repurposed to display the latest New York Times photography — and it works pretty well, even if I can’t yet flick a Mark Bittman video over to my television.

This is the fourth installment of our five-part series on the Times R&D group. Tomorrow we’ll conclude with their stabs at the future of advertising. A transcript of today’s video is after the jump.

Ted Roden: We were looking at that Jeff Han video wall, which I imagine you’ve seen.

[Clip from CNN]

Abbi Tatton: This is from John Laura, who’s in the northern part of the state. But this kind of thing above the highway seems to be a theme today.

[End of clip]

Roden: And our thought was, you know, these TVs, they’re so big and they’re so thin — they’ll get bigger and thinner, but not much. But one of the things you’ll be able to do with them is going to be touch. They’re going to add in new features. All TVs now have, are going to have Internet capability and, soon enough, probably touch.  So one of the things you’re going to be able to do with that is say, I want my stocks to be this big, I want my email to be this big, and then I want the TV to be this big. And then you sit back, and you can kind of watch your email filter in and then your stocks come in and then watch TV at the same time. And it was with that kind of thought that we built these apps on these screens. [...]

This is our commuter app that we’ve been working on. And this is pretty interesting. These are all things that are kind of publicly available, but again, on so many different sites. These are actually traffic cams that are from the New York City Department of Transportation. And you can get one at a time here. But what we’ve done is we all mapped our routes on the way into work here in the morning and grabbed — this is actually Nick’s route because he has the most interesting route — and we grabbed web cams from along the way showing the actual live traffic conditions. You can tell it’s raining out. These are totally live.

And what’s great about this is he can glance at it in the morning before he heads out and see what’s traffic like before he gets into the tunnel as he rides his scooter in from Brooklyn. And so he can say, do I need to take the tunnel or can I take the bridge? And he actually will look at this every morning. There’s an iPhone version of this, too, that he can pull out, and when he gets to an intersection just before it, where it says, should I take the tunnel or the bridge? And it actually saves a lot of time in the morning.

And also, we pipe in live traffic alerts from Yahoo, and all of our content is geocoded as well. And so the City Room blog has a lot of stuff throughout the city, so you see a lot of City Room content springing up on there. And it’s kind of a great way to get another quick glance overview of what’s going on.  

And the final piece to this is just a lifestream app.  Now they have these on the desktop. You can get these.  All this one does is filter in your RSS feeds, your TimesPeople feeds, Twitter, it’s got Friendfeed, it’s got Facebook. Kind of any social updates that you can have will show up on here. And these are all live. These are things from just the past few days. And this isn’t a new concept, but our thought was, what happens when this makes it into the living room, and how does that change the course of things?

So on your desktop, you’d click, you’d see a link here, and you would click on the link and it would open in your browser. But what happens when you’re not at your browser? So one of the thoughts was, we have— Mike recently shared a Mark Bittman video, and Mark Bittman is our resident cook, cooking guy. He does these great videos and great articles and great recipes around food and things like that, and they’re in the Times all the time. So what we can do with this here is, you would click on this, and it would take you off to the website, and you could look at the recipe and all that. But in the living room, what you can do is drag that over to the TV, and it plays right on the TV.

And, of course, that brings up the CustomTimes on my phone, too, since I’m in the room. It opened up, it opened up on my phone. And so now I can actually delve deeper in. I can see some photos from the shoot. Click on it here, and it shows it on the TV. And I can also click on the recipe, so I can actually follow along with what he’s cooking on the recipe. And then, if I wanted to, I could go to the website. You know, I could click through it on the link and read the actual big article he wrote along with it. But here in the living room, this kind of a much more sit-down approach to that.

And also, we did look at this as far as ads go. So let me just stop that there. So we looked at this as far as ads go because a lot of times, you know, part of this news feed is just things my friends are saying. And a lot of times my friends talk about products. Someone just bought an iPhone, or someone didn’t like their flight or whatever it is.  

And so we looked at how will advertising work in this world.  And so we can just take like here, Nick just tweeted. He’s a big clothes horse, and the new Ralph Lauren line is out, he says. It looks awesome. Now we can just look at that and say, this is the new line of Ralph Lauren, and we can parse that out and figure out what’s going on and figure out if we have an ad based on that. So we fling that over and see what he’s talking about there, and the ad comes up. Now that could easily be a video or even a coupon that gets sent to your phone or anything like that, but all kind of based around how this plays out in the living room.

In a corner of the research and development lab at The New York Times Co., they’ve prototyped a living room of the future. It’s not as whizbang awesome as you might hope — a lamp glows red or green depending on how the markets are doing — but it does feel like a reasonable conception of Living Room 2.0. Their major bet: as Internet-enabled televisions become more common, people will increasingly choose to consume web material on those huge, high-definition screens.

That wouldn’t, on its face, be an advantageous development for the Times, which produces the vast majority of its content in longform text you’d never consider reading on TV. But as Alexis Lloyd, a creative technologist in the R&D group, explains in today’s video, it may be possible to shift gears in the living room and emphasize the newspaper’s multimedia content. She demonstrates the concept with “Choking on Growth,” a major series on environmental damage in China from 2007.

This is the third in our weeklong series of videos from the R&D group, and it may be the one that’s easiest to imagine coming to pass. Laptop and desktop computers are already commonplace in the living room, Boxee is a huge hit, and Apple keeps plugging away at converging TV and the Internet. (On Oxygen’s The Bad Girls Club, the cast members check their email on a television in the living room. QED.)

Still, reimagining The New York Times in HDTV is a challenging leap. (You might recall the Times Co. made an unsuccessful foray into television with the Discovery Channel earlier this decade.) The newspaper produces a ton of multimedia content — certainly more than its competitors — but a satisfactory living-room experience would require video on a scale the Times isn’t yet producing. That’s why they call it the future.

You’ll see more of the R&D group’s living room in tomorrow’s video (yesterday‘s was also shot in there). After the jump, you’ll find a mock-up by design integration editor Nick Bilton, which adds a projector but is otherwise pretty faithful to the actual room. And below that, there a transcript of today’s video.

Alexis Lloyd: The main problem we see with content from The New York Times in the living room is that our primary form of storytelling is still long-form text, which works really well on paper, still works well on the web — but once you’re sitting ten feet from a television in your living room, that pretty much breaks down. But we do produce all this great multimedia content. It’s just usually pushed off to the side a little bit. So in this demo we are asking the question: Can we flip that paradigm around and use the media that works really well in the living room — the video and the images — and make that the spine of the story, but still pull in some of the text and pull in some degree of interactivity that you might want when you’re in the living room?

So I’ll show you this. In this case, I’m using a standard mouse to navigate this, but we’re also looking at a lot of devices like these air mice that I could sit and navigate from my sofa, as well as doing custom remote controls and interfaces like the kind that Mike showed you on CustomTimes.

So this is just a one-minute video that I’m going to start playing, and as the video plays there are these panels that appear that I can open up to show you contextual information about what’s being discussed in the video. So in this case, I can get background information about the turtle that’s being mentioned. It’s text, but it’s short, it’s big, and furthermore, it’s optional. So I can just open it up, read it, and then I’m back in the video. So it doesn’t take me out of that central experience of sitting back and being told a story, which is my primary kind of mode when I’m in the living room.

And we can do this with all kinds of content. So in that case, that was some, a piece of text that was related to what they’re talking about in the video. In this case, there’s a woman who is being interviewed. She’s written a book about mammals in China. I can open this up to read an excerpt from that book. Furthermore, it knows it’s a book, so there’s an e-commerce component that’s integrated into this. And I can just choose from this interface to buy the book. It goes into my Amazon one-click shopping process, and I’m back in the video. So I’ve done all this, but I haven’t been taken out of that basic experience.

And this is really pointing to the idea of creating more granular levels of metadata about content. So we have metadata about our videos as a whole, but now we can begin to say, at this particular point in time in the video, we have a related map or at this particular point in time, they’re talking about this lake. And we have a slide show about that. So I’m going to open that up.

And then you can see our photojournalism really has a place on the big screen because the photos are stunning at this size. And furthermore, the photos themselves have this more granular level of metadata where there are these hot spots that I can use to get deeper information about objects or people in the video — or in the photo, rather. So I can find out all about this toxic algae that’s growing on this lake as a result of chemical plants dumping on it.

And at the end of the video, we’ve also integrated some social functionality, so I can choose to share this video with other friends, and it pulls in the people I most frequently share with. I can say, I want to share this with Michael, and then it will go into any number of his social feeds and into our lifestream app, which Ted will show you in a moment. There’s also some contextual advertising in here, so you might be inspired to give to Unicef‘s clean water campaign after watching that video.

And furthermore, that’s just a one-minute video piece, but this series was a yearlong series. There is a huge collection of multimedia content that was created for it. So we started asking the question of can we use the metadata that we’re already creating for our content to allow readers and users different lenses into these large collection of media that might be overwhelming to them?

So in this case I have four different views that I can go into that are dynamically created from the metadata associated with it. So there’s an editors’ choice where I can say I just want to know what the New York Times editors think are the highlights of this collection or this package. But I can take that same content and I can sort it geographically, or I can sort it over time and see a timeline. Or I can sort it thematically and start to see relationships between different themes in the collection of media.

So those are some of the different ideas we’re looking at around how our content could be produced and packaged and repurposed for exploration in the living room.

Here’s the second of our videos from inside the research and development lab at The New York Times Co., where they’re envisioning how news will be consumed in two to ten years. (You can catch up on the series here.) Some of the goodies you’ll notice: a Samsung tablet, an iPhone, a Sony Bravia TV, and an application called CustomTimes that they’ve developed to work on all three devices.

The R&D group is obsessed with the ability to seamlessly transition among web-enabled gadgets. They’re not convinced that the future will land on a single, multipurpose contraption — like some sort of Kindle meets Chumby meets Minority Report. Instead, they predict consumers will connect to the Internet through their cars, on their televisions, over mobile networks, and in traditional browsers, while expecting those devices to interact and sync with each other.

Nick Bilton, the group’s design integration editor who narrated yesterday’s video, and Michael Young, the lead creative technologist who stars in today’s installment, won a major hacking event in 2007 with their startup Shifd (pronounced “shift”), which is an attempt to achieve some of that cloud-like portability. And the same philosophy is evident in the way they’ve conceived CustomTimes (which, it should be noted, is more a proof of concept than a product on its way to the marketplace).

One term I didn’t hear in our visit to the R&D lab last week was “platform agnostic,” a concept once championed by Times publisher Arthur Sulzberger Jr. and deputy managing editor Jonathan Landman to describe how the newspaper would offer its content on any medium desired by the audience, from e-readers to television.

That philosophy remains intact, I think, but the phrase’s meaning is worth some thought. One of the more pointed passages in Mark Bowden’s recent Vanity Fair profile of Sulzberger was a quote from Tom Rosenstiel, director of Pew’s Project for Excellence in Journalism:

When I first heard Arthur talk about being platform agnostic, I knew he was trying to suggest that he was not stuck in a newspaper mind-set. But I thought there were two problems with that language. One is, agnostics are people who don’t—who aren’t sure what they believe in. That’s the first problem. And the second problem is, in practice, there is no such thing as being platform agnostic. You actually have to choose which platform you work on first, which one comes first. [...] Platform agnostic means that all the online companies are going to zoom past you, because they’re going to exploit that technology while you’re sitting there thinking, Well, we don’t care which platform we put it on. You need to exploit the technology of each platform. You need to be, in fact, not platform agnostic but platform orthodox.

There’s no doubt that the R&D group — and probably Sulzberger, too — agrees with Rosentiel’s point. (In tomorrow’s video, you’ll see one way that they’re attempting to repackage multimedia content for different platforms.) But I think “platform orthodox” is a useful perspective from which to assess their work: How well does CustomTimes prepare for our gadget-juggling future?

A full transcript of today’s video is after the jump.

Michael Young: So I want to show you something we’re calling the CustomTimes. It’s our vision of a three- or four-screen application with customized or personalized version of The New York Times on web, mobile, TV in the living room, and then potentially in the car.

So we’ll start with the web component. I’m just showing a website here on a Samsung tablet. The idea is that you would come to it and you would initially configure it and you would seed it by selecting some sections from The New York Times of content that you want to follow. So just a very simple, clean interface here, a couple sections selected. Actually, I have them all selected. Let me uncheck a couple. Save this here. And what we do is we customize a web version here with all the sections that I selected. I’ll just scroll through quickly. Very clean, simple UI, headline summaries, and images when we have it. So now you will get the same information across the web, across your phone, and in your living room.  

So let me show you a couple of other concepts we have here. Next to each of these articles is a flag button. And the idea here is that anything on any of these sites — web, mobile, living room — you can take a piece of content, whether it’s a story or a graphic or video, and you flag it to either watch it later or read it later on a different platform. So one example is if I have video on the website here, I could flag it and say, send it to my living room. I want to watch it in HD when I get home. Another option is at the end of the day, if you had a couple of articles you didn’t read, you could flag it and say, send it to my car. And on some of the new cars, what we’re going to try to do is do text-to-speech on the article, send it to the car, and have it read to you on the way home.

So earlier I had flagged a couple of videos that I would want to watch in the living room when I got home. We have a George Clooney, a Mark Bittman, and a third one here. So they’re sitting and waiting for me on the TV. Here is the Mark Bittman piece. So I can use my TV remote to play these, but also now that these devices are becoming connected, I can use any of these devices as the remote.

So if I start to, if I click on George here, I could play the video on this device, but as I start to take him and drag him up, you’ll see this blue panel drop down from the top. So it’s the devices that I’m near or that I’m connected to. And you’ll see a little TV icon pop up there. So if I just drag George up to the TV and drop him there, the video will start to play on the TV.

[George Clooney video plays on TV.]

[...] So the same idea: here is this flag section on the CustomTimes and, again, George Clooney video sitting there waiting for me. And now we built some sensors into the software with CustomTimes so that it knows what devices are on the same network. So I don’t know if you can see it here, but next, underneath the George Clooney link, there’s a play link.  So for any video that’s on the phone here, it knows that I’m now in the room near the TV, and it gives me the option to play it on the TV. So the same idea: I can hit play here and it will play the George Clooney video on the TV. [...]

But we’re also looking at how advertising and brand messages from advertisers to fit into this world. So right here I have one of our new ad units for the website that are actually going to go live this summer. It’s just one of the new OPA units. It’s just an expandable ad. So we wanted to look at this to say, what parts of advertising, you know, could we send to a TV if we wanted to, for example? So in this case, it’s a Ralph Lauren ad that we mocked up. Any component of the ad I can actually take to look on the TV on a larger screen. So if it was, in this case it’s shots from the runway. So if I saw this and wanted to have a better look at some of these clothes, again I could just take this and drag it up to the TV and see the high-res image of this on the television. And you can do that with video, you can do that with slide shows from the runway, really any content from the ad message.

The New York Times Co.’s research and development group has some of the best views in their midtown skyscraper — 24 floors above the newsrooms, higher even than the executives’ suites. Developers in the core R&D group — with titles like “lead creative technologist” and, my favorite, “futurist-in-residence” — are charged by the brass 14 floors below them with anticipating how news will next be consumed.

Among their hunches: in the living room.

Josh and I visited the R&D group last week, and this week we’ll be running five videos showing how they’re looking at the future of news. Today we begin with design integration editor Nick Bilton, who runs through their thinking on e-reader devices, news consumption outside the web browser, and interactive advertising.

You’ll notice there’s a marketing or advertising component to nearly all of what the group is working on. While this is the first time much of the lab has been seen publicly, they’ve given similar tours to more than a hundred advertisers and agencies, Bilton told us. And keep in mind the company has an interest in appearing ahead of the curve to investors.

They drink better coffee in the R&D group, not the burnt stuff chugged by reporters on deadline. Maybe that’s because they have time to let the grinds brew: what they’re envisioning won’t reach anyone’s living room for at least two years — if at all.

Up there on the 28th floor, the group’s toys — e-readers torn apart, touchscreen displays, netbooks that bend in every direction — can feel a touch presumptuous for a company surviving debt payment to debt payment. It was just this winter when Michael Hirschorn loudly suggested in The Atlantic that the Times Co. could go out of business, “like, this May.” The Times will endure, in one form or another, and the R&D group is the beta version of the company’s future.

You’ll find the details of what Bilton and his colleagues are thinking about in each of the five videos, and I’ll address some of their key ideas as the week progresses. (Note: In today’s video, Bilton demos an Adobe AIR application that’s very similar to Times Reader 2.0, which is set for release this week.) There’s a full transcript of the video after the jump, and be sure to come back each day this week for more from our visit.

Nick Bilton: This is the core R&D group, and these are just some of the projects we’re working on. This is what we call the newspaper 2.0 table, and it’s looking at these next generation of reader devices and really trying to stay ahead of the curve with these devices.

There’s two things that we are doing here. One is trying to educate the company on where these devices are going, but the other thing is actually prototyping content on them. So this is an E Ink development kit that actually was broken in transit from Vegas last week. It’s a little chipped, but this was a device that we got from E Ink where we prototyped what content would look like on an E-Ink device that didn’t exist yet. And so we have the full layout with the typography and different user interactions that we can experiment with. And so this is really trying to prototype and understand where these devices are before they even exist and what our content will look like and how it will translate.

This is just some flexible e-ink. There is a big push for flexible displays and devices and where they’ll be. There’s been some breakthroughs in the past six months that will allow devices to become more flexible with the PCBs being more flexible, the chips are going to start to become more flexible over the next few years. And that’s really going to change these devices. The one question is: How do you tell someone that it’s bendable but not foldable? So.

Josh Benton: Gotta educate the customer.

Bilton: Gotta educate the customer. And then, you know, a lot of it is just trying to understand the user interaction and really trying to work with the manufacturers. We work with Sony and Kindle and all these guys. We work with this guy Rob Samuels, who is the project manager at nytimes.com for these devices, and we’re trying to work with all the device manufacturers to say, you know, this is how our content should work and how it should follow through.

Another big thing that we always explore are the netbooks. These put a whole different generation of people online, and we’ve been looking at how you tell stories on these machines. You know, some of them have foldable screens, some of the are touchscreen, they’re all different sizes, and we really have to understand how our content, the stories are told on there.

An interesting technology that is going to affect the e-book reader industry in the next year or so is the screen from the One Laptop Per Child. Mary Lou Jepsen came from One Laptop Per Child. She invented the screen, which is actually called Pixel Qi — Pixel Q-I. It’s based off the E-Ink technology and LCD, and it’s mashed together, and it creates a color version of E-Ink that you can actually switch between this LCD with full movement to E-Ink in low-light situations and low power and things like that. So she’s going to be shipping those devices, the screens in November or so which means that we’ll probably start seeing them in the market place in the next year or year and a half, which should be really interesting.

We talk a bit about making the paper more interactive and adding functionality. This is just a tikitag RFID chip, and so here is an ad for Chanel. So if I could put this on my computer, it will go off and get the appropriate ad that goes along with that. So it automatically knows because it’s RFID, it’s connected that it’s Chanel ad that goes along with this experience.

So it’s just really trying to explore and understand where RFIDs — there’s this company in Boston that’s starting to explore printing RFID in paper at a penny to five cents a piece, which could really open up different areas for advertising.  

As far as working with reporters, these are different GPS devices that we’ve been playing around with. We’ve given some to some reporters, and it actually automatically geocodes where they are, and whenever the time stamp of the story is uploaded. It then cross-correlates it and says, this is where this story or this photo has been filed from, or this photo, and it automatically puts it on the map.  And it’s a whole different method of story-telling that nobody is really required to get involved with. It does it automatically. So we did this with the Frugal Traveler and a couple of other reporters, and it’s been pretty interesting to see that happen.

Another application, going back to these news reader devices is, I mean, we’re looking at touchscreen constantly. [Dialog box appears on screen.] Thank you, Windows. [Laughter] This is the International Herald Tribune Reader that we’ve been working on with Adobe, and it’s built on Adobe AIR. And one of the really interesting features of it is that it can reformat and re-lay itself out accordingly depending on what size display it’s in. So if I’m on a screen this big, it will format and lay itself out. If I’m on a screen the size of one of those little notebooks it will, it’ll re-lay itself out that way. It does the same thing on the article level if I want to resize the font, I can go smaller and it reformats itself and fits in that thing.  

You’ve got the crossword that you can do. It’s got all the features from the web and even more sit-back experiences — like we have the news in video and the news in pictures that can become full screen. I can navigate through this way. And then another interesting feature is this browse feature where it sits back and it says, let me navigate the content just by flicking through, and I can go from section to section and article to article, and then just jump right in.  So it’s a really interesting visual way of navigating this content.

Benton: What do you think — there’s so much inertia and momentum in the old, the traditional web browser, and that’s how most people get their news electronicaly now. What do you think it’s going to take to get people to move to something like this? To step out of the browser and have an Adobe AIR application, or have a dedicated device, or interact in a different way. What’s the tipping point?

Bilton: Well it’s, you know, if I constantly — I mean, look at Twitter. If I kept going to twitter.com, and it turned out that it was a much easier experience for me to download an AIR application or Tweetie or something like that and just have it running constantly on my desktop. So why couldn’t I have this experience? I still go to twitter.com sometimes, so this is just an alternate for that. And this is also looking at devices. This is all offline reading. This is, you know, if I want to put this on a tablet PC and read it on the subway, then I can do that and it formats and fits for that experience.

You know, I personally think that the browser — there’s too much going on on there. I mean, what buttons do you use other than the back button and to actually type in a URL? So it could be a full-screen experience, it could be a desktop application. There should be a blur between those lines I think.