For all the criticism of Curley and the AP, he had a few really smart observations about the economics of news. Regarding the AP’s competition, which ranges from free news sites to CNN’s new wire service, he was realistic:

Our pricing has to be competitive. There are going to be more competitors. There are going to be fewer people who can afford us. This is a moment of tyranny in the marketplace. There are quality providers, and there are those who aren’t going to be able to sustain the revenues. We don’t expect to have the market share that we used to have.

Speaking broadly about the market for journalists and journalism, Curley was candid:

The truth is, again, the market for news is growing. But the reality is — and none of us can create some fantasy picture here — there is an oversupply, at least in the short term, of us. And so that is creating some differences in the market, and I see these being resolved by innovation and creativity over time.

Oh, there were silly remarks, too, like his misuse of the word “crowdsourcing” and his flat declaration that “we’re all done with random search.” (If that sounds like an ad for Bing, consider it another sign of a looming partnership between Microsoft and the AP.) Others will be interested in his explanation of how the AP will change the licenses for online distribution of its content.

Some comments that I reported on Friday aren’t contained in this recording, which covers Curley’s opening remarks and a question-and-answer session that followed. (Those additional comments are from a separate small-group chat with Curley after his speech, and my source has asked me not to post that audio.) You can download the audio, listen to it below, or read the transcript after the jump.

[See post to listen to audio]

Here’s the full transcript:

Thank you, Tom. I must say that I feel particularly emboldened to be introduced by someone from The Economist. I have been interviewed by The Economist twice this year, and nothing has made The Economist yet. So I’m feeling I can get away with whatever I want. Secondly, I understand that Gov. Palin has been in Hong Kong recently, and I further understand that she was paid something like $300,000 to appear in Hong Kong. Whatever happens today, you’re getting a bargain. [laughter]

You know, the attendance at journalism sessions and press club meetings these days is a record just about anywhere in the world, and that can be for only one reason. Well, maybe two. One, you’re coming to see how long you’re going to be employed. And, two, is there any hope? I can’t answer the first one, except for the people at the AP, and they’re fine. I can give you a little hope.

And I come with a message of optimism. I must say that for the first time in a decade, when the full sweep of the Internet impact became clear, I now see how we can get to other side, and I now see the leaders of the media organizations having resolved to get to other side. And so I’d like to talk to you in that regard today. No one is promising streets paved with gold or an easy on-ramp to the digital era. In fact, we have only just begun when it comes to figuring out what we have to do. But there is a sea change in attitude, and it starts with the realization that goes like this:

First of all, more people are seeking news more times a time in more places than ever before. And if you understand that, you understand that the market for content is growing. At the same time, the value of that content has been devalued. It’s now at the lowest level, I think, in history. And that’s what we’re about to change. And people have announced a number of different approaches, and many are beginning a number of experiments around what’s generically called pay models, some of which will work and some of which will go to the experimental dustbin.

But the reality is that all of us know that our content is valuable, and what we do, and the sacrifice that you make in asking the tough questions of people around the world, and the risks that people take, especially those front-line journalists, you’re now seeing a response. We deserve to be paid, and now it becomes a manner of trying to figure out how to do that. So the first change is an attitude change, and I can tell you that the CEOs of the media companies — in the United States, at least — have decided to stand up and say, this is the moment. We are not going to be taken to the cleaners. Our content is not going to be monetized by others, and it won’t be. We intend to participate in that stream, in that revenue stream.

The second is the new business models. People are willing to change, and now there is talk of several different business models, and I’ll be talking about some of those ways in a few minutes.

The third is that there is a willingness to do research and experiment, and there are serious budgets for research and experimentation, even during this dire economic setback and even given the fact that resources are strained. So they know that they have to change, and they intend to change right now.

It’s been said that the big mistake that we made, the orginal sin of the Internet, was to give our content away for free. That is not true. We have to engage. The public is consuming content on the web. We have to be a part of that or we will become irrelevant. That was not the mistake we made.

The mistake we made was not imagining more than one product. We took all the good stuff we created, whether for broadcast or for newspapers or even a couple of magazines that have the good sense never to quote me [laughter], and then took it all and put it on the web without much thought. Oh, the number of links, the lightness or heaviness of the page, how much multimedia. Oh, we went through handwringing sessions over that. We didn’t have enough multimedia. The links were too heavy. Or we had too many pictures, and it wasn’t loading fast enough. We spent endless amounts of time over those issues, and they were largely all irrelevant. What we should have been thinking about is a whole different stream of products. And so we are. And so that’s what you’re going to see evolve over the next few years.

You’re going to see what I consider to be a base. I know when it comes to AP, what we’re talking about is that there will be a base license. And you want our content, you display it, use it in your regular product, that’s fine. And display it on the web, that’s fine, that’s one price. If you want to take it and use it with syndication, if you want to use it with email services, if you want to use it and try to draw traffic through other aggregation plays that you’re involved with, there is an upside. And we expect to participate in that upside. So that would be one other way to go.

There are a number of premium products that can be imagined or content verticals, especially, seem to be robust areas of opportunity for growth. And, finally, price can be reserved, and there can be exclusives given, perhaps on a time-based measure, those who get access to that content and the rich multimedia or the metadata that comes with it, might get an exclusive for, oh, 20 or 30 minutes. And we think all those things have value. And those are the product streams that we’re imagining and beginning to create, some of which will be rolled out next year.

The key to that is what we call a news map or a guide to the authoritative source of news, which is to say, who breaks the news? You know, we’re all done with random search. Random search has worked well for the aggregators and the certain portals who have taken our good work and run off with it and given us pennies on the dollar and not paid us appropriately. We’re also done with the people who have taken our content and done crowdsourcing with it, put us in with others that have destroyed our brand reputations and have made these series of links that have been part of the self-referring loop and taken the audience away from the content sites that are broken.

AP members, more than 1,400 of them, have already agreed and are participating in a news registry that will work with editors as well as technology. It will still be curated by editors, and we will provide the links on a real-time basis to breaking news. If a story is broken by The Financial Times, we will take you to The Financial Times website, not The Times of India, even though the Times of India may optimize for search engines more so than The Financial Times of London.

And so that ability to create a news registry is one of the base steps, and we now have, as I said, more than 1,400 organizations participating in that news registry. We will launch and test the first product from this news map and the news registry in six weeks with ten organizations — nine newspapers and one sports statistics provider, a joint venture we have with News Corp., called STATS. And we are creating a news registry.

The key to what we have done is you have to begin by protecting your content. We, especially the United States, have been asleep. We could have done this in 1977, when a copyright law was passed, but we had natural protection by virtue of geography. And so we didn’t have to create copyright protection. The Digital Millennium Copyright Act of the late ’90s came and basically enabled Google and the Google wannabes to do what they are doing, and it’s time for us to take back the night. And it’s time for us to get control of our content, and so we shall do that.

We will put certain marks on all the content that’s working with us. They will include protocols: We will work with you, we won’t work with you. You’re licensed for this, you’re licensed for the full stream. And we will, again, make that available widely to all the participating news organizations. We just had a world summit of news agencies and agreed to extend it to them as well as all the content organizations.

We call this program at AP, 3P: protect, point, and pay. The protection part is to put in the rules that govern the content. These content markers can be stripped, but when the marker is stripped, it calls home, and it tells us, if somebody is stripping the marker on a regular basis, we can then act, either with protocols that restrict the content going there or legally, through legal means. And we intend to set up a service bureau that monitors the usage of content, when it’s downloaded, how it’s used and as well as whether the content is being used in a licensed or unlicensed fashion.

The next part is the point and to point to those who are breaking stories. So if you’re a part of an organization that is breaking stories, you are going to be rewarded. You are going to get the traffic that you deserve because your reporters have been aggressive enough to break the stories or to write in such a way or to produce multimedia in such a way that your content is attractive. It will come to you. It will not go to the free riders. It will not go to the others.

And finally, is the pay piece. And the pay piece is pretty exciting. So if you put the new products with new ideas in terms of how to go to market, you can begin to see that this market for content that is growing might be a market where we, the providers, the producers, start to get paid one way or another, through licensing, through behavioral contextual advertising, through just a fee for having access to that content. Or fee for services, for providing all these backend services that we’re providing, we expect to be able to get some fees for our services.

And also through our own self-referring network. If you go on to Google, normally the top 10, of the top 10 items appearing on Google, 9 of the top 10 will be part of a Google self-referring network. Everything in there is linked to some other part of Google. It’s really a brilliant business approach, and God bless them. And all that money has come to them, all 22 billion. Folks, they can share.

And now there’s something called competition. And I think we stand at an enviable moment where Microsoft and Google have decided to go to war, and we who produce content can start to figure out whether there’s an opportunity for us to help that sharing in a way that reverses the outflow of money from media and takes it back. And so that’s the efforts that we expect to see unfold over the next, especially, 3 to 15 months, and iI think you’ll see a lot more efforts and a lot more announcements that are a lot more bullish from media companies than you have in the last decade. And so that’s the reason for my optimism.

The other part is that we don’t expect to do any of this alone. We expect to do this with partners. And we expect to set up partnerships that we hope are enduring but at least will follow certain principles. We expect real-time metrics. We will no longer work with people who won’t work with us in real time.

Secondly, we are not talking about being luddites, throwing up the moat, and hoping to hold on by taxing negatively consumers who might try to access our content digitally. That is not what we’re doing. We fully understand we have to engage with the new audiences. We fully intend to engage. And we fully intend to use all the tools and ideas of the moment.

AP, in July, conducted an experiment during the Supreme Court nomination hearing for Justice Sotomayor. We began a blog. A blog is kind of an old idea, but we put a couple of twists on it. We worked with Twitter to help get the feedback. We took questions from consumers that they want answered of senators or of Justice Sotomayor. And when the question made sense, we tried to get it answered. We worked with Yahoo as a base, as a center of gravity. We don’t have our own commercial website, and we don’t see going there. But we do know we have to engage with consumers, and we do know we have to engage in a way that is up to what they want on a real-time basis. So we put those pieces in place.

We got a dozen newspapers to work with us, including The Los Angeles Times and The Washington Post, among them, and we ended up driving more than double the traffic that these news organizations have seen as a result of this experiment. Yahoo’s traffic was 27% greater than it normally is, and the traffic to the Sotomayor blog on AP was three times greater than the traffic to the regular Sotomayor story that AP did. So it is possible for us old organizations, 146 years old, in our case, to be reborn and to understand these techniques and to drive the audience.

What we need to do, then, is get paid. We estimate that if we were able to monetize, get the revenue we deserve from all that, then that would have exceeded $200,000 for one week’s extra effort. So that’s the type of thing we have to capture. and we expect to do that with partners, and we expect to do this on an ongoing basis.

Very soon AP will create a new desk called the nerve center. It will be a desk that engages with social media, and we expect to be changing our filing protocols. Filing protocols are a set of standards by which AP copy is filed around the world. We have 15 editing desks, and we expect everybody to get the news out under a certain set of protocols. Right now we have a protocol system. It’s AP, it’s got to be simple: one, two, three. One is to get a headline out. Two is the first paragraph of about 135 words, written in broadcast tense, not newspaper tense. And then three is whatever else you do with the story. We’re going go to zero to four. Zero will be borrowing from broadcast and have a promotional element to it, so that we will start promoting even before we move the story. Four will be a full multimedia effort. So these are the things that we see doing. We see them on an ongoing basis, and we see all our editing desks being empowered to work with these techniques and having that work through a new desk called the nerve center, so that we will engage with the social media networks.

Finally, while I’m here, I also want to say something that I think is really important when you come to the Foreign Correspondents Club of Hong Kong and for all of us when go out in our work. And that is, in this era, we really have to stand up and understand that we have to take up the cause even more vigorously for a free press everywhere we can. You know issues here far better than I, so I will not belabor them, but I do want to call attention to, one, a growing trend, especially among sports leagues, to try to take control of the venues in a way that takes the content creation away from us. We have really got to work together. The big guys in sports have all organized. They’ve gotten their lawyers. And we in the media have organized as well to fight these efforts, and we have to take whatever means necessary, including going to court. We have been to court a number of times, and we have been successful. But this is an area where we have to start guarding. So whether it’s government or sports leagues, at a moment where the digital world is opening up opportunities, it’s opening up, also, some possibilities for those who would take back transparency and put it back in the shadows. and so we need to campaign.

And the second, leads me to this. I urge all of you to come up with tactics that makes sense. What we have decided to do, as a coalition of news organizations in the United States, is to try to get one thing done a year. Pick one thing and get it done, get it fixed, put the spotlight on it. So it’s not a generic call for sunshine or a generic call for the free press. We’re all happy to do that, and we should. But let’s be effective, too. Let’s figure out what we have to do to get something done because at this moment, the stakes are high, and it’s too easy for the pretenders to walk away and say, oh, it’s OK. So all these things, we have to double-down. This is no time — now the economy is tough, there are a lot of excuses that we can have — but now more than ever, every journalist at every level has to double-down and figure out how to be effective and to get it done.

With that, I thank you and, again, I’m delighted to be a part of this. And that Sarah Palin check can come any time. AP’s running a little under budget this year. Thank you.

Question and Answers

[Question about news websites like the BBC that provide news for free.]

Well, here’s what’s going to be different. Anybody who wants our content is going to have to agree to our principles. So we will not grant a license unless the licensee agrees to participate under the terms and conditions that we want. So further re-syndication will be restricted, and we’ll be working with all the large organizations to do it in an intelligent way. And, again, this is not about throwing a moat up and trying to hope that the Internet world passes us by. I assure you we are not that stupid. We really want to figure out how you’re going to market. We want to help you go to market, and want to be a part of it. But if you want to take our content, put it in an email, re-syndicate it, frankly, we’re going to ask for additional fees, and we want an upside. So in the past, it’s been a capped or fixed license approach. So I expect over the next couple of years there will be some wonderful negotiating sessions, and I look at some of my sales people over there [laughter], knowing full well who will take the burden of these conversations, tell them how much I appreciate what they do.

But this is a moment, and it’s not just about AP. It’s about all of us. And what you hear when you talk to the media leaders now is that if we don’t do it now, we are toast. So we are going to stand up, and we are going to go for it. And will we have all the answers immediately? I assure you we won’t. Will we make mistakes? I will only speak for myself and say, yes. But we have to try, and try we shall.

[Question about what he says to young journalists regarding job security in the industry, referencing Global Post's model of journalists on retainer.]

A: Well, I’m much more optimistic than no job security and deep cynicism. The truth is, again, the market for news is growing. But the reality is — and none of us can create some fantasy picture here — there is an oversupply, at least in the short term, of us. And so that is creating some differences in the market, and I see these being resolved by innovation and creativity over time. Whether Global Post is one of them, that will be determined or not. But I can tell you that in the United States, in cities that have lost large amounts of coverage, that organizations are being formed. Some of them are being supported by philanthropic groups right now. Some of them are being supported by investors with a particular cause in a given area. And so you will see a lot of this experimentation. There won’t be gaps in the market for long. The gaps will be filled. And i would look at how those gaps are being filled.

The problem with the blogosphere right now is that there are only a few that can emerge at a level where it makes sense or they get paid. And I think that will be an area will there be a lot of growth, successful growth. I think the new methods of the Internet technology will do a sorting there, and there will be more people who will be able to make more of a living, before long, not overnight, through some of those efforts. So there will be a sorting out, and I would not advise somebody to just start blogging, no matter how good they are. Speaking again for myself, I benefited by having a couple of tough editors in my life who whipped me and whipped my writing and who guided me in the middle of the night to some better decision-making. And i worry right now that the pipeline for those journalists will be, right now is too small. And then all of sudden, once we start figuring out how to get paid better, there’s going to be more demand, and there will be a bit of a surprise on the upside, and we will be taking people who were not as trained as we would normally have.

So I think there will be some bumps along this road, but I’m pretty optimistic. Writing is a great skill. It helps you in any number of fields. And you really do have to know two things: It helps to be at least bilingual, and it certainly helps to do multimedia and be comfortable with the new cameras which are emerging and the production of multimedia content. So those are some pieces of advice i would give to young people.

[Question about why consumers would go to the AP's news portal if it "offers less content for more money than Google."]

Again, I may not be as stupid as I look. We are not creating a portal, we are not creating a site. We expect to be working with others who are investing billions in these efforts. But we expect to be working with people who will work with us, respect our principles. And we expect to be part of a very large ecosystem. The Sotomayor experiment, again, was indicative of the direction that we chose. We did not do it ourselves. We went out, we got a half dozen newspapers to experiment with, and then we had others [?].

She had an important ruling in a case: The New Haven Register joined the network, Miami Herald joined the network. There was interest in this. So what you’re seeing is an ability to put these networks together. We have, at AP, already networked. We have taken an old newspaper cooperative and used mobile as a way to create our first product of the digital cooperative. We have 1,400 newspapers that work with AP. You can customize, you can put in your local zip code, and you can get the local news for that zip code. So you have national, and you have local. These are the types of things we are doing, and we’re doing it with others. We have reached out to the magazine industry. We have reached out to the networks. We are just not looking at this as self-indulgence at all. This is very different. It’s about a network.

[Follow-up question about what he means by the "news registry."]

It is not a website. The news registry is a governance structure that has rights and protocols and lots of lawyers and technology people behind it. AP has spent a great deal of money creating and developing Internet technology going back to 2003. We used to be a satellite where we took everything and we broadcast that one product the same way to everybody. And we have become a multimedia, two-way database, and so we are able, we now allow editors and journalists to self-program how they get the AP content through the AP database. And we expect, now, with some associations with certain other portals and certain social network sites, to engage. But, again, we will do it with a set of protocols that we will put in place.

You opt in, and you tell us what your rules are. And you can work with AP doing it all under the terms and conditions you’ve set, or you can have the server at your building. Like, I can’t imagine, for instance, that some news organizations like Reuters would want us to have their Internet deal. So we would expect that if they work with us, the server would be at Reuters, etc. So you can customize it on your own or do it otherwise. We expect for those smaller or mid-sized organizations, that we will provide a service bureau, a dashboard of content, and usage metrics that they can use in building a case for their advertising. And we will have legal support for those who want to engage in activities against the free riders or the pirates.

[Question about the appeal of these initiatives to AP members from a member at a China-based news site.]

Well, great, OK, so I would say, here are four things. One, first of all, the quality of the content is everything, and the level of content and the responsiveness and the number of story breaks in terms of market share has grown, not decreased, over the last couple of years. Two, I do think you’ll see these protocols and these approaches being driven internationally, and if you’re breaking stories, you can be part of the index and the curated piece that would drive the self-referring network and traffic to you very directly. Three, I think that we will begin to see more customized products, and there’s no question that Asia, in general, and the larger China would be top of the list there. So it is our hope that our fledgling Chinese-language translation service moves out and that perhaps there are specialty products that can be created there. Finally, not exactly knowing where you’re going with your content, but I think the multimedia opportunities that we’re putting together and the partnerships that we see could benefit and drive additional revenues to you and hopefully to both of us. So I’d be glad to work specifically on those with you.

[Question about how the AP is responding to the "mini-rebellion" among members who are complaining about price while competitors like CNN emerge.]

Jim, we had more of a rebellion from metro editors last year. We have cut prices to broadcasters and U.S. newspapers $75 million. So that’s one response. The other response is that we see all these new business opportunities as something that they can use to drive revenues as well. So our outreach with the news guide, which would— We’re told that the number one thing that they want, that a news organization wants is credit for when it breaks a story. And this solves that problem. Right now, they don’t get credit, so that’s the number one problem that will get solved. The number two thing is they want traffic from the self-referring network. You heard what i said about the Google self-referring network. It works really well for Google. We’re talking about creating our own self-referring network, and we know that the traffic increase will exceed 20% as a base within that network once we put it together. So that’s another aspect for that.

Our pricing has to be competitive. There are going to be more competitors. There are going to be fewer people who can afford us. This is a moment of tyranny in the marketplace. There are quality providers, and there are those who aren’t going to be able to sustain the revenues. We don’t expect to have the market share that we used to have. We are hopeful that we will be able to sustain the relationships among the places that you have suggested by, number one, breaking more stories than anybody and also with a reputation for accuracy and, two, customizing products. We have one network that was raising some questions about our pricing, and we went back, and we looked: AP correspondents appeared on that network 144 times in the last 10 months, and that was a service that they got without being charged extra for. That helped changed that conversation. So, again, it’s going to be customizing for specific needs. That is the era that we live in. It is not one product fit all. It is that multi-tier product system. And I think all of us in media made a mistake a few years ago by not understanding that sooner, and that is what we have to change.

[Question about how the AP is different from Factiva.]

Did you say Factiva? First of all, we create real-time content. We have the journalists out there. That’s archives. So I think the archive providers are going to be challenged by the technology going forward. One of the partners that we’re in negotiations with is going to make a serious play for real-time archives, and one of the new products that we’re talking about is making our database and our historical database, including some video and film that we have going back to 1903, more easily available. I can tell you that one of our partners now has 50,000 dead-end— or 50 million dead-end searches a month, trying to get historical stories from us. That looks like a heck of a market if we opened that up and made it available. So that is one of the major new products that we see, and we think that’s a significant positive revenue stream going forward.

[Question about how consumers will benefit from the AP's plans.]

Well, this is the moment for the consumer. I think, first of all, John, keep reading. But for the last 10,000 years or so, it has been mass media one way or another, the other way. And certainly, going back to the [?] age and looking at the stories being told there, somebody put the paintings there and drawings for all of us to interpret. We’ve gotten a little bit more transparent since then, but now you’re talking about two-way and choice and being able to self-program and self-edit. Some people are excited by the Kindle. I find the iPod life-changing. And books available. I carry two iPods, and if one of them blows on a trip to China, I have an opportunity, I have insurance, I have a backup. So I really think this is the moment for the consumer.

The downside is, wow, you have to work at it right now. And I guess I would say my hope is there would be some editing functions and you will be able to get a richer source of information that you want customized for you for a bit of a price, maybe less than what you’re paying now, maybe a little bit more, depending. But you will be able to get the stuff that you want, that’s more driven to [?] and something that you create. And I think that’s the big payoff.

[Question about Curley attending a conference hosted by the Chinese government and the implication of that decision on the AP's support of a free press.]

So you can take the approach that we aren’t going to talk and we aren’t going to work. I don’t think that’s a very smart approach. I think people know who I am and what I stand for, and iI testified last week at a U.S. senate committee on openness and access to government, and my stands are known. I have given, I think, what was called a very strong freedom of the press speech in China in advance of the Olympics when it looked like they weren’t going to be as open as they were. And so this is part of the Olympic platform that they engage with the world, and this conference is part of that, to keep engaging. I think it’s in all our best interests to keep that conversation going and to stand up for the values that we believe in. You don’t have such freedom to get there, and I hope to have the opportunity to figure out how to say that so that you can travel more freely and that our people can travel more freely. Looking at what’s happened in China since 1972, when AP reopened its bureau, things are far better in 2009 than they were in 1972. I am pleased to salute China for its total progress, and if you look at what’s happened in the last 60 years, there are a lot more people in China living better, and they deserve a lot of credit for that.

And so I think that, I hope to be a good guest, but again, I will look for the one opportunity to say what has to be said. We’re not fully there. And I say that in the United States. It’s a good conversation to have. We have governments in the United States that would like to keep more things secret, a lot more things secret than they should. And so it’s a battle to be fought everywhere in the world by all of us.

Where, indeed. Since its annual meeting in April, the AP has been vocal, if not precise, about taking a harder line and negotiating better terms with unnamed “portals” that pay to distribute the consortium’s content. He made a similarly vague reference in a speech today at the Xinhua Beijing Media Summit. But in comments earlier this week, Curley was far more specific than ever before about the portals AP is talking to, the nature of those negotiations, and what he really thinks about Google.

“I think we stand at an enviable moment where Microsoft and Google have decided to go to war, and we who produce content can start to figure out whether there’s an opportunity for us to help that sharing in a way that reverses the outflow of money from media and takes it back,” he told an audience of journalists at the Foreign Correspondents’ Club in Hong Kong. That much has been reported.

But afterwards, chatting with a few attendees, Curley said he was negotiating a new partnership with Microsoft under conditions more favorable to the AP and its members. It was all captured in an audio recording provided to me by a reader. Here’s Curley:

We are only going to work with those who use our principles. We are not going to work with everybody. So if you don’t agree to our protocols, if you don’t agree to give us real-time metrics, we aren’t going to work with you. So when I sat down in the portal negotiations, you know, I said, this time is different. You have got to be able to give us the metrics. This is not about money. We’ll get to the money part of the conversation later. If you want our content, these are the things you have to do. And that’s what I outlined. If you can’t do that, or if you won’t do that, let’s not waste time. And so far, everybody’s doing technical due diligence, including us.

One of those “principles” to which Curley referred is privileging the original source of news over outlets that merely report what’s being reported elsewhere. (It’s worth noting the AP does a lot of aggregation itself, but the point here would be to include its members and point readers to wherever the story originated.) In addition to “real-time metrics,” they’re also expecting partners to support the AP’s system for tracking use of its content along with other “protocols” I wrote about at length in August.

Someone asked Curley if Microsoft was willing to accept the AP’s demands. “They have said very strongly that they would,” Curley responded. A bit earlier, he said of Microsoft, “They know how to have a conversation.” And what about Google? “I’m not talking about Google,” he said. “We haven’t talked. We haven’t talked. We haven’t talked with them in any serious way.”

That was a bit of a stunner for those of us who have assumed the AP was actively renegotiating its contract with Google, which has a license to publish AP stories. It was also notable because Curley and other AP executives had previously gone out of their way to avoid naming names. Not so in Hong Kong.

“The Digital Millennium Copyright Act of the late ’90s came and basically enabled Google and the Google wannabes to do what they are doing,” he told the audience, speaking without a prepared text, “and it’s time for us to take back the night.” (That was an unfortunate, though I’m sure unintentional, reference.) He also said this:

If you go on to Google, normally the top 10, of the top 10 items appearing on Google, 9 of the top 10 will be part of a Google self-referring network. Everything in there is linked to some other part of Google. It’s really a brilliant business approach, and God bless them. And all that money has come to them, all 22 billion. Folks, they can share.

Curely didn’t explain what he meant by Google’s “self-referring network,” but he made repeated reference to it and said the AP was planning “our own self-referring network.” That could have something to do with the AP’s plans for topic pages to rival Wikipedia in search results. After his speech, Curley added, “The programs that we’re undertaking, we expect Google to have a directly competitive response.”

As for Microsoft, Curley repeatedly insinuated a looming partnership that would extend beyond the existing license to distribute AP content. He teased a new Microsoft product (other hints here) that might be part of the deal:

The search war is going to be fought on a qualitative dimension. So Google famously has the ten blue lines, and then everybody else is going toward more visual and what they believe is more accurate. So EveryZing is all visual, all right, and Microsoft this month has some new technology that it’s unveiling that will be much more visually dramatic than anything you’ve seen before. Multimedia in ways you haven’t thought about yet. We’ve seen it, we’ve seen the technology.

If Microsoft and the AP come to terms, it could have huge implications for consumers and publishers, affecting some of the biggest news sites on the web: MSN and, given their search partnership, Yahoo News. (Curley had nice things to say about the AP’s experimental partnership with Yahoo for its coverage of Justice Sotomayor’s confirmation hearings.) This is the search engine — or, really, search partnership — that AP executives have been hinting at, the one that would privilege “the original source or the most authoritative source” of news. At AP, it’s called the News Registry.

Google, for what it’s worth, says that it already favors original sources, but it’s hard to imagine they’d agree to make any adjustments to search results specifically for the AP and its members. That could lead to some brinksmanship in which the AP hitches its wagon to Microsoft and hopes the company is successful in its renewed competition with Google. Or the AP and Google could finally start talking.

I’m still putting together the transcript of Curley’s remarks and will share that with some of the audio when it’s ready. (I made a call to the AP’s press office shortly before posting this and will update if there’s anything to add. UPDATE, 1:58 p.m.: Paul Colford, director of media relations for the AP, said he would let Curley’s presentation “speak for itself” but gave me this statement for context: “As discussed this week by AP President and CEO Tom Curley in Hong Kong, the AP news registry, announced in April and now in development, will greatly improve and quicken the discovery of authoritative news produced by the AP and its member news organizations and empower them to better serve their readers and customers.”)

These are just clues, and Curley was intentionally vague at points, but they’re bigger and clearer clues than we’ve previously heard from the AP.

Because we obtained the document, the AP put me on the line with its general counsel, Srinandan Kasi, who spoke for an hour about various issues, including his office’s view of copyright, fair use, and the republication of AP material. Mostly, he tried to avoid any fine lines, but I still learned a lot, and you might be interested as well. This is obviously a crucial issue in the future of news, so I hope the following discussion adds a little bit of information — if not clarity — to how one major news organization is approaching copyright on the Internet.

Headlines and ledes

Tom Curley, president and chief executive of the AP, raised a ruckus last month when he seemed to tell The New York Times that using an AP headline that linked to the original article would require a copyright license. Among the more entertaining responses to that position was a blog whipped up by developer Andy Baio, who used the AP’s own RSS feeds to republish headlines, ledes, and URLs in the style to which Curley appeared to object. Baio called the simple act of protest Associated Repress.

I described the site to Kasi, who told me: “I think that the person doing that: wonderful. We celebrate free speech.” But what if that site carried ads? Could the use of AP headlines and ledes ever amount to copyright infringement? “At some point,” Kasi said, “the variables start to come together that, absolutely, it would be actionable.” We were getting somewhere: Although Kasi didn’t want to lay out a rubric for the AP’s legal strategy, the most important variables appear to be frequency of use and whether that use constitutes a significant, competing, commercial business. So, no, Baio probably wouldn’t get a cease-and-desist letter for his barely read site, even if it were littered with ads, but a more prominent aggregator could.

News aggregation sites

One complication in discussing the AP’s copyright stance is that almost all of the major news aggregation sites — Newser, The Huffington Post, Google News — are customers of the AP and fully licensed to carry its full-length content. But the AP has said it is reviewing those agreements, no doubt hoping to extract more revenue from some of the most popular news sites on the Internet.

If, say, Newser were to balk at a pricier contract and begin treating AP content the way it deals with other news organizations — headlines, excerpts, links — I get the impression that the AP would take action. “There’s no question that we see value in headlines,” Kasi told me, “and that value in the headlines is that we’d rather that it point to our publishers’ sites than some other site, for example, if all the other site is doing is simply cutting and pasting our content.”

That said, Kasi was concerned about appearing litigious and dialed back some of the AP’s strongest public statements. (Statements, I might suggest, that could be good leverage in renegotiating those contracts.) Thus:

When you look at the things that we’ve actually enforced or pursued, it’s a small handful of situations. Even the ones where there’s a lot of noise being made, it is to point out the kind of conduct, of systematic conduct that we want to have addressed. But if you really push it to the extreme of, “OK, how many do we legally enforce in a court of law?” It’ll be less than the number of fingers on a single hand.

Wholesale copying

It’s true that fighting even clear cases of copyright infringement is generally a losing proposition. (Or as Kasi put it: “If we pursued every one of these things, we wouldn’t be focused on our real business.”) Without prompting, he mentioned the approach taken by the Fair Syndication Consortium, which I wrote about in June, to extract a share of revenue from ad networks that serve ads on spam blogs. That’s a defter approach than litigation. And I think some critics of the AP’s copyright stance would be surprised to hear this from Kasi, referring to a hypothetical piece of content:

Someone decides that they just want to cut-and-paste — a simple example — they decide to take the entire thing. And they put it on their blog and say, ‘Can you believe that this is going on?’ … In that example, we may simply decide: It’s a one-off use, it’s got social policy considerations, it’s useful to have this information. Perfectly fine. But if someone is making a living of waiting for these things to evolve and every single time took it and put it elsewhere, did something with it, we certainly will look at it.

Last month the AP favorably settled its lawsuit against All Headline News, which had been rewriting the consortium’s stories en masse without even so much as credit. Previously, the AP has gone after grayer areas of reuse: Its legal department persuaded the Drudge Retort to take down six excerpts of AP content that ranged from 33 to 79 words. It sounds like the AP is now backing away from that view in favor of only pursuing sites that constitute significant competition. Here’s Kasi:

There is tremendous interest in the content. People want to use it, which is good news. And the bad news would be, if there’s bad news, some of it falls outside of the purview of our permissioning and our understanding of what we would accept as acceptable practice, because if left untouched, it actually erodes our business opportunity in ways that are actually quite material. So I don’t know that there’s a magic formula because, really, we strictly follow this on a case-by-case basis.

Of course, the ambiguity of that position combined with the more-strident public statements of AP executives can create what David Ardia, our friend and director of the Citizen Media Law Project, called FUD: fear, uncertainty, and doubt. Amateur bloggers — the very people that the AP says it doesn’t intend target — might be the most likely to shy away from using AP content because they don’t have Arianna Huffington’s lawyers to write a brief for them. So when AP chairman Dean Singleton said this…

[See post to listen to audio]

…he may have been speaking to Google News or The Huffington Post or spam blogs — who knows — but the people who heard him loudest were the individual bloggers who constitute the web’s circulatory system.

Encouraging use of AP content

That’s too bad, because I believe the AP’s senior vice president of global product development, Jean Seagrave, when she insists, in reference to their recent plans: “This is not digital-rights management that says no you can’t. It says this is how you can.” Recent posts by Doc Searls and Bill Rosenblatt have been good in this regard. And Kasi told me, “To characterize this entire thing as driven by an enforcement mode misses the point entirely.”

The AP would like to encourage use of its content — even full content — under terms that might not be so different from the APIs released by The New York Times and NPR. (Then again, it might be very different. The AP thus far hasn’t said what restrictions it will attach to its APIs.) I asked Kasi for an example, and he said that a mobile developer who wanted to include the AP’s articles or videos in an iPhone application could do so, probably without paying for access. Addressing the hypothetical developer, he said, “If this becomes a runaway success, I want to be part of this kind of business arrangement with you. In the meantime, if you want to experiment, go at it.”

Further information

To explore these issues in more depth, I’d recommend the 7-minute interview with Kasi’s counterpart at The New York Times, Ken Richieri, which we wrote about last month. He moves the discussion away from fair use and into unfair competition. But if you’ve got an hour this weekend, better to check out the full podcast, which includes some good stuff from the AP’s lawyer in its lawsuit against Shepard Fairey.

And in the meantime, we await the AP’s next move.

— Why The Associated Press plans to hold some web content off the wire
— How The Associated Press will try to rival Wikipedia in search results
— What The Associated Press’ tracking beacon is — and what it isn’t

I spent an hour on the phone with Srinandan Kasi, the AP’s general counsel, on Tuesday night, so if you have questions about it, feel free to ask, and I might have an answer. You can download the document, view it in Scribd below, or wade through the full text after the jump.

UPDATE, August 14, 3:35 p.m. Andy Baio, responding to a commenter’s request, kindly ran the document through an OCR scan. Thanks, Andy! Two images from the original are not included, of course, but here’s the full text:

Protect, Point, Pay – An Associated Press Plan for Reclaiming News

Content Online: Part I

The evidence is everywhere: original news content is being scraped, syndicated and monetized without fair compensation to those who produce report and verify it. AP’s legal division continues to document rampant unauthorized use of AP content on literally tens of thousands of Web sites. The problem is quickly spreading to mobile, where new applications are cropping up daily that do little more than repackage the efforts of AP and others, siphoning off consumers and revenue from those whose content is being exploited.

If there is good news, it’s that the issue is now getting broader attention. A group of newspaper publishers, coordinated by the NAA, has organized to review new media challenges and opportunities. An influential federal appellate judge, Richard A. Posner, has advocated for changes to copyright laws to protect content originators. The American Press Institute (API) has recommended models to migrate online content from free to paid. And an array of entrepreneurs and organizations has cropped up to suggest various technology and business solutions.

In the past three months, AP has solidified its plan to address the urgent need to regain control of news content online, beginning with AP news. Dubbed AP3P by Tom Curley for “AP Protect, Point and Pay,” this three-pronged initiative provides a blueprint for AP to move forward in the digital world and offers a model for restoring the worth of authoritative journalism online.

It is difficult to overstate the importance of taking action at this moment. With its traditional media customers under unprecedented financial pressures, AP simply can’t continue to provide the same quality of global news coverage under the current rules, where secondhand news gets most of the eyeballs. Emboldened by the uncertain state of the law around content use online, third parties are moving quickly to fortify their own positions. AP has both business and legal imperatives to assert its intellectual property rights, make affirmative efforts to protect them and create a structured way to enforce them. Taken as a whole, the AP3P initiative will give AP a firm footing on which the cooperative and its members can build a successful platform for innovation. Its success, however, depends on cooperation of its membership and careful execution of each part of the plan.

Described below are the foundational steps of AP3P that we believe are necessary for AP to gain control of its content. We’ve also outlined ways in which interested members (a “content coalition”) could take advantage of the protections AP is building for itself to protect their own content. In Part 2, we will address how AP and its membership could leverage the work done to gain control of our intellectual property to build content-rich products that create revenue streams for the news industry.

AP Protect

Core to AP’s plan is identifying and protecting its news assets. AP Protect takes advantage of the underpinnings of AP’s strategy that have been in place since 2006 when it created “eAP” to database, catalog and format its content for wide-ranging digital uses. The eAP database established a central warehouse for all AP’s content, across media type. Metadata coding followed, to organize the content by category, famous names and other attributes. This past year, AP developers created a standardized news “microformat” to ensure that every piece of content can carry key pieces of metadata, plus AP’s terms of use and a “digital beacon” to track usage.

The methodical build-out of eAP positioned AP over the past 18 months to begin offering a set of services to members through the Digital Cooperative program. Those services started with metadata mark-up of member content and now include the AP Mobile initiative, as well as content exchanges among members via AP Exchange. Work on the news microformat, known internally as “hNews,” as well as development of the beacon has now advanced to the point that it can serve as the basis for another valuable service, AP News Registry.

In a breakthrough announcement earlier this month, AP’s microformat was formally endorsed by The Media Standards Trust, a global research and development group devoted to fostering higher standards in news. The trust is calling on news organizations worldwide to adopt more consistent news formats for online content. Broad adoption of this simple open-source standard way of describing news and content rights could quickly change the balance of power for content owners online by providing a uniform way for search engines and others to surface it.

AP News Registry

AP News Registry is a way to identify, record and track every piece of content AP makes available to its members and other paying customers. What makes it different from other similar efforts is that it is being designed at the outset with a rights framework that will provide an enforceable way for AP to grant and monitor specific rights in its content.

When AP distributes a news item, that content will be wrapped in a container that will include rights information and a tracking beacon that will send reports back to the core database each time the item is clicked on by an end user. The beacon will identify each piece of content, the IP address of the content viewer, the referring Web server and the time of use. The content will also be wrapped in a simple piece of code that will travel with the content wherever it is posted on the Web and that spells out, in both human and machine-readable forms, what may and may not be done with the content. This is the microformat that is described above.

Because the News Registry’s active tracking beacon would not be effective if the beacon were removed, the Registry also has a backup enforcement system. Based on current Web behavior, it is safe to assume that some users will intentionally or inadvertently remove the beacon. A “passive” tracking service will crawl the Web searching for AP content and identify the publishing Web page, an image of usage and the time of discovery. Matches will be queried against the active tracking database, and unauthorized uses will be pursued.

AP News Registry will leverage AP’s in-house expertise in building and maintaining content databases, tagging content with metadata and distributing content as well as reuse some components that make up the eAP infrastructure. The registry will also require capabilities not currently available in the eAP platform. Those components either will be developed in-house or acquired through strategic partnering.

Opening the Registry to Other Content Owners

AP News Registry is a necessary addition to the AP infrastructure. However, the potential benefits of a news registry to the U.S. newspaper industry and other news content providers is equally compelling, and widespread adoption of the standards that would be part of AP News Registry would be beneficial to those trying to replace rapidly declining revenue from traditional media with online revenue.

In particular, AP News Registry could provide members with:

1) Better business metrics. A rich database of content usage and payment data would give publishers a more complete view of how their content travels and is consumed.

2) More protection. An automated permissioning framework would allow publishers to gain control over how their content is displayed and distributed online.

3) Flexible payment models. The rights framework AP is creating will support all transaction models that get adopted by the market, from micropayments, to paywalls to advertising.

4) Leverage with partners. The widespread adoption of common standards would improve the industry’s ability to negotiate with partners, including search engines and content aggregators.

AP is working toward launching AP News Registry in beta test for tracking and reporting of AP content by Nov. 15, 2009. In Phase 2, it would be expanded to include content from members of the cooperative, beginning in Ql 2010. A production-quality release for AP and member content is targeted for July 1, 2010, accompanied by on-boarding a large number of content owners.

AP Point and Pay

Implementation of AP Protect through the AP News Registry is fundamental to the next two P’s in the AP3P plan: Point and Pay. These are simply a shorthand means to describe a whole new approach AP is taking to online product development to address the revolution in how people consume information in the digital era.

The purpose of Point is to leverage AP news content and information management tools to harness online traffic in ways that reduce misappropriation, expand audience and deepen engagement. Pay seeks to maximize revenue by aligning the commercial terms under which AP’s products and services are available on various platforms. It also includes the ability to monetize premium content in new ways, such as highly targeted advertising support and pay models.

AP management also has been reviewing the offerings of several entrepreneurs in this area and plans to continue working with some.

Clearly, search and social networking have transformed the ways a new generation gets its news. In Part 2 of this report, we describe patterns of news consumption around the death of Michael Jackson that demonstrate just how far we have come from the bookmark-your-favorite-Web site model of just a few years ago. Each day, more and more news is consumed somewhere other than on traditional destination Web sites.

The AP3P plan involves segmenting AP’s online products to broaden redistribution of what we call “utility” content, i.e, the type and amount of news that is quickly and easily available from other sources, to limit or prevent redistribution of the kinds of information AP provides uniquely to ensure that hypersyndication does not drive down its value, and to create a “news guide” in the form of landing pages to serve as a focal point for discovery of authoritative sources of news.

Over the last few months, we have been conducting a careful inventory of AP’s online content, how it is used and which attributes, features and content types are easily replaceable by other sources and which are uniquely created by the cooperative. We are now in the process of defining a new online product set segmented for different customers and licensed uses.

Taken as a whole, we believe AP3P has great potential to transform the way AP and its members do business online and ultimately unlock enormous new revenue potential for the content industry as a whole. Making this transformation, however, will not be simple, and comes at a time when the industry and the economy as a whole is under huge financial stress.

We believe AP3P offers the opportunity to change the conversation with the portals and drive traffic Рand new revenues — to original journalism. In the next section, we outline some of the ways this might work.

Part 2: Protect, Point, Pay -The Revenue Opportunity

In the past few weeks, amid all the industry discussion of new digital business models, a new milestone was reached in Internet news consumption.

Michael Jackson died suddenly on June 25, and within 30 minutes, the news absorbed 25 percent of all Web traffic. Online news sites logged an astounding 4.2 million visitors a minute, according to the delivery network Akamai.

Two of the biggest beneficiaries of that traffic bonanza were Twitter and Wikipedia, a couple of digital natives that would have been viewed as very unlikely news competitors even a few months ago. Indeed, a new pattern of consumption was validated in the confusing minutes that followed the first reports of Jacko’s death: Users shared; they searched and they clicked on Wikipedia.

In the course of only a few hours on the first day of the story, the Michael Jackson page on Wikipedia received 1.8 million visits. By Friday, the total reached 5 million visits.

For those with long Internet memories, the new routine of Twitter-to-Google-to-Wikipedia contrasts sharply with the behavior of users in August of 1997, when millions loaded and reloaded bookmarked news sites to get updates on the death of Princess Diana, another celebrity icon of similar magnitude.

The traffic pattern also represents an important variation in the established drill since Sept. 11, 2001, when users began using Google and other search engines as a shortcut to news snippets.

Now the news may be shared before it is even searched. And, what’s more, in cases where famous people, places and things are involved, you will undoubtedly find Wikipedia in the mix, with its battery of standing pages that are updated continuously and built to send people where they are looking to go.

The Wikipedia page on Michael Jackson is not very pretty to look at, but it has more blue hyperlinks than black type. Forget the “wiki” method of community updating, the key to Wikipedia’s success is that its pages are designed to catch traffic, provide key information and then send users on their way to deeper engagement on the subjects they’re interested in. In the four weeks ending July 4, the key search terms delivering visitors to Wikipedia were mostly famous names – Billy Mays, Farrah Fawcett, Megan Fox, Ed McMahon, David Carradine, John Dillinger, Steve McNair, Karl Maiden – all in the news for a variety of reasons, along with Jackson.

(Note: Google News was the single largest beneficiary of traffic for “Michael Jackson” searches, attracting 7.1 percent of the clicks in the four-week period measured. Wikipedia was a close second, receiving 6.8 percent. YouTube was third with 6 percent. CNN.com, the only traditional media outlet in the top 10, was tenth at 1.5percent.)

If the rise of Google taught people to search rather than surf Tor news, the phenomenon now seems to lead more often than not to a comprehensive, perpetually expanding resource called Wikipedia. Year over year, Wikipedia has gained impressively in every aspect of audience measurement, from total visitors (nearly 60 million) to minutes per month (19).

The lesson is as much a cultural case study as it is a competitive alert. With the help of social nets like Twitter and resources like Wikipedia, users are gaining even more control over their news consumption than Google originally provided through search and, later, Google News.

Moreover, if past milestones like Diana and 9/11 are any indication of how situations can solidify habits, the choices being made right now, amid a parade of famous-name news stories, could become routine. To that point, there was evidence in the early minutes of the Jackson story that visitors to Wikipedia were struggling to confirm the death. Editors at Wikipedia openly parried with people trying to make updates on the Jackson page to call the death. “Once again, he is not dead, just stop,” wrote one exasperated editor in a posting to the page.

But clearly, based on the numbers, that debate – and search for truth – didn’t seem to discourage the onslaught of traffic to Wikipedia, which was functioning, at least for some large portion of the Internet audience, as an authoritative source working to verify an important news development.

What It Means for AP and the Larger Industry

Traditional news providers are not necessarily built to exploit this activity. Destination Web sites, open or closed, are the principal resources that traditional publishers bring to this competition. More and more, as recent events demonstrate, those tools don’t completely measure up to the job. In the Jackson episode, publishers lagged the field distantly. The Los Angeles Times, at the heart of the story, managed to capture 0.79 percent of the “Michael Jackson” search traffic. It was the lone American newspaper site in the top 20. These numbers suggest that more publisher content was likely consumed through search engines and aggregators than on site, which is the crux of the issue now facing publishers.

To be sure, sites have optimized their news pages for search, and some publishers are now pushing out news and links via Twitter. AP Mobile, the first news organization to create a ‘push” function on Version 3.0 of the iPhone, sent a news alert on MJ’s death that resulted in a 350 percent spike in AP’s mobile traffic that day. But the war is being fought on several fronts at once, presenting a range of responses to consider.

For one thing, the Wikipedia model of standing, authoritative pages could be challenged. Beyond that, the deep resources of publishers could be aggregated for easier access; the mobile channel could be opened more widely for search and social consumption of news; and new applications for delivering personalized content directly to users could be plugged into publishers’ sites. These are just a few of the moves that could tap into the new demand that’s emerging from increasingly empowered news consumers.

These opportunities align directly with AP3P strategy described earlier. The three P’s truly become capital P’s if they can harness the scope and scale of the Digital Cooperative’s content and user base. Currently, more than 1,200 AP members are taking part in the Digital Coop’s content enrichment program, and about 1,000 are signed up to take part in the first commercial application of it, AP Mobile.

The microformat standard that is part of AP Protect could give the industry leverage in broader discussions with search engines and integration with social networking tools as they are developed. AP News Registry could provide detailed usage data across the industry. AP’s concept of a news guide could become the first step in a wider platform for the news industry to create richer products and foster deeper audience engagement.

A large and diverse content inventory also should provide new opportunities for search, contextual and behavioral advertising. Virtually all of the new pay models that have been proposed to publishers in the past several weeks are based on those assumptions. The network effect of plugging such a sprawling inventory into search engines, portals, social nets, mobile channels and pay applications could be substantial, just based on the lessons now being learned from the Wikipedia phenomenon.

A recent whitepaper from the American Press Institute has independently arrived at similar conclusions. The Newmedia Economic Action Plan recommends a “consumer-centric” model for publishers that moves beyond traditional Web sites. The report advocates developing new applications for users, charging for some content and using IP protection to capture more value from syndication. It further urges collective action – including “joint supplier negotiation” ~ to extract fair compensation from digital distribution channels, such as search engines and aggregators.

As the Web morphs once more and the culture shifts even more decidedly in the digital direction, these are no longer just theoretical opportunities. The scope of coverage that can be exploited from AP and its members constitutes a clear competitive advantage in an increasingly fragmented news marketplace. As the smartphone applications already attest, the combined content base covers all geographic bases and multiple news categories. Based on consumption patterns and competitive trends, they represent real choices to be made, and soon.

Next Steps

We’ve focused in this document on the immediate steps AP proposes to protect its own content, and set forth the larger opportunity for a coordinated effort by a coalition of members of the newspaper industry, based on the same principles.

Clearly, the organization and operation of any coalition must reflect pro-competitive principles and must not violate applicable antitrust and competition laws.

Any coalition must begin with shared values. The obvious overall aim of our effort is to support and promote authoritative journalism while protecting original content from unlicensed use. While the Internet has opened up exciting new opportunities for others to provide firsthand accounts and to comment on and share news, the critical role professional journalists play in newsgathering, sourcing, fact-checking and curating has been undervalued. Even as much of the news online still originates with newspapers, consumers often end up reading second-hand and, often, inaccurate versions. The coalition would align around developing business models that support quality journalism and encourage consumer interaction with news in a rights-protected environment.

The focus, then of the content coalition, would become:

* Participation in AP Protect and AP News Registry;
* Standards and principles for dealing with content aggregators;
* Shared content monetization systems; and
* Combined product development and syndication opportunities.

When The Associated Press said last month that it was building a “news registry” of AP content, most reaction focused on the so-called “tracking beacon” that will monitor usage across the web. I use quotation marks because, well, those are metaphors for technology that’s still in development: The AP document we’ve obtained says the registry, set to launch on Nov. 15, will “require capabilities not currently available.”

But there’s nothing particularly magical about the beacon, which will amount to JavaScript embedded in the online feeds that are distributed to clients. So when you read an AP article on the New York Times website, a script running in the background will take note of that usage. (It’s unclear how news organizations like the Times, which is particularly neurotic about the weight of its pages, will feel about the script.)

Tracking readers

The point, of course, is to identify uses of AP and potentially member content that isn’t licensed. So if someone copied an article’s source code onto his own site, by hand or automation, the beacon would follow along and, according to the document distributed to some AP members, “send reports back to the core database each time the item is clicked on by an end user. The beacon will identify each piece of content, the IP address of the content viewer, the referring Web server and the time of use.”

I immediately flagged “IP address of the content viewer.” In recent years, the recording industry has used the IP addresses of downloaders to pursue legal action against people sharing music online, leading to lots of ill will toward the RIAA. That said, recording such data isn’t all that unusual. Websites using basic analytics software already record the IP addresses of their users.

When I asked the AP’s general counsel, Srinandan Kasi, about it, he said the AP wasn’t interested in monitoring who specifically reads their content on unauthorized sites: “In writing this” — he meant the document — “obviously, theoretically anything is possible. But what you actually make the final available piece is a different thing. This is simply: These are the capabilities that are possible.” Later, he added, “If at some point this business goes there, they’ll be completely transparent about it. There’ll be all the disclosure and compliance issues.”

Removing the beacon

There was another passage in the document that struck me as weirdly written:

Because the News Registry’s active tracking beacon would not be effective if the beacon were removed, the Registry also has a backup enforcement system. Based on current Web behavior, it is safe to assume that some users will intentionally or inadvertently remove the beacon. A “passive” tracking service will crawl the Web searching for AP content and identify the publishing Web page, an image of usage and the time of discovery. Matches will be queried against the active tracking database, and unauthorized uses will be pursued.

By “intentionally or inadvertently remove the beacon,” doesn’t the AP simply mean copy and paste the text of an article? While there’s been some innovation in the field lately, it’s difficult to imagine how the beacon could survive the magic of Ctrl+C, and that’s an obvious limitation of the tracking system. But referring to that as removing the beacon called to mind the anti-circumvention portions of the Digital Millennium Copyright Act, which criminalize attempts to get around copyright controls like burning an encrypted DVD. Now, I can’t imagine the AP would have a legitimate claim there, but I gave it a go with Kasi, who said, “You may be giving it a lot more heavy reading than” intended.

So why use that language? “We need to worry about that sort of thing because, right, Zach, we’ve seen that happen….If some of these formats get stripped out, including the mythical beacon, then we need to have a way of knowing and being able to address that.” (I had just referred to the beacon as mythical.)

Rhetoric and reality

I told Kasi that there seemed to be a persistent disconnect between the AP’s rhetoric on copyright and what it actually cares about. He acknowledged that the consortium had not always effectively communicated its intent: “It’s easy to think that, when you read ‘beacon’ and given the issues of some other companies and so on, you can immediately jump to the conclusion, ‘oh, this is a persistent cookie that’s going to track this user across all kinds of sites.’ No.” That was surely a reference to Facebook’s poorly received advertising platform, also known as Beacon.

The AP’s graphic explaining the beacon and a new microformat was easily mocked and labeled “magic beans” by prominent tech blogger John Gruber. In the clip from the actual graphic at right, doesn’t it look like a faceless news consumer will be deposited in a toxic waste receptacle?

But in talking to Kasi, I came away with the same impressions as Columbia Journalism Review’s Ryan Chittum: that the AP isn’t interested in broadly pursuing copyright claims against republication of its content. In fact, they’re hoping to encourage distribution in certain ways, and there’s plenty of innovative stuff in the microformats they’re adopting. The AP just needs to clear up what kind of “rampant unauthorized use of AP content” — that’s from the document — they want to combat.

And that’s the topic of my next post.

This is the third in a series of posts on the AP’s online strategy. Photo of beacon, by Brenda Anderson, used under a Creative Commons license.

Yesterday we revealed plans by The Associated Press to hold back some content from member websites. (Great discussion going on there, by the way.) The primary motivation of that initiative is search: AP material that resides on hundreds of disparate sites at the same time will hardly rate in Google compared to a single page with hundreds of links pointing to it. That’s a fundamental tenet of search engine optimization.

The same philosophy is driving their plan to build “news guide landing pages” that will aggregate the AP’s content around subjects, places, organizations, and people. Think of the topic pages on sites like The Chicago Tribune, BBC, and others — except that the AP will be harnessing its vast network of members and customers in what could amount to a brilliant SEO play.

The landing pages were first mentioned at the AP’s annual meeting in April, but further details haven’t emerged until now. In material distributed to some members last month, the news guide is described as “a central location to which headlines, promotional products and other content developed by AP could point.” What that will mean in practice is similar to what you find in the digital content of other news organizations: All references in AP articles to, say, Bill Clinton would link to the landing page with aggregated content and other material about the former president.

But, of course, those links to the landing pages would come from member news sites with excellent PageRank, the key metric used by Google to determine search results. (For instance, CNN, which carries AP content, has the maximum and extremely rare PageRank of 10.) It’s easy to see how the AP’s landing pages could, in short order, shoot up near the top of results for popular, news-related search terms.

Competing with Wikipedia

The document that I referenced yesterday, the one labeled “AP CONFIDENTIAL — NOT FOR DISTRIBUTION,” includes four pages of sharp, if widely accepted, analysis of how news is consumed today. Referring to coverage of Michael Jackson’s death, it says:

Two of the biggest beneficiaries of that traffic bonanza were Twitter and Wikipedia, a couple of digital natives that would have been viewed as very unlikely news competitors even a few months ago. Indeed, a new pattern of consumption was validated in the confusing minutes that followed the first reports of Jacko’s death: Users shared; they searched and they clicked on Wikipedia….

The Wikipedia page on Michael Jackson is not very pretty to look at, but it has more blue hyperlinks than black type. Forget the “wiki” method of community updating, the key to Wikipedia’s success is that its pages are designed to catch traffic, provide key information and then send users on their way to deeper engagement on the subjects they’re interested in.

(According to Wikipedia traffic statistics, the site’s Michael Jackson page has been viewed more than 24 million times since his death.)

There’s further discussion of Wikipedia’s dominance in search results, which is a product of all the external links pointing to Wikipedia and a variety of other factors. As Mathieu O’Neil Sage Ross, the editor of in-house newsletter Wikipedia Signpost, told me yesterday, “Google juice goes in, swishes around, doesn’t come out.” And that’s clearly what the AP would like to emulate, although it’s less clear how they’ll generate many links beyond member and customer sites. The document states flatly, “The Wikipedia model of standing, authoritative pages could be challenged.”

Proof of concept

Most of the AP’s landing pages would be automatically generated, although “editorial curation” would also be possible. That’s the model followed by sites like The New York Times, which has had decent success with Times Topics. In an internal memo late last year, Times editors boasted, “Many months of SEO labor…helped promote our Credit Crisis page to the prominence it deserves; search for “credit crisis” on Google and our Topic Page comes up first.” (It should be noted, though, that the Times page has been passed since that memo; it currently ranks behind The Crisis of Credit, a terrific 11-minute video describing credit concepts made by a young man named Jonathan Jarvis as part of his master’s thesis at a design college. Take from that what you will.)

The AP is also hoping it can convince members to join the project and have their content aggregated on the landing pages as well. (Of course, plenty of websites, citing fair use, do that already without any formal partnership.) The material distributed last month notes that the landing pages could “facilitate paid distribution of AP and member content,” although I don’t get the sense that’s a priority. As with the strategy I described yesterday, there’s a real question of balance here: It’s obvious what the AP gains, but members will obviously want to know what’s in it for them.

An SEO firm called EveryZing recently produced a trial run of the AP’s landing pages, according to their vice president for client services and business development, Bob Fogarty. EveryZing has also created topic pages for Fox News and Newsweek. In the latter case, the project is actually called…Newsweekopedia.

Much of this strategy follows what Google vice president Marissa Mayer suggested in recent testimony to Congress. It’s also in line with the research of Matt Thompson, whose currently online community manager at the Knight Foundation. And I wrote about these ideas when Google News began including Wikipedia in its search results.

That’s one revelation from a document we obtained (labeled “AP CONFIDENTIAL — NOT FOR DISTRIBUTION”) that offers new insight into how the AP is planning to reinvent itself on the Internet.

The seven-page briefing, entitled “Protect, Point, Pay — An Associated Press Plan for Reclaiming News Content Online,” was distributed to AP members late last month. It provides greater detail about the tracking device that will be attached to AP content and describes their plans to create topic pages around news stories to rival Wikipedia and major aggregation sites. And in an hour-long interview last night, the AP’s general counsel, Srinandan Kasi, also shed light on how the consortium views reuse of its material across the Internet.

I’ll be wading through the document and what we’ve found in a series of posts beginning today. (You can subscribe to our RSS feed or follow us on Twitter if you don’t want to miss anything.) We’ll eventually post the full document, too. And as we go, feel free to comment and ask questions so we can flesh this out. I think you’ll find this stuff applies to all news organizations, not just the 1,500 newspapers that own the AP.

“Utility” vs. “unique” news

There are several intertwined issues to address, but the first bit of news that caught my attention was this passage, which refers to a previously described program they’re calling “AP Protect, Point and Pay,” or AP3P:

The AP3P plan involves segmenting AP’s online products to broaden redistribution of what we call “utility” content, i.e., the type and amount of news that is quickly and easily available from other sources, to limit or prevent redistribution of the kinds of information AP provides uniquely to ensure that hypersyndication does not drive down its value, and to create a “news guide” in the form of landing pages to serve as a focal point for discovery of authoritative sources of news.

That distinction between utility and unique content immediately made me think of fair use. The AP’s position on copyright has recently been the subject of heated debate — get a flavor for it here — and this seemed like a new wrinkle in their thinking about the issue than hadn’t been previously voiced.

So was that a fair read of the passage? No, said Kasi, the AP’s general counsel, “it’s not to suggest that there’s a legal distinction.” (Though the AP has generally stopped granting interviews about their copyright stance and wouldn’t speak to me about it last month, Kasi got on the phone after I informed them we were writing about this document.) It’s less about law than search engine optimization and the link economy.

Keeping “unique” content in one place

Utility content, Kasi told me, might be your traditional breaking-news story: “So a headline item that says, ‘Mid-air collision outside of New York and tourists die,’ let’s say. You can imagine, in the New York area, there are lots of media covering that story.” The AP would treat that content as it always has, putting it on the wire for members and customers to publish on their own sites. But other pieces of content — say, an infographic or a sidebar documenting the history of similar collisions — would be held off the wire and published only on a central AP site, Kasi said.

The plain-vanilla wire story, meanwhile, would point to the more in-depth material in the form of a link. He explained: “We have unique pieces of data, maybe, or we have a unique visual narrative, a graphic. We have unique photos, a photo gallery, and so on. How can you use some pieces of content to drive traffic to other pieces of content? That’s really what’s being addressed here.”

I should note that this has nothing to do with the AP’s print offerings, and the AP has various online feeds for customers that could see different changes. It’s a work in progress, and the document notes, “We are now in the process of defining a new online product set segmented for different customers and licensed uses.” Paul Colford, director of media relations, told me today that the state wires would not be affected.

The link economy

Plenty of people have observed — and the AP surely understands — that the consortium’s 163-year-old, print-centric methods of syndication don’t really make sense online, where a link can do the work of distributing content. That sounds like the impetus for this rethinking, but it will surely raise hackles among AP members accustomed to publishing that wire content on their own sites (not to mention selling ads against it). Kasi said members will appreciate the change because keeping unique content in one place will, among other benefits, improve its position in search results and maximize traffic. Here’s the key part of our conversation:

…the ability now to be able to make a decision that says, this is not something we want to generically put on the wire and send to everyone to publish everywhere. We instead think this would be useful for people to use as supplemental, to enrich their storytelling. But it’s available for them to be able to point to. And the reason to do that is, then you have a bunch of links that point to a particular piece of content. You have better search outcomes. You have better exploitation of the link value, if you will, to that piece of content. So there’s an ability to think of that piece of content differently because you’re trying to maximize traffic as compared to other content where the benefit is really about getting the initial engagement.

Well, that’s fascinating, and “link value” is exactly the term here. The AP would essentially be relying on its vast network of members to provide search engine optimization for its most unique content. (I’ll have more on that in the next post about the AP’s plans for standalone topic pages.) The plan is much more in the spirit of the web than traditional AP syndication, but I wonder how members will react. If you’re an AP member reading this, I’d love to hear your thoughts — in the comments, by email, or at (617) 496-6595.

It also raises the question of what a wire service should like 200 years after the invention of the telegraph. As Kasi told me:

Publication is not in isolation. It’s in the context of a vibrant ecosystem, platforms that are evovling all the time, consumptive patterns that are being defined and redefined dynamically…So utility content is simply saying there are forms of content or pieces of content whose utility would be to be able to drive pointing value, meaning directional value, to say, here’s a deeper story that has more of the story. It could be a link, it could be a tweet.

Clearly, the AP’s plans for fitting into that ecosystem are still a work in progress, so here’s your chance to offer feedback. And as I’ve been saying, more to come.