There is one huge, almost infinitely wide memory gap in our culture that can be summed up with this question: Where does the Internet go when it dies? Not the whole Internet, but the individual websites and pages that every day are modified and deleted, discarded and cached. Who can a journalist turn to when needing to look up the older version of a website, a retired blog, or a deleted Facebook post?
It turns out, not many people. The hole that Nexis plugs for academic papers and the newspapers of the world has few equivalents online. The once excellent Wayback Machine: Internet Archive — an attempt at a complete, Library of Congress-worthy web archive — is now fairly useless in today’s social-media driven web world, storing a slipshod record of photos, multimedia, and basically anything that’s not Web 1.0, and on top of that, taking up to a year for updates to appear in its index after its spider has crawled a site.
This election season, as candidates propped up their digital campaign booths online with Twitter feeds and new, snazzy websites, Darrell Silver, founder of the Perpetually Public Data Project, realized this was actually kind of terrifying. For all the thousands of reporters following candidates’ buses and rallies, there was no mechanism to follow the campaign trail online. Anything pledged on a candidate’s website could be wiped out with the click of a mouse — and without so much as a peep.
To fill this collective memory hole, for the 2010 midterms, Perpetually archived the websites — and the Facebook, MySpace, and Twitter accounts — of every politician it could find: all the major candidates for all 435 House and 37 Senate races. And it archived every change at every second of every minute: Flash, blog posts, photos, whatever — with the exception of YouTube videos, which have a copyright conflict, and he decided to discard.
The result has been a great experiment that’s made at least one news splash and brought the technology onto the Huffington Post. After the election, Silver added every congressperson — newly elected or not — and every governor to Perpetually’s database. Imagine the difference at some point in the future: Anyone will to be able to zoom into any point in the past, load up a politician’s website, and see how things stood on any given day in any given year. And then, a few clicks more, to be able to scroll through the politician’s history and to create a larger story about the politician over a wide, career-length timespan. “We’re trying to be the undebatable reference point for the source material and the proof of what happened when,” Silver said.
The site, thus far, has given that goal its best shot, although it has got a way to go. In terms of the breadth of his archive and the depth of its storage, Perpetually is peerless. ProPublica’s Versionista is perhaps his closest competitor, but for now it doesn’t track candidate sites, only the Whitehouse.gov site. Moreover, the Versionista platform shows only specific html-coded changes, so it monitors mostly text and lacks a screenshot archive, a complete record of images, and interactive elements. Silver had many of the same critiques — lack of interactive elements, a generally superficial archiving — for the Wayback Machine (not to mention its dinosaur lag-time in updating its archive).
If his database is the gold standard for Internet archiving, on Perpetually’s front-end — the site visitors use to navigate the database — the story was less nice. In the rush to get things up, a shaky vision for the project created the odd mess of creaky widgets, bridge-to-nowhere links, and brilliant data-archiving that were the site for the few weeks it was live.
The site as it existed is a good case study in how a great concept with poor execution can crash and burn — and then potentially redeem itself. In Silver’s defense, he had little time to get things together. Perpetually began archiving candidates’ sites in June — not knowing exactly what he would do with the data — and managed with only a team of five to have a website up for the general public by early October.
But it was painful to use. You could see that some idea, some vision, was at work, but it was hard to see how whoever was behind the thing actually thought they could pull it off. Links broke, videos gave errors, and community was non-existent. The annotations page — an absurd Tumblr-style page with no entries limit — with a larger user base would have sent an average laptop crashing to its knees, and text-diff mode gave an html page read-out, a fairly frightening chunk of words and symbols specializing in alienation and confusion.
The good news, though, is that as far as Perpetually’s future is concerned, its history doesn’t matter: Perpetually has gone into hibernation for a complete overhaul and redesign. “One of the things I learned is that there’s a huge amount of interest of tracking politicians who are nationally or locally interesting,” said Silver. “But you have to provide a lot better and more immediate goals and feedback.”
Silver’s looked at the Guardian’s expense-scandal tracker for ideas on how to use better crowdsourcing mechanisms, like promoting what’s interesting and highlighting top users. And he likes Versionista’s feed-subscription service that gives users instant notification of changes made by a specific candidate. Silver — who is far more of a tech geek than politico — just did not understand a political junkie’s motivations, but he’s clearly getting there, and it is likely that his redesign will showcase a savvy pastiche of social media tools he culls from around the Internet.
If these changes make the site user-friendly, journalists should rejoice. As it stands, the tools available to journalists to retrieve information about a candidate’s online campaign trail are unreliable and incomplete, jeopardizing online accountability. We’ve already seen how easily that can happen. Perpetually provides a common resource to circumvent this problem. “That ability to see, to go beyond the Wikipedia summary is vital to…the history to what this person is saying,” Silver put it.
Non-journalists — whoever these people might be — have reason to celebrate, too. It’s easy to imagine a day when early website incarnations have Americana value, like The Museum of Moving Image has archived online has rediscovered in presidential TV ads. The White House itself seems to be getting in on the idea. It’s created “frozen in time” portraits of previous administrations‘ websites, anointing them with the exclusive “.gov” extension along with the program.
These are big ideas — an institutional memory hole, the making of a blog into classic memorabilia — and the opportunity is there for Silver to make them a reality. But before any of that happens, he still has to get the details right. He says has set forth three things he believes his audience wants and that a remade Perpetually must do for them:
“People want to know about significant changes and want to research the candidates they don’t know about. [They] want to be kept up to date and want a way to do that really easily. The third thing they want is to participate. They all want to improve the election process and want to discuss and do it in an efficient way.”
News organizations, take note: Leading up to 2012, Perpetually’s a site to watch.