Web being carved up into info plantations

Nicholas Carr

It seems to be following the pattern that has always characterised popular media.

SEARCH AT Google.com on evolution or Iraq or AIDS, and the same site will appear at the top of the list of results: Wikipedia. Alter your search into one for John Keats or Muhammad Ali or Christianity or platypus or loneliness, and the same thing will happen. Pacific Ocean? Wikipedia. Catherine de Medici? Wikipedia. Human brain? Wikipedia.

In fact, if you Google any person, place or thing today, you're almost guaranteed to find Wikipedia at or near the top of the list of recommended pages. Despite its flaws, the amateur-written encyclopaedia has become the world's all-purpose information source. It's our new Delphic oracle.

The hegemony of Wikipedia is only the most striking manifestation of a broad and unexpected phenomenon: The world wide web is shrinking. I don't mean there are fewer sites than there used to be. On that measure, the web is bigger than ever. I mean that more and more of our time online is being spent at an ever-smaller number of megasites. The wilds of the Internet are being carved up among a handful of vast information plantations.

Web statistics tell the tale. The blogger Richard MacManus recently examined trends in online traffic over the past five years. He found that between the end of 2001 and the end of last year, the number of Internet domains expanded by more than 75 per cent, from 2.9 million to 5.1 million. At the same time, however, the dominance of the most popular domains grew substantially. At the end of 2001, the top 10 websites accounted for 31 per cent of all the pages viewed on the Net. By the end of last year, the top 10 accounted for fully 40 per cent of page views.

Much of the increase in traffic consolidation can, as Mr. MacManus points out, be attributed to the growth of social networks such as MySpace, Facebook and Bebo. In 2001, MySpace didn't even exist. Now, it is the most visited on the Net, accounting for a whopping 16 per cent of all page views.

The concentration is happening in other areas as well. Take search engines. According to Hitwise, Google now handles 65 per cent of all web searches, up from 58 per cent just a year ago. Despite aggressive attempts by Yahoo!, Microsoft, and Ask to boost their own shares of the search market, Google continues to widen its lead.

If Google is an example of the shrinking of the Net, it's also one of the causes. Because the search engine determines a site's relevance based on its popularity, as measured by, among other things, the number of other sites that link to it, it has had the effect of turning the web into a giant feedback loop. The more popular a site becomes, the more it comes to dominate search results, which ends up funnelling ever more links and traffic to it.

On the Internet, the big get bigger. It wasn't supposed to be like that. When the web arrived in the early 1990s, it was heralded as a liberating force that would free us from the confines of gated communities like AOL and Compuserve. The Internet was supposed to be an open, democratic medium, an information bazaar putting individuals on the same footing as big companies.

In the end, though, the Internet seems to be following the same pattern that has always characterised popular media. A few huge outlets come to dominate readership and viewership and smaller, more specialised ones are consigned to the periphery. Most of the largest sites are now in the midst of acquisition sprees or expansion programmes intended to extend their dominion. Just last week, MySpace announced it would buy Photobucket, the largest photo-sharing site; Facebook said it would expand into the classified advertising business; and Google chief executive Eric Schmidt said that his company has been acquiring small companies at the rate of one a week to build out its portfolio.

It may be that Internet users will revolt against the dominance of the mega-sites. But I wouldn't bet on it. All the signs point to a continuing concentration of traffic within the fences of the new information plantations. —

© Guardian Newspapers Limited 2006

(Nicholas Carr's next book is called The Big Switch. He blogs at roughtype.com)