

Wall of Spam. Photo by freezelight on Flickr. Some rights reserved

There is, in the words of Jeff Atwood, "trouble in the house of Google". It's not unrest within the company that he's talking about, though; it's externally among users who are beginning to find that when they try to do searches to evaluate or buy consumer items - such as dishwashers, or iPhone 4 cases - or to find a site that will give them some useful answers, that Google's results are awash with spam.

In fact, the problem that plagued the first generation of search engines such as Altavista now seems to be gaining traction on Google, which outdistanced those earlier rivals precisely because it dumped the spam so effectively.

Paul Kedrosky ("investor, speaker, writer, media guy, and entrepreneur") noted in a plaintive post in mid-December that

"Over the weekend I tried to buy a new dishwasher. Being the fine net-friendly fellow that I am, I began Google-ing for information. And Google-ing. and Google-ing. As I tweeted frustratedly at the tend of the failed exercise, 'To a first approximation, the entire web is spam when it comes to appliance reviews'."

Kedrosky noted that "Google has become a snake that too readily consumes its own keyword tail. Identify some words that show up in profitable searches - from appliances, to mesothelioma suits, to kayak lessons - churn out content cheaply and regularly, and you're done. On the web, no-one knows you're a content-grinder."

He also adds that "Google has to know this. The problem is too big and obvious to miss."

And indeed Google does know about this. Jeff Atwood at Coding Horror - one half of the creators of Stack Overflow (the other half being Joel Spolsky) and the other "Overflow" sites (which let people ask questions on a topic and the better ones get voted up - like a, ahem, better version of Quora) - had already noticed how sites which scrape Stack Overflow and its siblings actually rank better on Google than the original.

Now, Stack Overflow allows scraping, as long as there's a link back to the original (which cannot be a nofollow); yet even with this, it ranked below the scraping sites.

Atwood consulted Matt Cutts, Google anti-spam king:

"We did a ton of due diligence on webmasters.stackexchange.com to ensure we weren't doing anything overtly stupid, and uber-mensch Matt Cutts went out of his way to investigate the hand-vetted search examples contributed in response to my tweet asking for search terms where the scrapers dominated. Issues were found on both sides, and changes were made. Success!"

Except it isn't really success. It's a temporary respite. As Atwood points out moments later, "Anecdotally, my personal search results have also been noticeably worse lately. As part of Christmas shopping for my wife, I searched for 'iPhone 4 case' in Google. I had to give up completely on the first two pages of search results as utterly useless, and searched Amazon instead."

And it's not just purchasing non-essential items either. It's academic research too: in a guest post on Techcrunch, Vivek Wadhwa of the University of California at Berkeley assigned students to do venture capital research:

"I instructed my students to use Google searches to research each founder's work history, by year, and to track him or her down in that way.

"But it turns out that you can't easily do such searches in Google any more. Google has become a jungle: a tropical paradise for spammers and marketers. Almost every search takes you to websites that want you to click on links that make them money, or to sponsored sites that make Google money. There's no way to do a meaningful chronological search."

They ended up using Blekko, which is interesting because (1) it has a new approach altogether (well, new-ish) and (2) it's pretty hot in picking up what it thinks is spam.

Even the Shoeblogger was pained enough to break from channelling Manolo Blahnik to complain about scrapers ranking higher - sometimes consigning the original to the second page.

At which point you might point to Wikipedia, whose content is regularly scraped (allowed under its licence) by other sites, yet which (almost?) always outranks them. The answer seems to be, from various forums, that Wikipedia has got a special pass in Google's algorithms - as in, the engineers long ago made it a special case and downgraded sites that seem to just reuse its content.

(Matt Cutts hasn't been available via Twitter or other methods to comment on this or related matters, though several people have prodded him about it there; he seems to be taking some time away from it.)

The reason why this has happened is obvious: Google is the 900-pound gorilla of search, with around 90% of the market (excluding China and Russia), and there's an entire industry which has grown up specifically around tickling the gorilla to make it happy and enrich the ticklers. I've not come across anyone who describes their job as "Bing results optimisation", nor who puts that at the top of their business CV. Well, I'm sure there are people inside Microsoft whose job title is exactly that. But not outside it.

There are two lines of thought on what happens next.

1) Google comes back from the Christmas break newly determined to fix those damned scraping sites that don't originate content, because it says in its own webmaster guidelines that "Google will take action against domains that try to rank more highly by just showing scraped or other auto-generated pages that don't add any value to users."

The only value those scrapers add, in fact, is to Google, because they display tons of AdSense ads. (Well, you can make a fair bet that they aren't Bing's equivalent.)

Wait - the scrapers that dominate the first search page, the place from which 89% of clicks come (for only 11% of clicks come from the last 990 results out of the first thousand, or at least did in 2006, a number that has probably only shifted down since then) all benefit Google financially, even while it sees market share improvements? That's not quite the disincentive one might have hoped for that would make Google act.

2) People start not using Google, because its search is damn well broken and becoming more broken for stuff you care about by the day. This could happen. The question is whether it would be visible enough - that is, whether enough people would do it - that it would show up on Google's radar and be made a priority.

Over at Hacker News, the suggestions in the comments echo the idea that Google's search really isn't cutting the mustard any more ("vertical search" is the new watchword). Which means that really, Google does need to implement method (1) above. It might not notice if a few geeks abandon it - but once the idea really gets hold (as it will through the links they offer and comments they drop) that Google's search is broken, then the rout begins.

I haven't been able to get a comment from Google on this, though I'm sure it would run something along the lines of "Google makes every effort to make its search results the best and takes seriously the issues raised here."

Update: Google responded to this article: "Google works hard to preserve the quality of our index and we're continuing to make improvements to this. Sites that abuse our quality guidelines or prove to be spam are removed from our index as fast as possible". (For clarification, I didn't initially contact Google as it was a public holiday when I wrote the original article. Matt Cutts did not respond to Twitter contact as he is on holiday, Google says.)

It would be crazy not to. The question is whether it really can make a difference.

Bonus link from November, with Technology Review looking at the launch of Blekko: the (slightly judgemental?) Why Google is Choked With Spam.

What about your experience: does Google still work as well as it did for you? And if not, what are you using instead?