That Alexa isn’t good at giving reliable statistics is well known in IT. Any website like Alexa that tries to estimate traffic to a website in a similar way via indirect measurements will encounter the same issues it has. Those that work in IT often know what those issues are and know what the consequences are for the data that’s gathered via those methods.

The main issue that Alexa has is that it gathers the data it uses via users that installed the Alexa toolbar (or a toolbar that passes information to Alexa). This has as a result that demographic, used browsers, and even the country visitors are from influence the statistics that Alexa gathers about a website. This can introduce serious artefacts and biases into the collected data and basically makes Alexa data worthless. At best it can give you an idea about how well a website is doing, but that doesn’t mean that what you’re seeing matches reality. I’ve already written a far more detailed blog post about how Alexa works and why you never should rely on the data it provides; it’s just too unreliable.

The blog post that I wrote about Alexa were all sparked by Anthony Watts using Alexa data to claim he’s doing better than his competition. His website is certainly big and it could very well be the case that he is outperforming his competitors. But Alexa is not the tool that you can use for determining if you are doing better than your competitor.

One good example is how my site is doing compared to the website HotWhopper. This website is run by Sou and debunks claims made by climate science deniers, it often touches on similar topics as I do. We share some of the same target audience and in that sense this website is one of my competitors.

If you compare my Alexa ranking with that of HotWhopper you’ll notice that I’ve passed HotWhopper. According to Alexa I now have a ranking of 858,248, but HotWhopper has dropped below me with a ranking of 897,008 (lower number is better). What you also clearly can see is that HotWhopper is dropping in the Alexa ranking, at its height it had a ranking of 382,166. So clearly HotWhopper readership is on the decline while I’m at least keeping my readership. From this data you would also expect that I’m in the same ballpark with the number of visitors and pageviews that I get. Right? Wrong.

If we take a look at the Google Analytics data for both websites a very different story emerges:

The first graph is the number of pageviews my website gets on a monthly basis, the second one is for HotWhopper. Last month Real Sceptic had 4,146 pageviews and HotWhopper had 58,901 pageviews. HotWhopper had fourteen times the number of pageviews Real Sceptic received. I’m not even in the same league as HotWhopper, let alone in the same ballpark.

You see the same when you look at the number of visitors for each website (as far as I can tell Alexa weighs this more heavily in their ranking calculations):

Last month Real Sceptic had 3,015 visitors while HotWhopper had 28,320 visitors. HotWhopper had nine times more visitors than Real Sceptic.

What makes it even worse for the Alexa data is that readership for HotWhopper isn’t declining, it’s increasing. Readership did reach a plateau for a while, but the decline you see in the Alexa data isn’t there. That’s just an artefact that’s present in the Alexa data. This is why you never should use Alexa statistics to do any serious comparisons. The data Alexa uses doesn’t have to match reality at all.

I work for a software development company where we develop and maintain complex web retail software. We also link these retail websites to online campaign and tracking software for our clients. It has made me well aware of the limitations of certain technologies and products. So it’s no surprise for me that Alexa got the rankings for HotWhopper and Real Sceptic wrong. What did surprise me is how wrong Alexa was, I never thought it would deviate this much.

But I think I know why Alexa got it so wrong. It’s probably because I’ve written about Alexa a couple of times and I have a high ranking for my Alexa content in Google searches. My website is often displayed on the first page in Google, I can see that with the statistics I have in Google Webmaster Tools. What you need to realize about this is that people who search for information about Alexa rankings tend to be Alexa users, which means that there’s a good chance that they also have the Alexa toolbar installed.

Alexa gives the hint this is happening with the keywords it lists for Realsceptic (on the left) and HotWhopper (on the right):

This keyword top 5 doesn’t match any of the lists that I have for the most used search terms for my website. The list provided by Alexa also doesn’t match with the amount of traffic pages get on Real Sceptic. This is probably the reason Alexa thinks my website is doing better than HotWhopper. It also shows how easy it is to introduce a bias into the data Alexa gathers and how unreliable this makes the statistics Alexa generates.

These inaccuracies in Alexa statistics is why my colleagues laugh when I tell them that some use Alexa data for comparing websites. I hope you now understand why they don’t take Alexa serious and why you shouldn’t use Alexa statistics.

Note:

I wouldn’t have been able to write this if Sou hadn’t been so generous to share her Google Analytics data with me. As a thank you please do not forget to visit her site and check it out.

Update 2014-02-08 @ 18:07:

Sou also wrote about the discrepancy between her Alexa ranking and the actual traffic she’s getting. It’s a good supplement to what I wrote as it gives more detail into what her traffic statistics were doing compared to her Alexa ranking.