It’s time for another security report. You know, those reports that tally vulnerabilities, and then plot or graph them in such a way that their benefactors or clients come out most favourably. Ok, that might be a bit cynical, but fact remains that there is usually something wrong with such reports. The one that’s making its rounds across the internet today is certainly one of them. According to IBM, AIX is the most secure operating system, and Mac OS X the least secure. Not only is the report rather slim on details when it comes to operating system vulnerabilities, it seems like most websites reporting on this story have misunderstood what it was about.

The table making its way onto various websites can be found on page 40 of the report. Websites copying this table state that it lists the percentages of unpatched known vulnerabilities in each operating system (here or here) – but reading the accompanying text blob, I can’t help but think that’s not what the table lists. If I’m reading it correctly, the table lists the percentages per platform of the total amount of disclosed vulnerabilities. The text blurb:

X-Force tracks vulnerabilities by platform and has produced metrics this year

to show the operating systems with the most disclosed vulnerabilities. The following chart shows the operating systems with the most vulnerabilities documented in 2008. The top ten operating systems account for nearly 75% of all vulnerability disclosures affecting operating systems.

That seems pretty clear to me; they are looking at the share each operating system has of the total amount of reported vulnerabilities. Still, that doesn’t mean this report has any significant meaning on this specific subject. As has been said many times before – just counting vulnerabilities isn’t a good measure of security.

At the end of the day, what matters is not only quantity, but also quality. Any report on security that does not take severity into account is a little hard to take seriously when it comes to making general statements about a platform’s security record. The report does tell that 1% of the total amount of reported issues has the critical severity rating, but it doesn’t break it down per platform.

This report by IBM sates that AIX is the most secure, but by not having any information on severity, this conclusion reeks of “We from IBM recommend IBM…”, greatly reducing confidence in this report.