Jeffrey Jones, a researcher and the Security Strategy Director at Microsoft’s Trustworthy Computing group, recently posted a report (PDF format) that was featured in the CSO online magazine. The report compared the security track records of both Internet Explorer and Firefox, including both older (IE 6 and Firefox 1.0) and newer (IE 7, Firefox 1.5 and 2.0) versions. Jones came to the conclusion that, contrary to popular belief, Internet Explorer has experienced fewer security vulnerabilities than Firefox over the same periods of time.

Now, I can already hear some of you anxiously mashing the "Reply" button in order to point out that Jones' position as an employee of Microsoft has biased his results and thus they cannot be taken at face value. And Mozilla chief evangelist Mike Shaver has some serious problems with Jones' methodology; more on that below.

Jones has anticipated negative reactions and has encouraged readers to challenge his assumptions, analysis, and conclusions by pointing out flaws in his methodology. Jones collected and cross-checked his data from a number of sources in order to ensure its accuracy. Disclosed vulnerabilities for Internet Explorer were compiled from Microsoft security bulletins and for Mozilla's own bulletins for Firefox. Both sources were checked with the National Vulnerability Database (NVD) and sites such as Securityfocus.com, the BugTraq mailing list, Secunia.com, and Securitytracker.com.



Vulnerabilities for IE and Firefox. Data source: Jeff Jones.

The report looks at vulnerabilities over the last three years, breaking them down into High, Medium, and Low severity categories. Since November 2004, Microsoft has fixed 87 total vulnerabilities in Internet Explorer 6 and 7, while Mozilla has issued 199 fixes to Firefox 1, 1.5, and 2.0. In addition to looking at the total number of vulnerabilities, Jones broke the issue down into four categories of users: IE users who upgraded to new versions right away, IE users who held off from upgrading as long as they could, and Firefox users of both types. Interestingly, delaying upgrades made very little difference to the total number of vulnerabilities on either browser. As shown on the chart, Internet Explorer had significantly fewer vulnerabilities during this time period.

Jones also looked at the trends of vulnerabilities found in new software during its first year of availability: are Microsoft and Mozilla getting better or worse at creating secure software with fewer holes in it? He found that IE 6 had 26 vulnerabilities found in its first year, while IE 7 did slightly better with 17 (IE 7 on Vista, thanks to its improved security capabilities, had only 14). Firefox had a slightly different trend, with version 1.0 experiencing 66 vulnerabilities, version 1.5 upping the trend with 77, and version 2.0 beating both with 56. Still, these numbers are all higher than IE.

Of course, measuring fixed vulnerabilities is not the whole picture: what about unfixed holes? Jones researched this and found 24 disclosed but unfixed holes in Firefox 2, versus 21 in Internet Explorer 7. Jones did not look at earlier versions of either browser, nor did he attempt to estimate the number of undisclosed and unfixed vulnerabilities, which are by their very nature almost impossible to count. He did also not attempt to count the number of sites that attempt to exploit these vulnerabilities, which would also be a difficult task. While some may still think that malware only targets IE, the truth is that most malware today does a simple browser version check and loads the appropriate exploit code for IE or Firefox. Gone are the days of ActiveX-only malware.

A rebuttal from Mozilla

In a blog post today, Mozilla chief evangelist Mike Shaver savaged the Microsoft study. Shaver points out that Microsoft bundles its fixes together, which means that several IE defects may be repaired, but only one vulnerability accounted for. "We count every defect distinctly," argues Shaver. "We count the ones that Mozilla developers find in-house. We count the things we do to mitigate defects in other pieces of software, including Windows itself and other third-party plugins. We count memory behavior that we think might be exploitable, even if no exploit has ever been demonstrated and the issue in question was found in-house. We open our bugs up after we've shipped fixes, so that people don’t have to take our word for our severity ratings."

Shaver also criticizes Microsoft's disclosure standards and what he sees as a lack of transparency. He suggests that Microsoft spend more time fixing bugs instead of "hoping that defects aren't found by someone who they can't keep quiet." Mozilla, he argues, is much more transparent and aggressive when it comes to security. The result? "130 million Firefox users are safer for it every day."

Some people may criticize Jones' approach for cherry-picking the time frames to work in Microsoft's favor: the three-year window neatly coincides with the release of IE 6 for Windows XP SP2. This service pack was the culmination of a massive two-year refocusing on security by Microsoft that mandated security training for every developer in the company. Jones is right to be proud of his company's accomplishments on the security front, but one suspects that if this same test were run again with an earlier start date, the results would have been rather different.