Kernel vulnerabilities: old or new?

Benefits for LWN subscribers The primary benefit from subscribing to LWN is helping to keep us publishing, but, beyond that, subscribers get immediate access to all site content and access to a number of extra site features. Please sign up today!

A quick search of the CVE database turns up 80 CVE numbers related to kernel vulnerabilities so far this year. At one recent conference or another, while talking with a prominent kernel developer, your editor confessed that he found that number to be discouragingly high. In an era where there is clearly an increasing level of commercial, criminal, and governmental interest in exploiting security holes, it would be hard to be doing enough to avoid the creation of vulnerabilities. But, your editor wondered, could we be doing more than we are? The response your editor got was, in essence, that the bulk of the holes being disclosed were ancient vulnerabilities which were being discovered by new static analysis tools. In other words, we are fixing security problems faster than we are creating them.

That sort of claim requires verification; it is also amenable to being verified by a researcher with sufficient determination and pain resistance. Your editor decided to give it a try. "All" that would be required, after all, was to look at each vulnerability and figure out when it was introduced. How hard could that be?

So, the basic process followed was this: pick a CVE entry, find the patch which closed the hole, then dig through the repository history and other resources in an attempt to figure out just when the problem was first introduced into the kernel. In some cases, the answer was relatively easy to find; others were sufficiently hard that your editor eventually gave up. One especially valuable resource in the search turned out to be the Red Hat bugzilla; the developers there (and Eugene Teo in particular) go out of their way to document the particulars of vulnerabilities. Sometimes, the commit which introduced the bug was simply listed there. The "git gui blame" utility is also quite useful when doing this kind of research.

About 60 of the 80 vulnerabilities listed above were dealt with in this way before your editor's eyes crossed permanently. The results can be seen in the following table. Let it be said from the outset that there will inevitably be some errors in the data below; the most likely mistake will be assigning blame to a commit which actually just moved the vulnerability from somewhere else. That may lead to a bias that makes vulnerabilities look more recent than they really are. That said, a best effort has been made, and things should not be too far off.

Some other notes relevant to the table:

No attempt was made to find the origin of vulnerabilities which were present in the initial commit which began the git era during the 2.6.12 development cycle. Anything which was already present then can certainly be said to be an old bug.

Some parts of the code have been changed so many times that it can be truly hard to determine when a vulnerability was introduced; places where your editor give up are marked as "unknown" above. One could maybe come up with a real answer by bisecting and trying exploits, but your editor's dedication to the task was not quite that strong.

A couple of these bugs are old in a different way - CVE-2010-1188 was fixed in 2008, but was only understood to be a security issue in 2010. Anybody running a current kernel would not be vulnerable, but bugs like this can be nicely preserved in enterprise kernels for many years.

Looking at when the vulnerabilities were introduced yields a chart like this:

So, in a sense, the above-mentioned kernel hacker was correct - an awful lot of the vulnerabilities fixed over the last year predate the git era, and are thus over five years old. It seems that security bugs can lurk in the kernel for a very long time before somebody stumbles across them - or, at least, before somebody reports them.

According to the information above, we have fixed dozens of vulnerabilities since 2.6.33 without introducing any. The latter part of that claim might be charitably described as being unlikely to stand the test of time. There were (at least) 13 vulnerabilities fixed in the 2.6.35 cycle, 21 in the 2.6.36 cycle. We can hope that fewer vulnerabilities were added in that time; it seems certain, though, that (1) the number of vulnerabilities added will not be zero, and (2) it will probably take us five years or more to find many of them.

There may be some comfort in knowing that a large proportion of 2010's known security vulnerabilities are not a product of 2010's development. Indeed, assuming that a fair number of the old vulnerabilities are a bit older yet, one can also claim that they are not a product of the "new" kernel development model adopted in the early 2.6 days. That claim could be tested by extending this research back into the BitKeeper era; that is a task for a future project.

Your editor remains concerned, though, that it is too easy to put insecure code into the kernel and too hard to discover the vulnerabilities that are created. Analysis tools can help, but there really is no substitute for painstaking and meticulous code review when it comes to keeping vulnerabilities out of the kernel. At times, it is clear that the amount of review being done is not what it should be. There may well come a day when we'll wish we had found a way to be a bit more careful.

