When a company like Microsoft needs to fix a security flaw in one of its products, the process is normally straightforward: determine where the bug lies, change the program's source code to fix the bug, and then recompile the program. But it looks like the company had to step outside this typical process for one of the flaws it patched this Tuesday. Instead of fixing the source code, it appears that the company's developers made a series of careful changes directly to the buggy program's executable file.

Bug CVE-2017-11882 is a buffer overflow in the ancient Equation Editor that comes with Office. The Equation Editor allocates a fixed-size piece of memory to hold a font name and then copies the font name from the equation file into this piece of memory. It doesn't, however, check to ensure that the font name will fit into this piece of memory. When provided with a font name that's too long, the Equation Editor overflows the buffer, corrupting its own memory, and an attacker can use this to execute arbitrary malicious code.

Normally the work to fix this would be to determine the length of the font name and create a buffer that's big enough to hold it. It's a simple enough change to make in source code. If that's not possible—there are occasional situations where a buffer can't easily be made bigger—then the next best solution is to limit the amount of data copied to it, truncating the font name if it's too long to fit. Again, this is a simple change to make in the source code.

But that doesn't appear to be what Microsoft did here.

Analysis of Microsoft's patch strongly indicates that the company didn't make changes to the source code at all. Instead, it appears that the flaw has been fixed by very carefully modifying the Equation Editor executable itself. Normally when a program is modified and recompiled, there are ripple effects from this compilation. Low-level aspects of the compiled code will change slightly; the recompiled code will use registers slightly differently, functions will be placed at different locations in memory, and so on. But none of that is in evidence here; side-by-side comparison of the fixed program and the original version shows that it's almost entirely unaltered except for a few bytes in a few functions. The only way that's likely to happen is if the bug-fixing was performed directly on the program binary itself without reference to the source code.

This is a difficult task to pull off. The fixed version includes an extra test to make sure the font name is not too long, truncating it if it is. Doing this extra test means adding extra instructions to the buggy function, but Microsoft needed to make the fix without making the function any longer to ensure that other, adjacent functions were not disturbed. To make space for the new length checking, the part of the program that copied the font name was ever so slightly deoptimized, replacing a faster routine with a slightly slower one, and freeing up a few bytes in the process.

The inspection even suggests that this isn't the first time that Microsoft has had to make such fixes; a few instructions were found to be strangely duplicated in the original, broken version of the program. This kind of thing would happen if a previous modification made the program's code slightly shorter.

A look at the Equation Editor's embedded version information also gives clues as to why Microsoft had to take this approach in the first place. It's a third-party tool, developed between 1990 and 2000 by a company named Design Science. That company still exists and is still producing equation editing software, but if we were to guess, Microsoft either doesn't have the source code at all or does not have permission to make fixes to it.

Word nowadays has its own built-in equation editing, but Equation Editor is still supported for backwards compatibility to ensure that old documents with embedded equations continue to be usable. Still, we're a little surprised that Microsoft fixed it rather than removing it entirely. It's truly a relic from another era, coming long before Microsoft's considerable investment in safe coding practices and exploit mitigation techniques. Equation Editor lacks all of the protections found in Microsoft's recent code, making its flaws much easier to exploit than those of, say, Word or Windows. This makes it something of a security liability, and we'd be amazed if this font bug is the last one to be found.