I was tempted to feel bad for McAfee in the wake of a disastrous Wednesday upgrade. We all make mistakes and things must be bad there right about now. But it's hard to feel sorry for them, given what's turned up.

I feel really sorry for the McAfee users who got the company put out on Wednesday. Many McAfee VirusScan Enterprise customers using Windows XP SP3 had their svchost.exe (a key Windows system binary which hosts Windows services in its process space) flagged as malicious. I was tempted to feel sorry for McAfee too; we all make mistakes and things must be bad there right about now. But it's hard to feel sorry for them, given what's turned up.

The most shocking revelation was uncovered by Ed Bott at ZDNet: McAfee has admitted to its customers that it followed shoddy quality assurance procedures in this matter. Specifically, the release was not tested on Windows XP SP3, the configuration on which it borked the system. It's hard to think of a worse single configuration to leave out. Late last night McAfee .

This sort of thing has happened in the past, and the danger of it increases all the time. The nature of malware has forced AV vendors to push out ever more frequent definition updates, to the point where Symantec's "pulse updates" come out every five to 15 minutes. The pressure to keep up with malwarenot to mention the pressure to keep costs downcan lead vendors to scrimp on testing. This appears to be what McAfee did.

You might well ask what McAfee is doing scanning Windows system binaries anyway. I know I did. It turns out that they do whitelist these files as a general matter, but in this case things got complicated. Like most security products, McAfee's scan memory for signs of infection. The malware (W32/Wecorl), the definition for which generated the false positive, is one which attempts to attack svchost.exe by inserting itself into the file and modifying the program to run it. It's a classic file virus. McAfee scanned the file because the false positive occurred in a memory scan of the svchost.exe process; this caused it to flag the file.

Exactly what happened is better explained by McAfee in a FAQ it posted last night. Three specific versions of svchost.exe under XP SP3 were affected, and McAfee provides the MD5 hashes. It seems Microsoft changes this file in updates without changing either the size or file version.

If you still need to remediate systems, McAfee has provided a FAQ for that too. McAfee also says it is adding new QA protocols to ensure this don't happen again.

Analysts and other vendors are spouting off about this, and some of the talk is misleading or distasteful. Prevx CEO Mel Morris issued a statement mischaracterizing the malware involved in the false positive and insinuating that products such as theirs, which don't rely on implementation-specific definitions, don't have such problems. There is a lot to be said for Prevx's approach, but to imply that it are immune from false positives due to programmer or testing error is just plain dishonest.

David Ulevitch of OpenDNS, not a McAfee competitor, argued that this incident shows the advantage of cloud-based solutions: "Fixing 1,000 cloud-based scanners is a heck of lot easier than fixing millions of desktop end-points." McAfee, of course, uses some cloud-based scanning through its Artemis system, and one of the measures it plans to use in reaction to this incident is to create an expansive whitelist in Artemis. Personally, I don't see a huge advantage for cloud-based systems here; whitelists are fairly static things. And removing the definition wasn't the hard part of the remediation; it was getting a usable svchost.exe back on the system.

Sunbelt Software is offering a deal (six months of free maintenance) to angry McAfee customers. Business is business I guess. Don't assume that Sunbelt is immune from such problems, but then it doesn't claim to be.

As I said above, this sort of problem is not unprecedented; if the false positive had been on sol.exe (Windows Solitaire) it would have been embarrassing, but a minor affair. This was a perfect storm of bad news for McAfee and its customers. But in this business you make a lot of your own weather, and McAfee's inadequate testing is the culprit here.

Originally posted on PCMag's Security Watch blog.