The Security Update 2011-003 that Apple released on Tuesday directly addressed the Mac Defender malware threat in two ways: it changed the way malware files are detected by enabling automatic daily updates, and included code to remove at least two of its variants. Despite this, malware developers had a version available that skirts past Apple's protections within about eight hours. Apple's patch suggests it plans on being more active in addressing possible malware threats, but is Apple ready to take on the role formerly limited to vendors like Norton, Intego, and Sophos?

We'll try to answer that question by first detailing what specific malware protections exist in Mac OS X, and what changes Apple implemented in the latest security update. Then we'll consider how Apple may plan to take over malware protection for its platform.

File Quarantine

Apple first introduced the File Quarantine system in Mac OS X 10.5 Leopard. That system would tag files that were downloaded from the Internet and not known to be safe with a small bit of "quarantine" metadata, including a flag that it might not be a "safe" file, where it was downloaded from, and the time it was downloaded. When a user attempted to open a file with quarantine metadata, the system would warn the user to make sure the file was safe before opening.

Apple enhanced the File Quarantine system in Mac OS X 10.6 Snow Leopard. The system now includes a malware definitions file that contains information to identify known malware threats, including OSX.RSPlug.A and OSX.OpinionSpy. When a quarantined file is first double-clicked, the file is scanned against the definitions to see if it matches any known malware. If so, Mac OS X will warn the user that the file "will damage your computer," noting the specific malware detected and offering to move it to the Trash.

Security Update 2011-003 made three important changes to Mac OS X and the File Quarantine system. First, it included a definition to detect two Mac Defender variants, OSX.MacDefender.A and OSX.MacDefender.B. Users downloading either of these variants will be warned when the trojan is downloaded and begins to install.

Known malware in the File Quarantine blacklist include: OSX.RSPlug.A

OSX.Iservice

OSX.HellRTS

OSX.OpinionSpy

OSX.MacDefender.A

OSX.MacDefender.B

OSX.MacDefender.C

Second, it changed the way malware definitions are updated. Before Tuesday, Apple only updated this file occasionally with OS updates or security patches. For instance, a definition for OSX.OpinionSpy was added to the definitions list by the Mac OS X 10.6.7 update. Now, your Mac will by default check for an updated definitions list daily, and automatically download those updates when available. You can turn this feature off in the Security pane of System Preferences if you prefer, but you'll only get updates to the file when you install future system updates or security patches.

Third, Mac OS X can now detect Mac Defender running on your system and remove it. "If MacDefender malware is found," according to Apple, "the system will quit this malware, delete any persistent files, and correct any modifications made to configuration or login files." Admin users will get a notification that it was removed on their next login.

Apple's response to growing malware threat

The improved File Quarantine system and the added Mac Defender removal feature are surely a welcome response to the increasing problem of this trojan. However, as noted by ZDNet, approximately eight hours after Apple pushed out the patch to tackle Mac Defender, malware authors had a version out in the wild that could bypass the new protections. It seems Apple has joined a classic cat-and-mouse game with malware authors.

The question is, does Apple really want to play this game? And how long can it keep up?

When Mac Defender first appeared, the threat was considered very low because it originally required an admin password to install. That added an additional barrier that it was believed would prevent most users from installing the software, which ironically masqueraded as real antivirus software. The trojan would then throw up numerous fake "virus detected" warnings in an attempt to scare unsuspecting users into sending a registration payment over an unsecured online payment system.

However, Apple Geniuses and third-party service technicians told Ars that Mac Defender and its variants had begun to become a problem for an increasing number of users. Later, malware authors developed a version of the software that did away with the admin password requirements, installing directly into a user's own separate Applications folder. While that would prevent access to other users' data, it could still send up fake virus infection notifications that could scare the user into "registering" the software. Those who ponied up the $40 or so handed over their credit card information to a group of ne'er-do-wells.

While Apple originally took the tack of ignoring the growing Mac Defender problem, last week the company made a public acknowledgement of the situation and offered a support document that explained how users could get rid of the trojan. It also promised a security patch that could automatically detect and eliminate the malware.

Apple made good on that promise with Tuesday's update. But now the company will continually have to look for variants and quickly update its malware blacklist if it wants to stay on its current path. This is something that AV vendors have been doing for years, but Apple (so far) doesn't have a track record for speed when it comes to such matters. It took Apple nearly 10 months to update the definitions file for OSX.OpinionSpy; it took the company 22 days to even acknowledge the Mac Defender was a problem.

It took Apple at least 24 hours to respond to the new variant of Mac Defender, dubbed OSX.MacDefender.C, judging by the publication time of an article about the update by Italian blog Spider-Mac. We were able to verify that the definitions had been updated by checking the file online, but our local machine had not yet updated the definitions file locally. (The actual timestamp of the file can't be checked unless your machine has updated the file locally.) One consequence of the way Apple's 24 hour automatic updating is that depending on when your machine checks for an update and when Apple publishes it to its servers, you might have to wait until 23:59 before you have the updated definitions on your machine.

Apple continually touts the fact that Mac OS X suffers from few, if any, malware infections, especially scary "PC viruses." But that may be changing. Security researcher Charlie Miller, known for his repeat Pwn2Own wins targeting Macs, reminds us that Mac OS X isn't necessarily more secure than other operating systems, particularly when it comes to the kind of social engineering that makes malware such a problem.

"[Mac OS X] has vulnerabilities, and it will let you download and run malware," Miller told Ars recently. "The difference is that there simply isn't that much malware written for it. The bad guys have focused all their energies at Windows however, as market share for Macs continues to inch up, that equation is going to change and bad guys will begin to focus in on Macs. When the bad guys decide to go after them with gusto, it'll get ugly fast."

Though Apple touts Mac OS X's security advantage, the company still recommends using antivirus software. "The Mac is designed with built-in technologies that provide protection against malicious software and security threats right out of the box," advises the Mac OS X Security page on Apple's website. "However, since no system can be 100 percent immune from every threat, antivirus software may offer additional protection."

That may be good advice for a growing number of users, especially those in networked environments or those who lack the technical savvy to avoid the bad stuff. While Apple's File Quarantine solution identifies just five known malware threats, and only removes one of them, antivirus software can scan e-mail attachments for Mac or Windows malware, preventing users from inadvertently spreading them to coworkers, colleagues, and friends. And vendors like Intego have a reputation for updating definitions files shortly after new malware is identified.

If Apple can't take over for more traditional AV vendors and security researchers, though, iOS and the Mac App Store may provide clues to how Apple may deal with the malware problem in future versions of Mac OS X. iOS is essentially a closed ecosystem, where software can only be installed via the App Store. Applications must be digitally signed by the developer, and iOS will refuse to install or run software that is modified in any way. Apple further randomly checks applications submitted to the App Store to make sure they don't gather user data or perform other nefarious tricks.

The Mac App Store is essentially the same setup, except for Mac OS X. Applications are digitally signed, and Mac OS X could refuse to run them if the software is modified on the way to the user. Users can still install software from any source on Mac OS X, and we believe Apple won't eliminate that ability any time soon (if ever). But perhaps Lion or another future version could be configured to only install and run software acquired via the Mac App Store (you know, for the paranoid types). While most advanced users wouldn't stand for such limitations, less sophisticated users may be willing to only get software from the Mac App Store, especially if Apple could guarantee increased security.

Listing image by Photo illustration by Chris Foresman