While malicious software has long been a near-daily annoyance for Windows PCs, Mac users have become accustomed to not worrying about malware. Threats arise from time to time—in January of this year, for example, a Trojan horse made the rounds in pirated copies of Apple’s iWork software—but most Mac users these days are probably running computers without antivirus software.

Apple has encouraged that habit, too, by frequently touting the Mac’s resistance to malware in its advertising materials, especially when compared to Windows. But with the release of Mac OS X 10.6 (Snow Leopard), Apple has finally decided to subtly step up its fight against malware, much as it has done in the past with antiphishing features in Safari. For the first time, the Mac OS contains a built-in system that detects malicious software and attempts to protect users from inadvertently damaging their computers.

How does it work?

Beginning with Mac OS X 10.4, Apple built a download validation system called File Quarantine into its operating system. In OS X 10.5 (Leopard), this manifested most frequently as a dialog box that popped up when a user first opened a file that was downloaded from the Internet via Mail, Safari, or iChat. The warning revealed which application downloaded the file, from what site, and at what time. It gave the user the option to continue opening the file, to cancel, or to view the Web page from which it had been downloaded.

In Snow Leopard, Apple has enhanced File Quarantine to also check files against known malware, pulling from a list of malware definitions at System/Library/Core Services/CoreTypes.bundle/Contents/Resources/XProtect.plist. As of this writing, the file contains only two definitions: the OSX.RSPlug.A Trojan Horse, first discovered in 2007, and the OSX.iService malware embedded in the pirated iWork installer mentioned earlier. However, Apple told Macworld that the list of definitions can be updated via Software Update.

If you try to open an infected file, Snow Leopard will present you with a stronger warning, saying that the file may damage your computer and suggesting that you move it to the Trash. As with the download validation dialog box, you have the option to continue or cancel, but if the file is on a disk image, there’s a button to eject the image; if, on the other hand, the file is already on your hard drive, that button instead invites you to move the file to the Trash. If you’ve enabled Safari’s Open "Safe" Files After Downloading preference, you will automatically be prompted with the dialog box when the download completes and the file opens. Unlike the more general warning, the malware warning doesn't disappear after the first instance; it will reappear each time you open the file.

File Quarantine seems to serve mainly as a gatekeeper for files downloaded from untrusted sources: think of it as a layer between the user and the untamed wilds of the Internet. Snow Leopard defines an expanded list of applications for which it "quarantines" downloaded files (marking that they've been downloaded from the Internet). So if you download a file via your Web browser (including Safari, Internet Explorer, Firefox, OmniWeb, Opera, Mozilla, Camino, and more) or an e-mail client (Mail, Entourage, or Thunderbird) or you receive a file via iChat, then it will be checked for malware when you open it. However, if you grab an infected file from another source, such as an FTP site, a file-sharing service like BitTorrent, or a program that’s not covered by Apple’s system, then you’re out of luck—the system won’t detect it.

Most important, Apple’s system appears to contain no way to clean malicious software off your Mac after it’s been infected. For that, it seems you’ll still need to turn to third-party antivirus products.

Does it work?

In our tests, the malware system successfully detected the OSX.RSPlug Trojan horse upon trying to open a file infected with it. The dialog box appeared regardless of whether the file was located on a disk image or the computer’s hard disk, as long as the file had been downloaded onto that computer via one of the applications that Apple's system checks.

Because Apple uses the com.apple.quarantine extended attribute (which stores metadata about the file) to record the information about malware, that information can actually travel from Mac to Mac. However, whether the metadata remains with the file depends on exactly how the file is transmitted. If it’s copied via OS X’s file system—to a flash drive, for example, or via the Finder’s built-in file sharing—then the malware mark will stay emblazoned on the file like Hester Prynne’s big red A. However, if you transfer the file through another method—say, via FTP—that metadata will be lost. (There is one exception: zipping the file using OS X's built-in compression tools will keep the quarantine attribute present even if you transfer the file via FTP.)

Of course, malware protection is only as good as its definitions. It’s unknown how often Apple plans to update the virus definitions in Snow Leopard: such updates could be bundled into Security Updates and point releases the way that security patches currently are, either on an ad hoc basis as new threats arise or as a more regular set of updates delivered through Software Update. Apple has been criticized in the past for its sluggish response to security threats, so how it will handle this new system remains to be seen.

What does it all mean?

Now that OS X has built-in malware support, what does that mean for Mac users? Well, here are a few things it doesn’t mean.

It doesn’t mean that a flood of malware will suddenly overwhelm Mac OS X. Yes, Apple’s integration of an anti-malware system is a tacit admission that Mac OS X is far from immune to malicious software, but the company's response is more a prudent precaution than a reaction to an impending tide of evil software.

It also doesn’t mean that Mac users can go about downloading files willy-nilly, with no regard for safety. As always, every computer user, regardless of their computing platform, should take certain precautions: download files from trusted sources; don’t open e-mail attachments from unknown senders; make sure you assign strong passwords to your accounts. Malware prevention software can keep you from being caught unaware, but it doesn’t give you carte blanche to be irresponsible, any more than having a car alarm means you should go out of your way to park your car in a dangerous neighborhood.

And it doesn’t mean that third-party antivirus software makers like Symantec and Intego are going out of business. That’s often a concern when Apple jumps into an established software field, but as the company told Macworld, “The feature isn’t intended to replace or supplant antivirus software, but affords a measure of protection against the handful of known Trojan horse applications that exist for the Mac today.” Snow Leopard’s protection is more of a preventive measure than a cure for malware.

In sum, this added security is a good thing for most Mac users, especially those who have long eschewed antivirus software: we now have an additional level of protection that we didn’t have before. It's not bulletproof, but the next time you look a gift horse in the mouth, at least you'll know whether it's full of Greek warriors.