2012 was an "exciting" year for OS X security—at least if you're a security expert or researcher. There were plenty of events to keep people on their toes. Although Apple took some egg on the face for some of them, overall, the company came out ahead when it came down to keeping users safe.

At least that's the opinion of some security researchers who followed OS X developments throughout the year.

Back to the Flashback

Remember Flashback? That malware first made its way onto the Mac in 2011, but never became widespread enough for most users to even become aware of it—until earlier this year. Suddenly, Apple was faced with arguably the first truly high-profile malware to appear on OS X, right as Apple was appearing more than ever in the media.

The incident sparked plenty of hemming and hawing about the end of "security through obscurity" for Apple. Researchers and pundits alike argued that Apple's continued popularity could only lead to more attacks on security, whether they occur on iOS or the Mac. Indeed, it's hard to deny that malicious attacks on Mac users are increasing in frequency, and Apple did take some flak for talking a big security game for so long while simultaneously leaving open a Java hole for two whole months after it was first patched by Oracle.

Removal of Java

But despite this stumble, the Flashback fiasco was the catalyst for one of the most meaningful decisions Apple made in order to beef up OS X security.

"Flashback both led to Apple removing Java from their default installs, and prompted them to release a dedicated cleanup tool," security researcher (and former security engineer for Obama for America) Ben Hagen told Ars. "When an OS vendor releases a dedicated cleanup tool, you know things are bad."

Hagen pointed out the need existed for Apple to release its own Flashback cleanup tool because the Mac anti-malware market and user base "is relatively immature." But the bigger decision to come out of Flashback was to reduce the role of Java in OS X users' lives as much as possible, unless the user specifically installs it.

"The removal of Java was a very interesting decision and de facto statement by Apple. Java on user systems has become a notorious vector for exploitation; with new, remotely executable vulnerabilities coming out several times last year," Hagen said. "Removing Java both simplifies Apple’s position and provides a safer default state for its users."

Noted (and notorious) Mac and iOS "hacker" Charlie Miller agreed with this assessment, going so far as to say it was the most significant decision Apple made in 2012.

"Nowadays, the amount of effort required to write an exploit for OS X is roughly the same as that for writing one for Windows. Because of the bigger payoff for Windows exploits (more users) there are almost no OS X exploits in the real world. But the exception is for Java exploits," Miller told Ars. "For Java applet sandbox escape type exploits, the same exploit will work on Windows and OS X."

Miller credited Java for practically the entire reason we're actually seeing exploits that affect Mac users pop up in the wild. "Therefore, anything Apple does to reduce Java's install base in OS X is a security gain that still gives them some real life improvements," he said.

Movement to signed security model for apps

But even as Flashback was going down and Java was on the way out, Apple was already in the process of making some other major changes to the way users interact with apps on the Mac. A new feature in Mountain Lion, released in the summer of 2012, would (by default) restrict the origin of third-party apps installed on the system, therefore protecting the user from inadvertently installing apps from malicious or unknown sources.

Called Gatekeeper, this feature required Apple's developer ecosystem to either sign their apps with a registered certificate—holding them at a higher level of responsibility for when things go haywire—or selling their wares through the Mac App Store and giving Apple its 30 percent cut. The reaction from the developer community was surprisingly non-panicked, with most telling us they were cautiously optimistic about the level of control still given to users, should they opt to throw caution to the wind and install any apps they please.

And when we followed up with Mac developers several months later, they remained largely positive about the effect of Gatekeeper on both the app ecosystem and users. "I think GateKeeper is a huge boon to end users—it’s effective against man-in-the-middle and masquerade attacks, and the latter is a very common vector for malware," Delicious Monster's Wil Shipley told Ars in September. Iconfactory's Craig Hockenberry agreed: "I definitely think that GateKeeper is helping end users. I know that whenever I click on a download link and see that the developer hasn't signed their app, I think twice about installing it."

Indeed, the overall sentiment around Gatekeeper has been more positive than some of us expected, and security experts appear to be happy about how smoothly it has gone so far.

"From a security perspective, Apple’s continued movement towards the App Store for OS X and the addition of a strict signed security model for applications was a significant move toward a more controlled ecosystem," Hagen told Ars this week. "The curated App Store model lets Apple provide some quality control and sight over which applications are available to end users. It also goes pretty far in limiting user exposure to malware in the form of user-downloaded applications (Fake AV applications, spyware, and the like)."

Just in time for a high-profile "hack"

Flashback, the eventual removal of Java, and the launch of Mountain Lion weren't the only security-related topics that rippled out from the Mac-using world in 2012. The August "hack" of Wired editor Mat Honan made huge headlines not just for its magnitude—Honan's iPhone, iPad, and Mac were entirely wiped out by remote attackers, and he failed to make a backup—but also because of what technologies were involved. Namely, Apple technologies—ones that were all associated with Apple's newest version of its cloud services, iCloud.

It wasn't entirely iCloud's fault. Amazon was involved too, and the attackers were able to socially engineer both Amazon and Apple into giving them the kind of access they needed in order to destroy Honan's digital life.

So what does this have to do with Apple, really? The company wasn't directly involved in the erasure of Honan's data, but as Hagen pointed out, the incident was significant in 2012 because it "highlighted both social flaws in several well-known online account systems, and used Apple’s iCloud as a liability."

By compromising Honan's iCloud account, his devices became vulnerable to a remote-wipe attack. "This is a new problem for many consumers; a failure to protect one of their online accounts, can lead to their own devices actually becoming 'useless,'" Hagen told Ars. "This attack highlighted the need for Apple and other organizations to protect account access from social attacks, and the need for individuals to treat their Apple accounts with extra sensitivity."

Indeed, Honan's hack caused many of us—geeks, "regulars," and reporters alike—to change our passwords, set up two-factor authentication, and ensure we had solid backup plans in ways we only talked about previously. Both Amazon and Apple ended up changing their policies to prevent similar attacks in the future as well. This one doesn't have a particularly happy ending, but Honan's loss is our gain, at least when it comes to being security-minded.

Looking to 2013

So 2012 was a volatile year for Apple and OS X security, but overall, "I think [Apple's] in pretty good shape," Miller told Ars.

But as with most things, there's always room for improvement. What should happen to the OS X landscape as we move forward into 2013?

Miller wants to see more transparency out of Apple. "One thing I'd like to see is more transparency and interaction with the security community. Their BlackHat talk where they didn't take questions was a bit of a farce," Miller said. "I'd like to see them communicate more with how they do their testing, how the App Store review process works, answer questions about their security, etc."

Indeed, transparency and Apple are usually two words that can't show up in a sentence together, but Apple CEO Tim Cook has slowly (and carefully) begun changing things when it comes to Apple being open with the world. But what Hagen wants to see goes beyond just talking about what Apple is doing—he wants to see the community itself step up to take responsibility for its own security as well.

"I think the AV/Anti-Malware offerings for OSX will need to mature quickly in order to meet near-term threats. Microsoft's route of branding their own offering was great for the end user, it would be great if Apple took a similar approach," Hagen said. "The unification of App Store and system updates simplifies things; OSX users will need to get in the habit of applying updates in a timely manner."