Firefox's powerful add-on system is arguably one of the browser's best features, but it is also occasionally a source of problems for Mozilla. Policing the add-on ecosystem to ensure that third-party code doesn't degrade the quality of the Firefox user experience is a major challenge. It's a problem across the ecosystem of web browsers, and some vendors, like Microsoft with its upcoming Metro version of Internet Explorer, don't allow third-party plugins at all. In contrast, Firefox users have a sea of add-ons at their disposal, but there is danger lurking below the surface.

The problem is being compounded by external software that hijacks browser settings and circumvents mechanisms that the browser uses to protect users from intrusive add-ons. Mozilla's Michael Verdi wrote a blog post last week drawing attention to the issue and calling out some questionable behavior observed in Ask's browser toolbar.

When the browser displays the prompt, the Ask toolbar updater paints a giant green arrow over the screen, pointing to the "Allow this installation" checkbox.

A number of popular free Windows applications, such as the Trillian instant messaging client, bundle the Ask.com toolbar in their installers. Once installed, the Ask toolbar will take up residence beneath the browser's navigation bar. It will also change the user's default home page to Ask.com and change the browser's built-in search box so that Ask.com is the default search engine.

Of course, none of those behaviors are all that surprising. Crapware toolbar plugins from various vendors have been around for years and have historically afflicted users in similar ways across multiple browsers. Mozilla tackled the issue last year by augmenting Firefox's add-on system with a protection mechanism that disables forcibly side-loaded add-ons at startup and then displays a prompt, giving users control over which ones are enabled.

The problem that Verdi raises in his blog post is that the latest version of the Ask toolbar installer takes over the prompt screen and instructs users to allow the installation to continue. The manner in which it does so could potentially create confusion among users, giving them the mistaken impression that the browser is instructing them to enable the add-on.

The Ask toolbar updater, which is a background process that is left running on the computer, monitors Firefox in order to determine when the browser is showing the rogue add-on protection prompt. When the browser displays the prompt, the Ask toolbar updater paints a giant green arrow over the screen, pointing to the "Allow this installation" checkbox. It also paints a yellow bubble at the end of the arrow instructing the user to click the checkbox.

Ask.com didn't intend for this overlay to mislead users. The company apparently made an effort to reduce the potential confusion by putting the phrase "Powered by Ask" in small letters on the yellow bubble. The company also contacted Mozilla before implementing the overlay.

But this effort isn't really sufficient to prevent confusion. As Verdi explains, Mozilla's end-user support community frequently sees users struggling to get their default search engine and home page back after the settings have been altered without their consent by third-party applications. Users tend to blame the browser for the changes and don't realize the actual cause. Like many similar pieces of software, the Ask toolbar doesn't restore the user's original settings when it is uninstalled.

In a video demonstration on the blog post, Verdi showed how the toolbar is bundled with Trillian. When a user launches the Trillian installer and simply clicks the "next" button all the way through without reading the individual pages, they get the toolbar by default without realizing that it is being installed.

Verdi filed a report about the Ask toolbar in the Mozilla issue tracker last month, asking for the rogue add-on prompt to be improved to prevent tampering. When it became clear that the overlay was actually being implemented by the Ask installer outside of the browser, the report was closed with the "WONTFIX" status because there is no technical means by which such tampering can be prevented.

Memory overload

Another problematic add-on that was recently called out by a Mozilla employee is the McAfee Site Advisor, a product that integrates with the user's Web browser and displays safety ratings for websites. Nicholas Nethercote, who leads Mozilla's MemShrink project, issued a warning about an "appalling memory leak" in the McAfee browser add-on.

He investigated the add-on himself after seeing reports from users about inflated memory usage. He found that it has a severely negative impact on Firefox's memory consumption and is so bad that can potentially impacts the browser's performance and stability.

"This morning I tested Site Advisor 3.4.1 myself, and found that, when enabled, it leaks every single content compartment that is created. In other words, most of the JavaScript memory used for any page opened with Firefox is never reclaimed," he wrote. "In terms of memory consumption, this is pretty much the worst possible behavior for an add-on. This excessive memory consumption is likely to cause Firefox to run much more slowly and crash much more often."

He filed a bug report earlier this month recommending that Mozilla consider blacklisting the add-on if McAfee proved unable to resolve the issue in a timely manner. Fortunately, McAfee began working to address the problem when it was brought to its attention. The company released an update this week that reportedly resolves the large memory overhead.

This isn't the first time that problems have arisen with McAfee add-ons. The company's SiteAdvisor and ScriptScan add-ons were both scrutinized by Mozilla last year due to crashes and bugs. These incidents further illustrate how third-party code can degrade the Firefox user experience. Users who don't realize that an add-on is responsible for the problems they experience in Firefox will likely blame the browser itself rather than the actual culprit.

Combating bad add-ons

Mozilla polices its add-on ecosystem and routinely verifies the safety and stability of third-party add-ons that are hosted on the official addons.mozilla.org (AMO) website. Mozilla recently expanded the AMO review process to include more intensive analysis of memory consumption so that it can catch add-ons that suffer from serious leaks.

A significant number of popular add-ons are installed from external sources, however, which means that they aren't subjected to Mozilla's review process. Justin Scott, who leads Mozilla's add-on team, reported last year that only 25 percent of the 600 million add-ons used every day by Firefox users are hosted in AMO.

Mozilla has very little influence over the add-ons that are hosted outside of its own repository. When reaching out to the developers of seriously flawed add-on fails, Mozilla's last line of defense is the blacklist. The blacklist mechanism is a remote kill-switch that allows Mozilla to immediately disable abusive add-ons. It is used only in the most extreme situations as a solution of last resort.

The blacklist is used to disable add-ons that expose users to serious privacy or security risks. It has been used ten times so far in February to terminate add-ons that behaved like malware. Mozilla publishes a full list of blacklisted add-ons for transparency purposes. A brief look at some of the recent entries show behaviors like stealing the user's Facebook cookies or injecting additional advertisements in Web pages. Mozilla typically takes action against such add-ons quickly. In cases of extremely obvious malicious behavior, an add-on can be blacklisted within mere hours after being reported.

The blacklist can also be used to kill buggy add-ons that are having prodigiously negative consequences for performance and stability. One noteworthy precedent is an incident that occurred last year when Mozilla decided to block Skype's toolbar add-on for Firefox. The Skype add-on caused 33,000 Firefox crashes in a single week and degraded performance so badly that it made DOM manipulation 300 times slower in certain cases. The add-on was removed from the blacklist after the problems were resolved.

The challenges of maintaining an ecosystem

The difficulties that Mozilla is facing from add-ons are similar to those faced by any platform that is open to third-party software. Insulating users from the detrimental effects of malware and poorly-implemented applications is a major challenge. To Mozilla's credit, the organization has handled the issue impressively well, with a focus on transparent enforcement and user freedom.

In light of the situation, it's not hard to understand why Microsoft chose to ban Internet Explorer plugins entirely in the Windows 8 Metro browsing experience. The downside of such an extreme choice, however, is that it will significantly reduce the browser's flexibility for users who rely on useful third-party enhancements. It's not clear yet whether Microsoft's decision will lock popular tools like the Evernote Web Clipper and 1Password out of the Metro flavor of Internet Explorer.

There are analogous issues at play in the broader operating system space. The proliferation of problematic software is putting pressure on platform vendors to impose increasingly restrictive policies on their respective ecosystems and erect technical barriers that limit the flexibility of third-party code.

Apple's move to mandate sandboxing in the Mac App Store and encourage code signing for external applications is raising serious concern among some longtime Mac OS X developers and users. But Apple feels that such measures are necessary to protect users from the small but growing threat of Mac OS X malware. Microsoft is similarly responding to the malware threat by moving towards a more restrictive approach to managing third-party software for the Metro environment.

The challenge of creating an ecosystem that offers rich extensibility without compromising the safety of end users is likely to be a major problem in the software industry for many years to come. The manner in which platform vendors balance user empowerment and security will be a major that factor that shapes the contours of modern computing.