Incident summary

Updates – Last updated 14:35 PST May 14, 2019. We expect this to be our final update.



If you are running Firefox versions 61 – 65 and 1) did not receive the deployed fix and 2) do not want to update to the current version (which includes the permanent fix): Install this extension to resolve the expired security certificate issue and re-enable extensions and themes.

If you are running Firefox versions 57 – 60: Install this extension to resolve the expired security certificate issue and re-enable extensions and themes.

If you are running Firefox versions 47 – 56: install this extension to resolve the expired security certificate issue and re-enable extensions and themes.

A less technical blog post about the outage is also available. If you enabled telemetry to get the initial fix, we’re deleting all data collected since May 4. (May 9, 17:04 EDT)

Mozilla CTO Eric Rescorla posted a blog on the technical details of what went wrong last weekend. (May 9, 16:20 EDT)

We’ve released Firefox 66.0.5 for Desktop and Android, and Firefox ESR 60.6.3, which include the permanent fix for re-enabling add-ons that were disabled starting on May 3rd. The initial, temporary fix that was deployed May 4th through the Studies system is replaced by these updates, and we recommend updating as soon as possible. Users who enabled Studies to receive the temporary fix, and have updated to the permanent fix, can now disable Studies if they desire.For users who cannot update to the latest version of Firefox or Firefox ESR, we plan to distribute an update that automatically applies the fix to versions 52 through 60. This fix will also be available as a user-installable extension. For anyone still experiencing issues in versions 61 through 65, we plan to distribute a fix through a user-installable extension. These extensions will not require users to enable Studies, and we’ll provide an update when they are available. (May 8, 19:28 EDT)

Firefox 66.0.5 has been released, and we recommend that people update to that version if they continue to experience problems with extensions being disabled. You’ll get an update notification within 24 hours, or you can initiate an update manually. An update to ESR 60.6.3 is also available as of 16:00 UTC May 8th. We’re continuing to work on a fix for older versions of Firefox, and will update this post and on social media as we have more information. (May 8, 11:51 EDT)

A Firefox release has been pushed — version 66.0.4 on Desktop and Android, and version 60.6.2 for ESR. This release repairs the certificate chain to re-enable web extensions, themes, search engines, and language packs that had been disabled (Bug 1549061). There are remaining issues that we are actively working to resolve, but we wanted to get this fix out before Monday to lessen the impact of disabled add-ons before the start of the week. More information about the remaining issues can be found by clicking on the links to the release notes above. (May 5, 16:25 EDT)

Some users are reporting that they do not have the “hotfix-update-xpi-signing-intermediate-bug-1548973” study active in “about:studies”. Rather than using work-arounds, which can lead to issues later on, we strongly recommend that you continue to wait. If it’s possible for you to receive the hotfix, you should get it by 6am EDT, 24 hours after it was first released. For everyone else, we are working to ship a more permanent solution. (May 5, 00:54 EDT)

There are a number of work-arounds being discussed in the community. These are not recommended as they may conflict with fixes we are deploying. We’ll let you know when further updates are available that we recommend, and appreciate your patience. (May 4, 15:01 EDT)

We’ll let you know when further updates are available that we recommend, and appreciate your patience. (May 4, 15:01 EDT) Temporarily disabled commenting on this post given volume and duplication. They’ll be re-enabled as more updates become available. (May 4, 13:02 EDT)

Updated the post to clarify that deleting extensions can result in data loss, and should not be used to attempt a fix. (May 4, 12:58 EDT)

Clarified that the study may appear in either the Active studies or Completed studies of “about:studies” (May 4, 12:10 EDT)

We’re aware that some users are reporting that their extensions remain disabled with both studies active. We’re tracking this issue on Bugzilla in bug 1549078. (May 4, 12:03 EDT)

Clarified that the Studies fix applies only to Desktop users of Firefox distributed by Mozilla. Firefox ESR, Firefox for Android, and some versions of Firefox included with Linux distributions will require separate updates. (May 4, 12:03 EDT)



Late on Friday May 3rd, we became aware of an issue with Firefox that prevented existing and new add-ons from running or being installed. We are very sorry for the inconvenience caused to people who use Firefox.

Our team identified and rolled-out a temporary fix for all Firefox Desktop users on Release, Beta and Nightly. The fix will be automatically applied in the background within 24 hours. No active steps need to be taken to make add-ons work again. In particular, please do not delete and/or re-install any add-ons as an attempt to fix the issue. Deleting an add-on removes any data associated with it, where disabling and re-enabling does not.



Please note: The fix does not apply to Firefox ESR or Firefox for Android. We’re working on releasing a fix for both, and will provide updates here and on social media.

To provide this fix on short notice, we are using the Studies system. This system is enabled by default, and no action is needed unless Studies have been disabled. Firefox users can check if they have Studies enabled by going to:

Firefox Options/Preferences -> Privacy & Security -> Allow Firefox to install and run studies (scroll down to find the setting)

Studies can be disabled again after the add-ons have been re-enabled

It may take up to six hours for the Study to be applied to Firefox. To check if the fix has been applied, you can enter “about:studies” in the location bar. If the fix is in the active, you’ll see “hotfix-update-xpi-signing-intermediate-bug-1548973” in either the Active studies or Completed studies as follows:

You may also see “hotfix-reset-xpi-verification-timestamp-1548973” listed, which is part of the fix and may be in the Active studies or Completed studies section(s).

We are working on a general fix that doesn’t use the Studies system and will keep this blog post updated accordingly. We will share a more substantial update in the coming days.

Additional sources of information: