From MozillaWiki

Feature Flagging with Normandy Pref Rollout

Normandy Pref Rollout is a feature that allows Mozilla to change the default value of a preference for a targeted set of users, without deploying an update to Firefox. This document focuses on the use of Pref Rollout as a mechanism to enable feature flagging in Firefox.

Background: Feature Flags

It isn't always desirable to turn on a feature for all users at once, or exactly when a new version of Firefox is released. In this case, it is common to make the feature controllable via a flag, or a switch that can be used to turn the feature on and off. In Firefox this is generally a preference. There are then several mechanism we can use to turn this flag on to enable the feature off-train, Normandy pref rollout is a simple, safe, and featureful way.

Check list

If you want to make a change to Firefox via preference rollout, here are the things you must do.

Make a feature, controlled by a preference, and land it in the tree

Have someone from the Normandy team look at the feature, to check for compatibility

Get the feature and preference tested by PI

Decide what metrics you will use to determine success

Decide a timeline for deployment

File a rollout bug

Send an intent-to-ship email, see below for details on its content

Get approval from Release Engineering

Create a Normandy recipe

Get Normandy recipe approved

Ship it!

Capabilities of Normandy Pref Rollout

Normandy can change one or more preferences for a targeted set of users. This can be used to enable a feature that is controlled by a flag. Targeting can be based on many criteria, including:

Firefox version

channel (release, beta, nightly)

a percentage of users

country

Firefox locale

installed add-ons

profile age

any preference value

many keys in Telemetry

Rollout should be considered permanent. Once a user matches a rollout targeting criteria, that user is enrolled in the rollout until it is explicitly undone on the server (rollback), or the built in defaults change to match the rollout (graduation). A version change, removing an add-on, changing a preference, or any other change in targeting criteria will not automatically remove a user from a rollout.

When not to use Pref Rollout for feature flags

You should not use Normandy's pref rollout feature for a feature if any of the following apply:

If you are trying to answer a question, the feature is experimental in nature, or you need multiple branches. Consider a Shield preference study

The feature is not already in the target version of Firefox, in a disabled state System add-ons are able to ship new code off-train

You are targeting a version of Firefox older than 61.

The feature requires a restart after the preference is changed

When to use Pref Rollout for feature flags

Normandy pref rollout may be suitable if

The feature is already confirmed to have a positive effect.

It is undesirable to wait until the next train switch to turn on the feature.

Gradual rollout is desired.

The new value is expected to become the default value in future versions.

The feature is already on in Firefox, and should be turned off or adjusted.

The feature is well tested by QA and automated tests.

Additional considerations:

Feature flags are not a way to skip the trains.

Feature flagging is not a replacement for thorough testing.

Normandy pref rollout is not generally compatible with preferences that require a browser restart.

Intent to Ship

When sending an intent-to-ship email, it is important to be clear about the following items:

The bug of this rollout, filed above

This is an off-train feature flag change

The testing plan, and the current state of testing

The timeline and rollout plan

The targeted audience, including channel and version

What would cause the feature to be rolled back

Rollback

Sometimes mistakes happen. A bug is found, the feature doesn't interact well with an important website, or there is some conflict found late in the process. In this case, a rollback is needed. This is a separate step, simply disabling the Normandy recipe will not cause a rollback.

Rollbacks can be targeted separately from rollouts, but should generally re-use the same targeting as the rollout. Rollbacks do not specify the preferences to change or their new values, but instead specify a specific rollout by name to undo.

Rollouts are semi-permanent: they won't change unless some other action is taken. Rollbacks are actually permanent. Once a rollout has been rolled-back, it can't be re-rolled-out without starting over with a new rollout.

Relationship with Shield

Shield is a program for studies and experimentation on Firefox. Shield also use Normandy to make changes to Firefox, including changing preferences. The difference between Shield and preference rollout is one of intent. Preference rollout is meant for permanent changes that we are sure of. Shield is meant for testing variations and figuring out what, if anything, is the best thing to do.

In short, Shield is for asking questions, and preference rollout is for applying the answers that come out.