MozMEAO SRE Status Report - 5/9/2017

Here’s what happened on the MozMEAO SRE team from May 3rd - May 9th.

Current work

Bedrock (mozilla.org)

Bedrock multi-region RDS provisioning

Work continues to move Bedrock from Deis 1/Fleet to Kubernetes. The team has implemented Terraform automation to provision RDS instances in multiple regions.

Demo deployments

Jenkins deployments have been restructured, and demos now build in main pipeline. This was a meaty PR from pmac, and a motivation to upgrade Deis Workflow to the latest version (more info below).

Next actions:

create persistent development, staging, and production applications using RDS (Postgres)

enable deployments to new apps in Jenkins

Cloudfront distribution and integration testing

MDN

We’re working on migrating custom Apache config for MDN directly in Kuma/Django for the eventual move to AWS. Most of the Apache rewrites/redirects have been implemented in Kuma, with only a few remaining.

Basket

The FxA team would like to send events (FXA_IDs) to Basket and Salesforce, and need SQS queues in order to move forward. We automated the provisioning of dev/stage/prod SQS queues, and passed off credentials to the appropriate engineers.

Kubernetes / Deis Workflow

Deis Workflow has been upgraded to latest version (2.14.20) in Virginia and Tokyo. We hit a few snags during the first upgrade, as our Workflow install has some customization that wasn’t applied. Subsequent upgrades should be easier, as we’ve automated the process via a script (with minor tweaks in this PR).

Snippets

Snippets-stats is running in Tokyo and Virginia.

snippet-stats was already running on our Deis 1 clusters in Oregon and Ireland, however Giorgos enabled it on our Virginia and Tokyo Kubernetes clusters.

Metrics have been validated for snippets-stats in Virginia and Tokyo.

Application memory/CPU limits and autoscaling have been configured in Tokyo and Virginia.

Issues with HTTP_X_FORWARDED_PROTO header not set for for snippets-*.virginia.moz.works

We created a generic http to https redirector service that runs in Kubernetes. This allows Kubernetes to handle forwarding http to https for us without having custom implementations in each application. However, there remained an issue in our current ELB setup where HTTP_X_FORWARDED_PROTO was not set, and thus Django cannot be aware whether a connection is secure or not.

pmac has implemented an alternative to X-Forwarded-Proto using an HTTPS env var and a SWGIRequest subclass.

Thanks to Giorgos and pmac for their hard work on this!

Decommission webwewant.mozilla.org

webwewant.mozilla.org has been decommissioned. All requests to webwewant.mozilla.org are now being forwarded to https://www.mozilla.org.

Future work

Decommission openwebdevice.org

Waiting for some internal communications before moving forward.

Nucleus

We’re planning to move nucleus to Kubernetes, and then proceed to decommissioning current nucleus infra.

Basket

We’re planning to move basket to Kubernetes shortly after the nucleus migration, and then proceed to decommissioning existing infra.

New Kubernetes cluster

We’ll be creating a new Kubernetes cluster in Portland so we can take advantage of EFS to support MDN in that region. We currently run many of our services from Portland, Virginia, and Ireland. The new cluster will be created in an entirely new VPC, and existing resources will not be shared.