Over the course of the last year, the software development teams at The New York Times have been evaluating Google Cloud Platform for use in some of our future projects. To do this, we’ve surveyed a wide variety of software management techniques and tools, and we’ve explored how we might standardize building, testing and deploying our systems on GCP.

Our newly formed Delivery and Site Reliability Engineering team came up with two methods of deployment with Google Container Engine and Google App Engine for computing environments and the open source implementation of Drone as a continuous integration tool. As a result of this work, we are open sourcing two plugins for Drone: drone-gke and drone-gae respectively.

Container Engine is Google’s managed Kubernetes container orchestration platform. Kubernetes is an open source project that provides a declarative approach to managing containerized applications, enabling automatic scaling and healing properties. It encourages a common standard of how our applications are designed, deployed, and maintained across many independent teams. And because Kubernetes pools compute resources, developers can run many isolated applications in the same cluster, maximizing its resource usage density.

App Engine is a mature serverless platform Google has offered since 2008. It is capable of quickly scaling up and down as traffic changes, which is ideal for many scenarios at The New York Times when you consider possible sudden spikes from breaking news alerts or the publication of the new daily crossword every weekday at 10 p.m.

Drone is an open source continuous integration and delivery platform based on container technology, encapsulating the environment and functionalities for each step of a build pipeline inside an ephemeral container. Its flexible yet standardized nature enables our teams to unify on a plugin-extensible, ready-to-use CI/CD pipeline that supports any custom build environment with isolation, all with a declarative configuration similar to commercial CI/CD services. The result is the ability for developers to confidently ship their features and bug fixes into production within minutes, versus daily or weekly scheduled deployments. As a containerized Go application, it is easy to run and manage, and we hope to contribute to the core open source project.

Google provides an excellent set of command line utilities that allow developers to easily interact with Google Container Engine and Google App Engine, but we needed a way to encapsulate those tools inside of Drone to simplify the workflow for developers. Luckily, plugins for Drone are simple to create as they can be written in Go and are easily encapsulated and shared in the form of a Docker container. With that in mind, the task of creating a couple reusable plugins was not that daunting.

drone-gke is our new plugin to wrap the gcloud and kubectl commands and allow users to orchestrate deployments to Google Container Engine and apply changes to existing clusters. The Kubernetes yaml configuration file can be templated before being applied to the Kubernetes master, allowing integration with Drone’s ability to encrypt secrets and injecting build-specific variables.

Here is an example Drone configuration to launch a Go application in a container via a Kubernetes Deployment resource into Google Container Engine:

# test and build our binary build : image : golang : 1 . 7 environment : - GOPATH = / drone commands : - go get - t - go test - v - cover - CGO_ENABLED = 0 go build - v - a when : event : - push - pull_request # build and push our container to GCR publish : gcr : storage_driver : overlay repo : my - gke - project / my - app tag : "$$COMMIT" token : > $$GOOGLE_CREDENTIALS when : event : push branch : master # create and apply the Kubernetes configuration to GKE deploy : gke : image : nytimes / drone - gke zone : us - central1 - a cluster : my - k8s - cluster namespace : $$BRANCH token : > $$GOOGLE_CREDENTIALS vars : image : gcr . io / my - gke - project / my - app : $$COMMIT app : my - app env : dev secrets : api_token : $$API_TOKEN when : event : push branch : master

And the corresponding Kubernetes configuration:

kind : Deployment apiVersion : extensions / v1beta1 metadata : name : {{ . app }} - {{ . env }} spec : replicas : 3 template : metadata : labels : app : {{ . app }} env : {{ . env }} spec : containers : - name : app image : {{ . image }} ports : - containerPort : 8000 env : - name : APP_NAME value : {{ . app }} - name : API_TOKEN valueFrom : secretKeyRef : name : secrets key : api - token -- - kind : Service apiVersion : v1 metadata : name : {{ . app }} - {{ . env }} spec : type : LoadBalancer selector : app : {{ . app }} env : {{ . env }} ports : - port : 80 targetPort : 8000 protocol : TCP

And the corresponding Kubernetes secrets configuration:

kind : Secret apiVersion : v1 metadata : name : secrets type : Opaque data : api - token : {{ . api_token }}

drone-gae is our new plugin to wrap the gcloud and appcfg commands and allow users to make deployments to Google App Engine standard environment with Go, PHP or Python or to the flexible environments with any language.

Here’s a very basic example of all the configuration required to launch a new version of a Go service to Google App Engine’s standard environment with a second step to migrate traffic to that version:

deploy : # deploy new version to GAE gae : image : nytimes / drone - gae environment : - GOPATH = / drone action : update project : my - gae - project version : "$$COMMIT" token : > $$GOOGLE_CREDENTIALS when : event : push branch : master # set new version to 'default' , which migrates 100 % traffic gae : image : nytimes / drone - gae action : set_default_version project : my - gae - project version : "$$COMMIT" token : > $$GOOGLE_CREDENTIALS when : event : push branch : master

Deploying new versions to the flexible environment requires a little more work, but it’s straightforward when using the plugin. We first use a build step to test and compile the code, then a publish step to build and publish a Docker container to Google Container Registry (via the drone-gcr plugin) and finally, we kick off the deployment via our new plugin.

# test and build our binary build : image : your - dev / golang : 1 . 7 environment : - GOPATH = / drone commands : - go test - v - race ./ ... - go build - o api . when : event : - push - pull_request # build and push our container to GCR publish : gcr : storage_driver : overlay repo : my - gae - project / api tag : "$$COMMIT" token : > $$GOOGLE_CREDENTIALS when : branch : [ develop , master ] event : push deploy : # deploy a new version using the docker image we just published and stop any previous versions when complete . gae : image : nytimes / drone - gae action : deploy project : my - gae - project flex_image : gcr . io / my - gae - project / api : $$COMMIT version : "$${COMMIT:0:10}" addl_flags : - -- stop - previous - version token : > $$GOOGLE_CREDENTIALS when : event : push branch : develop

We hope open sourcing these tools helps other engineers who want to leverage Drone as a continuous delivery solution. We are also looking to the community to take a look and help us harden our systems. Please raise an issue if you find any problems and follow the contributing guidelines if you make a pull request. For further reading and more documentation, you can visit the code repositories on Github:

github.com/NYTimes/drone-gae

github.com/NYTimes/drone-gke