Successful CD = Organizational Alignment.

A successful CD process aligns an organization’s technology, processes and people so that deployment becomes a streamlined, automated activity.

The CD and DevOps culture implies that we should be automated as f*ck.

Technology (Tools)

Yes, CD and DevOps methodologies centers around automation. This includes continuous integration, creating environments, deployment strategy, release communications, performance verification and rollback strategy. Having overall visibility and dashboards with analytics and reporting would be awesome.

You do eventually want to automate everything. It does take a lot of work but it is the easiest to achieve.

But it’s NOT all needed from day one.

I do believe that CI should be a prerequisite to Continuous Delivery. You must have some form of Continuous Integration (CI) already in place as CD is an extension of CI.

Process (Workflow)

The middle of the pyramid or the process layer is medium difficulty. Deciding on the CD process typically requires you to know where you want to be and also reviewing your current development and release workflow. Which parts do you keep, improve, or throw away to help you achieve your CD goals?

Our teams have mostly followed the GitFlow workflow branching strategy which has a develop branch and master branch. We also cut a release branch and a hotfix branch depending on the release requirements. In the CD world, we decided on only having a master branch and working from feature branches similar to GitHub’s workflow.

Another area to look into is what your release communication looks like? Does it have too much overhead or not communicated enough? What’s the format? Who should be notified?

People (Culture)

This is by far the hardest part. There will be hesitations, fears and questions initially. How do we find time to invest additional engineering resources to build CD pipelines and automation?

“Let’s stop the line and build continuous delivery pipelines!” said no product owner ever

Product would see any investment here as trade-off between improving pipeline and features.

Anyone should be able to release software to production at any given time (even multiple times a day).

Developers accustomed with tossing their “dev complete” code over the fence to QA will be concerned that no central team members oversees quality.

QA team who currently owns the release process may feel uneasy handing off a part of their role and ownership of quality.

See a quick summary of benefits for each audience in the next section.

What’s In It for Business/Product, QA and Devs?

Business & Product

Get faster and more frequent feedback from users and production systems

Release becomes less painful, reduces team burnout, improves team productivity and more smiles

Catalyst for improving engineering practices — tests, pair programming, higher levels of automation

QA

Frees up time for feature & exploratory testing

Frees up time for improving & adding test automation

Frees up time for earlier involvement on product stories

Can act as QA coach and guide the release process & documentation

Developers

Ship things soon as they’re “Done” and not wait for a release train handled by another team

Code is still fresh on your mind

Faster production feedback loop

Solve the Process and People Part First.

Present to your team on what CD is, its benefits, success factors and how it works. Represent this change as an experiment that requires continuous improvement. Make them part of the feedback loop.

Make the release process transparent and accessible via good documentation.

Find your first developer team members (users) who are willing to push code to production. Ensure they are happy.

Drive Success Factors.

Quality is everyone’s responsibility. Not just QA.

Encourage developers to test as early as possible during the delivery process.

Encourage stronger collaboration between DEV + QA.

Encourage Developers to do “Pair Releases”

Small modifications to existing release process is OK in the interim

Know how long each part of your release process takes so you can measure improvements from automation and refining the process.

CD Culture — Release is a Process, Not a Role

One of the biggest thing on the culture side is to recognize that anyone should be able to release software to production at any given time (even multiple times a day). This includes QA, Devs, Product Owners, and even Engineering Managers and Leaders. I understand if you need to take a minute and let this soak in for a bit.

Quality is everyone’s responsibility. When QA owns the release, it is easy for developers to toss their code over the fence. When developers deploy code themselves, they must own the quality. This means writing good automated tests and trusting their test.

In practice, I see developers doing most of the releases in the CD world. They take their code from development, to testing, to production release. The definition of “done” is now truly code that is working in production and delivering value to users (not just “dev complete”) as soon as it is ready. You write it, you ship it.

Be a Leader and Find Your First Followers

“The most common unscalable thing founders have to do at the start is to recruit users manually. Nearly all startups have to. You can’t wait for users to come to you. You have to go out and get them.” — Paul Graham

Part of doing things that don’t scale means you need to go out and recruit your first user, no matter how laborious it is. The user in this case is a developer who is willing to push their code to production and follow the documented process.

This may require building the muscle memory from doing the release yourself. This will help you quickly see the value with releasing code to production vs passing it to another team. It will help bring clarity to the problem you are trying to solve.

In the past I’ve often built things to solve my own problems. Within the same vein, this means that I only have to find my peers. In this case, another developer who often finishes their code but wishes that they could fully complete their work by seeing it released to production rather than waiting for the release train.

Once you find that developer team member, do it together. Be the release buddy. From there you can identify “CD coaches” that can further guide the rest of your team on the process including providing feedback.

Reinforce Positivity with the First Followers

“You should take extraordinary measures not just to acquire users, but also to make them happy.” — Paul Graham

One of the things I did when an engineering team member did their first code release to production themselves was to publicly recognize and congratulate them. I know this may seem small (hey, QA team members been doing this as part of their role for years!), but keep in mind some developers never done this before.

Having done it myself in the past, I empathize with our developers who release code to production for the first time. Often there is anxiety attached to releases (like the site throwing errors and affecting users after your deployment). Satisfaction comes quickly for the developer when code they just completed and tested in development minutes ago is working in production and bringing value to users.

I want to make sure that as an early adopter of the CD process, the engineer feels that signing up to do this was one of the best choices they ever made on our engineering team.

One of our Software Engineers doing a “Dev-protocol release” for the first time. He was “buddying up” with his highly attentive Engineering Manager on the release. This gave the Software Engineer a great experience even though the CD process was with an “early, incomplete, buggy product.” He was next to my desk so I captured this moment of him in action.

Making The Release Process Transparent and Accessible

Our QA team were the only members doing official releases for many years. Release requires multiple steps across various tools and communication channels.

The happy path goes something like this:

Notify in release room that a release candidate branch (rc-branch) is about to be created from develop branch Deploy rc-branch to staging environment Automation test passes including UI + API Integration (QA currently uses GhostInspector & Runscope) Verify all Jira tickets are all tested & accepted as part of release testing Any manual smoke test is completed with no blockers (QA currently uses TestRail) rc-branch is merged into master Unit Tests (UT) 100% pass (circleCI) Merge master down to develop Deploy master to production (we use Jenkins) Once the release is on production, QA will check UI + API sanity automation results QA verifies performance and quality using New Relic and Datadog/ELK Send out release email with deployed Jira tickets and statuses

The key here is to have clear documentation around the current release process and ensure developers have access to the tools and know what to do when a release pass/fails every step of the way.

Fortunately, our QA Manager did a great job documenting and walking the product development team through each step and coaching our product developers along the way.

Small modifications we made as we rolled out the “Dev-release Protocol” (aka Developer managed releases):

QA will not release on Tuesdays and Fridays. It’s a good transition to encourage developers to do releases. This immediately free’d up QA’s time from managing daily releases to other tasks. Instead of naming the release candidate branch as rc-YYYYMMDD (rc-20180704), we prefixed it with dev-YYYYMMDD (dev-20180714). This allowed us to distinct between a release done by QA as part of the traditional daily release train and one done by a Developer. When a “dev-release protocol” release is done, developers are not required to run through the manual TestRail smoke test cases. Developer releases would typically include smaller set of changes in which case automated test suites should pick up regression bugs.

Developers + QA Collaborate on Feature Testing

With the traditional release cycle, developers would finish their code, merge into an integration branch and call it “dev complete.” The code then catches a release train driven by QA team members (usually once per day for us but could be once a week or sprint in other organizations). This often puts the feature testing done at time of release testing, often delaying other “production ready” code from going to users if the release branch is not stable or heavy feature requires more in-depth testing.

Successful CD requires an immediate shift to testing early as possible in the delivery process. Rather then waiting for your QA team to test your merged feature for the first time, developers should deploy their branch to a shareable test environment. This plays well into partnering with QA on feature testing as needed and breaking down the Dev + QA wall.

From Pair-Programming to Pair-Releasing

I’d like to introduce “Pair-releasing.” It’s like grabbing a buddy and holding your hands together during a release.

In the CD world, there is technically no release phase since code is continuously delivered to production environment and users. Code that is “production” ready can be shipped at any time. Even if it’s one line of code.

A strong suggestion is for the developer who is looking to do the deployment find a release buddy. This is often another developer who also have code commits that is about to be released also, but could be a QA team member or Engineering Manager familiar with the features or code base going out.

Every programmer is already familiar with “Pair-programming.” I’d like to introduce “Pair-releasing.” It’s like grabbing a buddy and holding your hands together during a release.

What Do Release Buddies Do?

For starters, all developers with code commits that are about to be released should be notified. While they do not have to actively manage the release, part of the CD Culture is owning the quality. They should at least do a quick sanity check of their code in production.

If problems do creep up once code is in production (performance issues, increased errors), decisions will need to be made — whether to fix the issue or rollback to previous good build or not. Sometimes that could be a business decision and not a technical decision.

Review Apps (AdHoc Testing Environments)

Automated Way:

When code is pushed up to a branch, the CD pipeline will automatically spin up an ephemeral environment. You get a nice unique virtual app url representing your deployed branch of the application. CI would of course run your automated unit tests. Having an environment will provide a place for automated Integration and UI Tests to execute. Your team will also have an environment for doing any acceptance or exploratory testing as the developer iterates on the branch. This ensures that before code is merged into the main-line, we have the highest level of confidence from testing on these environments.

Manual Way:

Use your existing fixed test environments. Encourage developers to deploy their branch to the test environment. Pair up with QA on the testing early if needed. Do not merge code into integration branch until feature is tested and done.

ChatOps for Releasing

Automated Way:

Imagine having a slack or hipchat release room where you can type a command kicking off deploys to any environment including test, staging and production. This command would also “@” mention notify all relevant product & engineering team members that a release about to happen. It would notify you via chat results off the automated tests.

Manual Way:

Perform release notifications manually. Look at the code commit history on your integration branch and identify developers who contributed to the code that is about to be released. Call out to the release room giving heads up that a release is being done. The concept is the same if you are using GitFlow with release branching strategy. Notify all relevant developers that you are about to cut a release branch for staging deployment.

Performance and Quality Verification

Automated Way:

When code is deployed to production environments, it hooks into APM solutions (we use New Relic) to verify performance or log solutions (we currently use ELK and DataDog) to verify quality (errors/exceptions). Based on thresholds and rules (perhaps even machine learning), the CD pipeline can rollback to previous good build if performance or quality verification fails.

Manual Way:

The simplest thing we can do here is to require who ever is doing the production release to pull up the APM or log tools to check for errors. If we’re seeing high % increase of response times or error rates, then we can decide to rollback to the previous good build or not.

At Time of This Writing

With the help of my awesome team members including Product Engineers, QA, Engineering Managers and Team Leads, we’ve been able to successfully evangelize for Continuous Delivery.

The CD Pipeline and automation efforts by our Infrastructure Engineer started but was placed on hold as we had to shifted some engineering priorities in Q2 of 2018. In a positive way, this was also catalyst for firing up the CD culture and practicing CD methodologies well before automation was ready.

What is great about Continuous Delivery is that you can continuously refine it, respond to change, and make your customers and your team members happy.

I am super proud to see a handful of our developers successfully handling releases for our consumer facing WEB Front-end and Back-end/API projects with positive and constructive feedback even if we had to initially do many things that don’t scale.