You often hear the great stories of companies doing continuous delivery in the Linux world. The tools mentioned are usually the likes of Puppet, mCollective, Salt, etc… But what is a .Net team to do when it wants to implement a good Continuous Delivery (CD) pipeline for .Net on Windows? This is the challenge we set out to solve when we decided we would “do CD” for our new micro-services architecture on Amazon Web Services (AWS).

Why not use the same tools as the other guys?

It was really just a matter of using the best tool for the job. We could have used the same tools as our Linux teams (in fact we experimented with mCollective in the past) but in our situation it really felt more like trying to punch a square peg into a round hole. Those tools are great for Linux but not (yet?) for Windows and .Net.

Domain’s deployment pipeline overview

At a very high level, our CD pipeline looks like this:

We will look into each of these steps in further detail in future posts, but this is how everything fits together in a nutshell:

Build

Even though we have been using automated builds for years, we were still facing challenges with our build process. As we had the opportunity to start fresh on this we decided to take another look at our build pipeline. This is what we ended up with:

Goal: Every developer should be able to run a full build of any of our apps locally. There is nothing worse than having a build issue, making a small change to fix it, waiting for the build to be picked up from the queue, and then finding out after a few minutes after that the problem is still not fixed. Rinse and repeat… This is a waste of time just as much as it pollutes your source control history.

Solution: We’ve been using PowerShell and psake to create a single generic build script that works for all our use cases. If you haven’t heard of psake before, it is basically a replacement for MSBuild. MSBuild is not avoidable completely as it will still build your solution, but you then avoid the angle-bracket tax of an XML-based build process. Goal: The build process shouldn’t require any application, or specific-version of an application (other than the .Net framework) installed on the computer running the build.

Solution: The tools are packaged with the rest of the build process (more on that in the next point) . Having the build process completely self contained makes it really portable and it’s then easier to run different versions of each tool if needed. Goal: With our new micro-services architecture, we end up with a few separate repositories. Each of them have their own copy of the build scripts and tools (see the point earlier about being self-contained) so they have to be easy to upgrade.

Solution: NuGet to the rescue! Just as we are using NuGet to distribute code packages between repositories, we are now packaging our build script and tools as 2 NuGet packages. So if something changes in our build pipeline, it’s just a matter of spending a minute or 2 updating the package in each repository. This actually happened a lot as we were still putting the scripts together and proved to save a considerable amount of time. Goal: The build process should run the unit tests.

Solution: This is a pretty standard setup. We are using NUnit and OpenCover to run our tests and generate a test coverage report for every build Goal: Builds should be fast, so that we have a tight feedback loop if something goes wrong.

Solution: This is more as a result of our micro-services architecture but our build time for each service is now down to 2 minutes or less. Goal: Wouldn’t it be nice if all it took to deploy to UAT, staging or production was a commit/merge to a specific branch? “Deploy-on-commit” was one of our targets.

Solution: We are using Bamboo for our build orchestration and the amazing Octopus Deploy for our deployment orchestration. As Octopus Deploy has been built API-First, this means we can get Bamboo to trigger a deployment to the right environment (based on the code branch) by using the Octopus API. This is awesome!

Infrastructure

This is really not my specialty and our talented DevOps team will write a lot more about this soon.

As mentioned earlier, we are hosting our new services on Amazon Web Services. AWS has a lot of tools available to help with automation of standing up machines. This is how it goes for us:

We use CloudFormation to create templates of our environments. A typical template would be able to create 2 web servers, a load balancer, an auto-scalling group, etc… and set the right permissions on those. Windows Server 2012 Core is our OS of choice. Using the Core version allowed us to boot up each of those machines much faster. As our DevOps lead loves to say: “this is Windows without windows”. One of the unexpected side effects of using Core is that we have to script everything we do as there is simply no other way to get things done. So we just picked the latest AMI for Windows Server 2012 Core and got CloudFormation to use it as a base. That’s were the magic is starting to happen. Thanks to PowerShell Desired State Configuration (DSC) we are able to make sure that all our servers (or at least all the servers in the same cluster) are configured the same way. It’s Puppet for Windows if you will, and will ensure you don’t ever have any configuration drift. Our configuration sets are usually stored in Amazon Simple Storage Service (S3) From that point, it’s just a matter of getting DSC to install a few tools, especially the Octopus Tentacle. The tentacle is Octopus Deploy’s deployment agent which needs to be installed on each server you are deploying to. At the time of writing this post, Octopus doesn’t really support auto-discovery and we need to register each machine with the Octopus server. We are doing that using the Octopus DSC module.

At this point, our load balancers are able to detect unhealthy machines and terminate them, while our auto-scaling groups will be responsible for replacing these machines as well as scaling up and down. That’s right, we now have auto-scaling and auto-healing, and that’s pretty cool!

Deployment

Ok, now that you know what our building blocks are, it’s time to tie them all together and deploy our new packages to the new machines. And for this we use Octopus Deploy. I have to say that we were not convinced that Octopus Deploy would be suitable for deployments in an auto-scaling environment on AWS (it seems to be more directed at Microsoft Azure when it comes to cloud) but it has done the job brilliantly for us so far. One of the best thing is that it has been built API-first which means that anything you can do from the Octopus dashboard you can also do with the API. It is also a very polished product and we haven’t really had any issue with it for deployment of our new micro-services on AWS or of our legacy applications on-premise.

Octopus Deploy requires your applications to be packaged as a NuGet package before they can be deployed. It is actually not as hard as it sounds. A NuGet package is just a zip file with an XML manifest after all. The Octopus team provides a tool called OctoPack to help with the packaging but we decided to stick with the standard NuGet command line tool to package our app. The reason for this is quite simple: we are already using NuGet’s command line tool to package various components in our build process (not entire application) and it was very easy to re-use that.

Once you’ve got your packages you’ll need to host them somewhere so they can be picked up by Octopus. There is a built in NuGet feed available in Octopus Deploy but we preferred building our own NuGet gallery so we could have multiple private feeds. We basically use one feed per team so that it’s easier for everyone to keep track of their apps. You could just as well use a service like MyGet or a tool like ProGet to do the job.

We’ll go into the details of our Octopus setup in a future post but it is pretty straight-forward. Octopus handles application deployment, IIS configuration and PowerShell script execution on remote machines, so it was relatively painless to set up once we knew exactly what we wanted.

So now that we’ve got our builds, our infrastructure, our deployment automation we just need to trigger our deployments on commit. As amazing as this feature sounds, it was actually the easiest part of the process. As mentioned earlier, Octopus Deploy has been designed API-first, so we just wrote a few lines of PowerShell that Bamboo executes at the end of a successful build. Using Bamboo’s plan and branch variables we are able to target a specific Git branch to a specific AWS environment.

The part that still amazes me is the speed of this process. Because we tried to keep everything lean and simple, we are able to go from Git commit to software running on production within 5 minutes!

The release train

Every quarter we run Innovations Days, 2 days where engineers can work on any idea they like. We usually get lots of great ideas prototyped, but this time one in particular was a bit different. Jason, one of our DevOps engineer decided to take the idea of a release train to the next level and hooked up Octopus Deploy to a real electric train!

This may sound a bit quirky but this is actually a great addition. With a few different teams regularly releasing software, it could become tricky to keep up with the changes and know what is going on. With this train we now know exactly when a deployment is starting, which application is being deployed and which version (the release train is using the PowerShell speech API to read out those details).

If you are interested in finding out more about the train don’t miss Jason’s post about the subject.

Conclusion

We’ve been using this process for the last few months and it has been fantastic so far. Releases have become a non-event once we got over the thrill of seeing the deployments happen without any interaction.

We are now up to a point where our test engineers are deploying to production themselves once they are happy with the quality of a build. No developers or DevOps are required anymore as everything is automated.

As expected, it has also enabled us to release much more often and be a lot more reactive:

In the course of the last 30 days, we have had over 150 production releases

A defect found in the morning can usually be patched a few hours later

After a recent micro-service was launched, quick builds and deployments allowed a dozen production deployments in half a day to address performance issues we were experiencing.

We have also been talking about our Continuous Delivery pipeline for .Net at Sydney’s Continuous Delivery meetup on the 17th of September 2014. This was a great experience and the slides are available below:

If you are interested in our Continuous Delivery pipeline, watch this space as we will be posting more details over the next few weeks/months.