If I had a nickel for every time our deployment strategy for a new or different environment was to edit a few config files and then run some batch files and then edit some more config files, and then it goes down in a steaming pile of failure, I would buy a LOT of Sriracha.

(Picture http://theoatmeal.com/)

Here’s a config file. Lets say we need to edit that connection string:

<Setting name="ConnectionString" value="Data Source=(local); Initial Catalog=SportsCommander; Integrated Security=true;" />

Now let’s say we are deploying to our QA server. So after we deploy, we fire up our handy Notepad, and edit it:

<Setting name="ConnectionString" value="Data Source=SCQAServ; Initial Catalog=SportsCommander; Integrated Security=true;" />

OK good. Actually not good. The server name is SCQASrv not SCQAServ.

<Setting name="ConnectionString" value="Data Source=SCQASrv; Initial Catalog=SportsCommander; Integrated Security=true;" />

OK better. But wait, integrated security works great in your local dev environment, but in QA we need to use a username and password.

<Setting name="ConnectionString" value="Data Source=SCQASrv; Initial Catalog=SportsCommander; UserID=qasite; Password=&SE&RW#$" />

OK cool. Except you can’t put & in an XML file. So we have to encode that.

<Setting name="ConnectionString" value="Data Source=SCQASrv; Initial Catalog=SportsCommander; UserID=qasite; Password=&SE&RW#$" />

And you know what? It’s User ID, not User ID.

<Setting name="ConnectionString" value="Data Source=SCQASrv; Initial Catalog=SportsCommander; User ID=qasite; Password=&SE&RW#$" />

OK, that’s all there is too it! Let’s do it again tomorrow. Make sure you don’t burn you don’t burn your fingers on this blistering fast development productivity.

I know this sounds absurd, but the reality is that for a lot of people, this really is their deployment methodology. The might have production deployments automated, but their lower environments (DEV/QA/etc) are full of manual steps. Or better yet, they have automated their lower environments because they deploy there every day, but their production deployments is manual because they only do it once per month.

And you know know what I’ve learned, the hard and maddeningly painful way? Manual process fails. Consistently. And more importantly, it can’t be avoided.

Storytime

A common scenario you see is a developer or an operations person (but of course never both at the same time, that would ruin the blame game) is charged with deploying an application. After many iterations, the deployment process has been clearly defined out as 17 manual steps. This has been done enough times that the whole process is fully documented, with a checklist, and the folks running the deployment have done it enough times that they could do it in their sleep.

The only problem is that in the last deployment, one of the files didn’t get copied. The time before that, the staging file was copied instead of the production file. And the time before that, they put a typo into the config.

Is the deployer an idiot? No, as a matter of fact, the reason that he or she was entrusted with such an important role was that he or she was the most experienced and disciplined person on the team and was intimately familiar with the workings of the entire system.

Were the instructions wrong? Nope, if the instructions were followed to the letter.

Was the process new? No again, the same people have been doing this for a year.

At this point, the managers are exasperated, because no matter how much effort we put into formalizing the process, no matter how much documentation and how many checklists, we’re still getting failures. It’s hard for the mangers to not assume that the deployers are morons, and the deployers are faced with the awful reality of going into every deployment knowing that it WILL be painful, and they WILL get blamed.

Note to management: Good people don’t stick around for this kind of abuse. Some people will put up with it. But trust me, you don’t want those people.

The lesson

The kick in the pants is, people are human. They make mistakes. A LOT Of mistakes. And when you jump down their throat on every mistake, they learn to stop making mistakes by not doing anything.

This leads us to Mooney’s Law Of Guaranteed Failure (TM):

In the software business, every manual process will suffer at least a 10% failure rate, no matter how smart the person executing the process. No amount of documentation or formalization will truly fix this, the only resolution is automation.

So the next time Jimmy screws up the production deployment, don’t yell at him (or sneer behind his back) “how hard is it to follow the 52-step 28-page instructions!” Just remember that it is virtually impossible.

Also, step back and look at your day to day development process. Almost everything you do during the day besides writing code is a manual process full of failure (coding is too, but that’s what you’re actually get getting paid for). Like:

When you are partially checking in some changes to source control but trying to leave other changes checked out

When you need to edit a web.config connection string every time you get latest or check in

When you are interactively merging branches

When you are doing any deployment that involves editing a config or running certain batch files in order or entering values into an MSI interface, or is anything more than “click the big red button”

When you are setting up a new server and creating user or editing folder permissions or creating MSMQ queues or setting up IIS virtual directories.

When you are copying your hours from Excel into the ridiculously fancy but still completely unusable timesheet website

When, instead of entering your hours into a timesheet website, you are emailing them to somebody

When you are trying to figure out which version of “FeatureRequirements_New_Latest_Latest.docx” is actually the “latest”

When you are updating deploying database changes by trying to remember which tables you added to your local database or which scripts have or have not been run against production yet

It’s actually easier to find these things than you think. The reason is, again, it is just about everything you do all day besides coding. It’s all waste. It’s all manual. And it’s all guaranteed to fail. Find a way to take that failure out of your hands and bath it in the white purifying light of automation. Sure it takes time, but with a little time investment, you’ll be amazed how much time you have when you are not wasting it with amazing stupid busywork and guaranteed failure all day.