In Shooting Yourself in the Foot with Customer Branches, I described a colleague's current work project -- specifically, how they multiplied the amount of code they had to maintain for every new customer profile.

That's not their only problem. His current frustration stemmed from inadequate testing (and very poor QA management) due to poor handling of code freezes.

A popular (but wrong) theory of software development argues that development progresses in linear and predictable cycles. You gather the specifications. You design the subsystems. You code features. You stabilize a release branch and close it to all but bugfixes (this is the feature freeze). Then you perform detailed QA to try to find and diagnose all of the bugs possible before the release.

That's a very popular theory -- it's almost as popular as it is completely and unarguably wrong.

Management has realized that eventually they have to ship software. There are always features to add and bugs to fix and subsystems to refactor, but the only way to release software is to draw a line on the calendar and say "We're shipping whatever's ready then." So far, so good.

Of course, you can't ship without this cool feature. You can't ship when the long-promised feature isn't ready. You can't ship with this many bugs. You can't ship because there's always something someone wants, and if they have to wait another 18 months for the next version, they'll take their money and find someone else to pay for the software. There's always a rush and a crunch at the end of a release cycle when everyone suddenly realizes that this time, they're not kidding about the release date.

Try being in QA when this happens. All of a sudden, the day before the code freeze, a dozen feature branches that weren't ready land on trunk. QA hasn't seen them. QA didn't even know they built. (Some of them don't.)

Now QA has the unenviable task of working around the damage of features that may not even meet the basic needs of customers but absolutely can't slip because the next release is months or years away. QA gets in arguments about features, because they don't work. Developers don't want to hear that. Management doesn't want to hear that. Change is bad during a freeze. Bugs in design or implementation or requirements take too long to fix, and the relentless drums of "But we promised to release this month!" keep counting down the days.

Eventually, a release candidate escapes, as an admission that no one knows for sure if the software works, and wouldn't it be swell if someone might try to test it, because everyone thinks it probably works, because the project manager has spent the last several weeknights in the bug tracker, reclassifying bug severities and reassigning them to later milestones because one of the criteria for a release candidate is "Zero blocking bugs" and another is "Zero critical severity bugs" and if the difference between getting a bonus this year or making the quarterly numbers means that customers will file bugs in a few months that everyone argued about fixing this week, well, there's a chance customers won't notice. All software has bugs anyway.

Bugs linger in the bug tracker. Their priorities and severities mysteriously change. Duplicate bugs pile up. The software gets released and everyone takes a long weekend....

... and on Tuesday morning, they're back at the grind, preparing for the next patch release cycle to fix all of the problems they knew about before but couldn't fix, because you can't make changes during a code freeze, because no one knows if the code works, because you can't create a release candidate with that bug count, because releasing software is a very big deal, because it's just too darn hard to prepare a release with all of this ceremony. At least they have daily status meetings, though.

Meanwhile, another company down the street knows what "done" means. They keep their bug count low. They can release a new version of their software stable enough for customers to use every day, if they want. Releases aren't a big deal. They're boring and common place. (How do they do it? See The Rapid Release Tautology.)

If you're the first company, the second company has already eaten your lunch, and they'll continue to do so every week, until you fail or until you figure out how to release software.