I previously discussed how distributed version control systems (DVCS) can help with keeping the tip always compilable. DVCS is also useful in making sure the tip always passes a test suite, or maintains any other standard of quality. It does this by giving more control where control is needed.

When you first hear about DVCS, that statement seems counter-intuitive. How can a decentralized system give more control to the central authorities? The key is that by giving up some control where it wasn’t needed, you gain more control over the important parts, sort of like guarding a prisoner in a 10×10 cell instead of a 10 acre field.

In our company we have product-specific code for a number of embedded products, and a large base of code shared between products. Because developers typically only have the hardware for the product they are working on, someone who makes changes to the shared code can only test it on that one product. As a result, although breaking your own product’s build is quite rare, shared code changes that break other products’ builds are much too common.

So how can we set up our branches to mitigate this problem? The answer lies in examining who we want to have control over each branch. At the same time, we think about what our ideal log would look like.

We want the “official” branch for a product to consist of a series of successfully tested builds. We want to be able to take any revision of that branch at any time with confidence. Obviously, the person best suited to controlling that branch is the lead tester for the product. The log at that level would look something like this:

Here we have the log for a fictional Product A. Notice we only have one person committing here, the lead tester for product A. This responsibility could be rotated among all the testers, and could be enforced by only giving the product A testers write permissions on the branch, or more loosely enforced just by social convention. The important thing to notice is that the test group has more control over the product’s official branch than the typical centralized model, where all developers have commit access.

Okay, so where do the developers come in? Developers like to have control, but it looks like you just took a whole bunch of control away from them. For that we expand the log to the next level:

At this level you can clearly see which features made it into each promoted build. Developers for product A have full control over this development branch and can set permissions on it as they see fit. This includes preventing the test group from writing to this branch if desired, because all they need is to be able to do is pull. In other words, each group has full control over exactly the areas they need it. A developer’s view in their daily work looks like this:

This shows the changes for Product A as if Product A is the most important product in the world. All the shared changes from Product B are hidden behind the plus sign, which you only click to expand if you want to see the details. A developer on Product B would see a similar view of Project B’s development branch, as if Product B is the most important product in the world.

Here you can also see two possible approaches to receiving changes from the shared code. One is what Amy did in revision 2.2.1. For her change, she knew she needed some changes in shared code from Product B in order to proceed with her work, so she merged them in. The other alternative is an assigned branch manager thinking it’s been a while since we synced up, so he more formally merges the changes in. You can do both if you want.

Notice that either way the developers for product A have full control over when shared changes get pulled into their product’s build. If the shared changes cause a product-specific compile or run time error, Amy simply doesn’t commit them until she has worked with the Product B developers to get it resolved. In the mean time, Alice, Arnold, and the test team are all working from a clean baseline.

Another method we use to improve code quality is code reviews. We use an online collaborative review tool, and it generally takes a few days to finish one. In the mean time, you start work on your next change, going back as needed to fix the defects found in the review. Turns out DVCS is useful in this situation as well, because we can create a new branch for our work as soon as the review is started, like so:

In conclusion, distributed version control offers many flexible ways to increase build stability through more local control and judicious design of a branching model. What branching models have you found to be successful?