This insight came to me while working on the current project. We are developing a rather complex Django application, and when it came to a listing of dependencies we tried to keep it simple (if not to say naive).

So we just had requirements directory with base.txt, test.txt, and local.txt in it. base.txt had a list of direct dependencies constrained to compatible release that was latest when we introduced the package. Like so:

Django~=1.9.0

djangorestframework~=3.5.3

During a year of active development, this list grew a bit to 72 base dependencies, 31 test and 13 local. Among those 72 there are five libraries that we developed primarily for this project, they move fast but in the entirely controlled environment.

More often than we would like to, some of our external dependencies are breaking something we rely on. And of course, they do it without major version increment.

Naturally, we came to the idea of pinning dependencies and converted each of the files in requirements directory to a pair: base.in + base.txt. The former is a list of direct dependencies almost without version constraint at all. The latter is a recursive list of all dependencies and sub-dependencies with their versions being hard-pinned (e.g., ==3.5.3).

This move solved the pain point — now dependency updates stopped not suddenly breaking the deployment. But each time we had [frequent] internal library release, we had to update pins and redeploy the main application.

The first solution was just to drop version pins for internal libraries, so the latest release was picked during the build. We had this setup for quite some time until we needed to make a synchronized change between an internal library and the main application. We either had to be very attentive to deployments timeline or…

Complete the dependency management workflow with soft-pins for internal libraries. So we started to set compatible releases to them (e.g. ~=1.2.3.post123+a7b327). Now, all minor fixes are going to the next deployment without changes to the base.txt file, and major version increments fence against breaking changes.

I have packed this solution as a Python package that you can adapt for your project: https://github.com/peterdemin/pip-compile-multi