Why you should run one service per container

In my last post I described how I set up a single Docker image that runs several services using runit. I also said right from the start that I do not recommend to do so. Now I’ll describe why I think multiple containers is the way to go. This is probably mostly preaching to the choir, but here we go. I’ll make the assumption that the single-container solution we’re criticizing it build around runit.

What’s wrong with the single-container approach?

Probably lots of implicit dependencies between services

You may have service A talking to service B on a Unix domain socket, service B talking to service C via 127.0.0.1:some-port while everybody assumes that everything is reachable via localhost. Then service A really is nginx and will also be used as http proxy because you know, it is already running.

“Well. It works, doesn’t it?” Yes, maybe. But you’re missing out on the chance of clearly separating concerns here. If you run each service in a separate container, this forces you to apply a 12-factor approach to “backing services” and clearly document the links between the services. Of course this also means you can get rid of your single-machine setup at any time and move the containers around, put them behind a load balancer or caching proxy etc. without affecting the clear interface between services that you’ve now achieved.

Huge docker image sizes

As the single-container approach tends to treat the container like a Virtual Machine, the size of the image is just like a VM image - ours were > 1 GB. They also invite developers and admins to poke around inside the running container, because they want to inspect things - and because they need to. You’ll not necessarily saving on combined image sizes if you use multiple containers, but multiple smaller ones are easier to handle. Both at image build time and at image pull time.

Signals are broken

If you run multiple services using runit, you will use runsvdir as master process (i. e. PID 1). runsvdir’s signal handling behavior however, is not optimal. It will only send SIGTERM to its children when it receives SIGHUP, but not tricky to actively SIGKILL them after a timeout.

runsv’s default signal handling is not optimal either. A TERM signal will send a TERM to its child. No KILL there either. This could be worked around with customizing control of the “control characters” runit supports.

What this means is that it’s hard to achieve a graceful shutdown of services without lots of manual twiddling, and probably replacing runsvdir. We never bothered to do so.

Advantages of a multi-container approach

You can scale services independently now

In our single-container setup we had web app, Redis-based queue and workers in the same container. Moving the Redis to a datacenter-wide one and splitting web app and workers into separate containers immediately gave us better scalability. We can now scale web app and workers independently if we want to do so.

Much improved logging

If you run each service in a separate container, you can use Docker’s log stream to inspect the output of the service. And if you want to go one step further, like we currently do, you can switch Docker’s default logging system to syslog and get centralized logging almost out of the box.

Additional security per container possible

As you’re running each service in a separate container, you can tighten up security depending on the needs of the container. You can run the container with a separate user from the start. No need to switch users in the container. You can run the container with a read-only filesystem, remove any kernel functionality not needed, remove network access unless necessary, etc.

Potentially much smaller images

As you don’t need people to poke around in containers, you can do some pretty drastic steps to reduce image sizes. There are automated tools for that like docker-slim. But you can also achieve it manually.

In one experiment I performed, the original 1 GB image that was reduced to 425 MB in the single-container scenario could be trimmed down to 140 GB by manually removing any files that the service does not really need. These includer /usr/?bin, /?bin, /var, /usr/share and others.

You’re not fighting the system any longer

The Docker creators intended Docker to be a solution for application containers (i. e. one process per container), so that’s the use case where Docker shines. Any time you’re bending a technology in a way that it’s not intended to be used you are fighting an uphill battle that’s probably not worth it.