This post takes a look at how to speed up your Docker-based builds on Travis CI, CircleCI, GitLab CI/CD, and GitHub Actions with Docker layer Caching and BuildKit.

Contents

Docker Layer Caching

Docker caches each layer as an image is built, and each layer will only be re-built if it or the layer above it has changed since the last build. So, you can significantly speed up builds with Docker cache. Let's take a look at a quick example.

Dockerfile:

# pull base image FROM python:3.8.3-slim # install netcat RUN apt-get update && \ apt-get -y install netcat && \ apt-get clean # set working directory WORKDIR /usr/src/app # install requirements COPY ./requirements.txt . RUN pip install -r requirements.txt # add app COPY . . # run server CMD gunicorn -b 0 .0.0.0:5000 manage:app

You can find the full source code for this project in the docker-ci-cache repo on GitHub.

The first Docker build can take several minutes to complete, depending on your connection speed. Subsequent builds should only take a few seconds since the layers get cached after that first build:

Sending build context to Docker daemon 721 .4kB Step 1 /7 : FROM python:3.8.3-slim ---> 091e4db1c781 Step 2 /7 : RUN apt-get update && apt-get -y install netcat && apt-get clean ---> Using cache ---> 74f2b1dca14e Step 3 /7 : WORKDIR /usr/src/app ---> Using cache ---> 0afd94dba053 Step 4 /7 : COPY ./requirements.txt . ---> Using cache ---> 7249c15cbee8 Step 5 /7 : RUN pip install -r requirements.txt ---> Using cache ---> 13bd0d323961 Step 6 /7 : COPY . . ---> Using cache ---> 1bc421036d94 Step 7 /7 : CMD gunicorn -b 0 .0.0.0:5000 manage:app ---> Using cache ---> 18df0fe6ba3f Successfully built 18df0fe6ba3f

Even if you make a change to the source code it should still only take a few seconds to build as the dependencies will not need to be downloaded. Only the last two layers have to be re-built, in other words:

Step 6 /7 : COPY . . ---> f99ca1babd63 Step 7 /7 : CMD gunicorn -b 0 .0.0.0:5000 manage:app ---> Running in 0054b494cdc6

To avoid invalidating the cache:

Start your Dockerfile with commands that are less likely to change Place commands that are more likely to change (like COPY . . ) as late as possible Add only the necessary files (use a .dockerignore file)

BuildKit

If you're using a Docker version >= 19.03 you can use BuildKit, a container image builder, in place of the traditional image builder back-end inside the Docker engine. Without BuildKit, if an image doesn't exist on your local image registry, you would need to pull the remote images before building in order to take advantage of Docker layer caching.

Example:

$ docker pull mjhea0/docker-ci-cache:latest $ docker docker build --tag mjhea0/docker-ci-cache:latest .

With BuildKit, you don't need to pull the remote images before building since it caches each build layer in your image registry. Then, when you build the image, each layer is downloaded as needed during the build.

To enable BuildKit, set the DOCKER_BUILDKIT environment variable to 1 . Then, to turn on the inline layer caching, use the BUILDKIT_INLINE_CACHE build argument.

Example:

export DOCKER_BUILDKIT = 1 # Build and cache image $ docker build --tag mjhea0/docker-ci-cache:latest --build-arg BUILDKIT_INLINE_CACHE = 1 . # Build image from remote cache $ docker build --cache-from mjhea0/docker-ci-cache:latest .

CI Environments

Since CI platforms provide a fresh environment for every build, you'll need to use a remote image registry as the source of the cache for BuildKit's layer caching.

Steps:

Log in to the image registry (like Docker Hub, Elastic Container Registry (ECR), and Quay, to name a few) It's worth noting that both GitLab and GitHub have their own registries for use within your repositories (both public and private) on their platforms -- GitLab Container Registry and GitHub Packages), respectively. Use Docker build's --cache-from option to use the existing image as the cache source Push the new image to the registry if the build is successful

Let's look at how to do this on Travis CI, CircleCI, GitLab CI/CD, and GitHub Actions, using both single and multi-stage Docker builds with and without Docker Compose. Each of the examples use Docker Hub as the image registry with REGISTRY_USER and REGISTRY_PASS set as variables in the CI builds in order to push to and pull from the registry.

Make sure to set REGISTRY_USER and REGISTRY_PASS as environment variables in the build environment: Travis CI CircleCI GitLab CI/CD GitHub Actions

Single-stage Builds

Travis CI:

# _config-examples/single-stage/.travis.yml os : linux dist : bionic env : global : CACHE_IMAGE : mjhea0/docker-ci-cache DOCKER_BUILDKIT : 1 before_install : - set -e - echo '{"experimental":true}' | sudo tee /etc/docker/daemon.json - export DOCKER_CLI_EXPERIMENTAL=enabled - docker run --rm --privileged docker/binfmt:a7996909642ee92942dcd6cff44b9b95f08dad64 - curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - - sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" - sudo apt-get update - sudo apt-get -y -o Dpkg::Options::="--force-confnew" install docker-ce=5:19.03.8~3-0~ubuntu-bionic - docker info before_script : - docker login -u $REGISTRY_USER -p $REGISTRY_PASS script : - docker build --cache-from $CACHE_IMAGE:latest --tag $CACHE_IMAGE:latest --build-arg BUILDKIT_INLINE_CACHE=1 "." after_success : - docker push $CACHE_IMAGE:latest

CircleCI:

# _config-examples/single-stage/circle.yml version : 2.1 jobs : build : machine : image : ubuntu-1604:202004-01 environment : CACHE_IMAGE : mjhea0/docker-ci-cache DOCKER_BUILDKIT : 1 steps : - checkout - run : name : Log in to docker hub command : docker login -u $REGISTRY_USER -p $REGISTRY_PASS - run : name : Build from dockerfile command : | docker build \ --cache-from $CACHE_IMAGE:latest \ --tag $CACHE_IMAGE:latest \ --build-arg BUILDKIT_INLINE_CACHE=1 \ "." - run : name : Push to docker hub command : docker push $CACHE_IMAGE:latest

GitLab CI/CD:

# _config-examples/single-stage/.gitlab-ci.yml image : docker:stable services : - docker:dind variables : DOCKER_DRIVER : overlay2 CACHE_IMAGE : mjhea0/docker-ci-cache DOCKER_BUILDKIT : 1 stages : - build docker-build : stage : build before_script : - docker login -u $REGISTRY_USER -p $REGISTRY_PASS script : - docker build --cache-from $CACHE_IMAGE:latest --tag $CACHE_IMAGE:latest --file ./Dockerfile --build-arg BUILDKIT_INLINE_CACHE=1 "." after_script : - docker push $CACHE_IMAGE:latest

GitHub Actions:

# _config-examples/single-stage/github.yml name : Docker Build on : [ push ] env : CACHE_IMAGE : mjhea0/docker-ci-cache DOCKER_BUILDKIT : 1 jobs : build : name : Build Docker Image runs-on : ubuntu-latest steps : - name : Checkout master uses : actions/ [email protected] - name : Log in to docker hub run : docker login -u ${{ secrets.REGISTRY_USER }} -p ${{ secrets.REGISTRY_PASS }} - name : Build from dockerfile run : | docker build \ --cache-from $CACHE_IMAGE:latest \ --tag $CACHE_IMAGE:latest \ --build-arg BUILDKIT_INLINE_CACHE=1 \ "." - name : Push to docker hub run : docker push $CACHE_IMAGE:latest

Compose

If you're using Docker Compose, you can add the cache_from option to the compose file, which maps back to the docker build --cache-from <image> command when you run docker-compose build Example:

version : '3.7' services : web : build : context : . cache_from : - mjhea0/docker-ci-cache:latest image : mjhea0/docker-ci-cache:latest

To take advantage of BuildKit, make sure you're using a version of Docker Compose >= 1.25.0. To enable BuildKit, set the DOCKER_BUILDKIT and COMPOSE_DOCKER_CLI_BUILD environment variables to 1 . Then, again, to turn on the inline layer caching, use the BUILDKIT_INLINE_CACHE build argument.

Travis CI:

# _config-examples/single-stage/compose/.travis.yml os : linux dist : bionic env : global : DOCKER_COMPOSE_VERSION : 1.25.4 CACHE_IMAGE : mjhea0/docker-ci-cache DOCKER_BUILDKIT : 1 COMPOSE_DOCKER_CLI_BUILD : 1 before_install : - set -e # install docker - echo '{"experimental":true}' | sudo tee /etc/docker/daemon.json - export DOCKER_CLI_EXPERIMENTAL=enabled - docker run --rm --privileged docker/binfmt:a7996909642ee92942dcd6cff44b9b95f08dad64 - curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - - sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" - sudo apt-get update - sudo apt-get -y -o Dpkg::Options::="--force-confnew" install docker-ce=5:19.03.8~3-0~ubuntu-bionic - docker info # install docker compose - sudo rm /usr/local/bin/docker-compose - curl -L https://github.com/docker/compose/releases/download/${DOCKER_COMPOSE_VERSION}/docker-compose-`uname -s`-`uname -m` > docker-compose - chmod +x docker-compose - sudo mv docker-compose /usr/local/bin before_script : - docker login -u $REGISTRY_USER -p $REGISTRY_PASS script : - docker-compose build --build-arg BUILDKIT_INLINE_CACHE=1 after_success : - docker push $CACHE_IMAGE:latest

CircleCI:

# _config-examples/single-stage/compose/circle.yml version : 2.1 jobs : build : machine : image : ubuntu-1604:202004-01 environment : CACHE_IMAGE : mjhea0/docker-ci-cache DOCKER_BUILDKIT : 1 COMPOSE_DOCKER_CLI_BUILD : 1 steps : - checkout - run : name : Log in to docker hub command : docker login -u $REGISTRY_USER -p $REGISTRY_PASS - run : name : Build images command : docker-compose build --build-arg BUILDKIT_INLINE_CACHE=1 - run : name : Push to docker hub command : docker push $CACHE_IMAGE:latest

GitLab CI/CD:

# _config-examples/single-stage/compose/.gitlab-ci.yml image : docker/compose:latest services : - docker:dind variables : DOCKER_DRIVER : overlay2 CACHE_IMAGE : mjhea0/docker-ci-cache DOCKER_BUILDKIT : 1 COMPOSE_DOCKER_CLI_BUILD : 1 stages : - build docker-build : stage : build before_script : - docker login -u $REGISTRY_USER -p $REGISTRY_PASS script : - docker-compose build --build-arg BUILDKIT_INLINE_CACHE=1 after_script : - docker push $CACHE_IMAGE:latest

GitHub Actions:

# _config-examples/single-stage/compose/github.yml name : Docker Build on : [ push ] env : CACHE_IMAGE : mjhea0/docker-ci-cache DOCKER_BUILDKIT : 1 COMPOSE_DOCKER_CLI_BUILD : 1 jobs : build : name : Build Docker Image runs-on : ubuntu-latest steps : - name : Checkout master uses : actions/ [email protected] - name : Log in to docker hub run : docker login -u ${{ secrets.REGISTRY_USER }} -p ${{ secrets.REGISTRY_PASS }} - name : Build Docker images run : docker-compose build --build-arg BUILDKIT_INLINE_CACHE=1 - name : Push to docker hub run : docker push $CACHE_IMAGE:latest

Multi-stage Builds

With the multi-stage build pattern, you'll have to apply the same workflow (build, then push) for each intermediate stage since those images are discarded before the final image is created. The --target option can be used to build each stage of the multi-stage build separately.

Dockerfile.multi:

# base FROM python:3.8.3 as base COPY ./requirements.txt / RUN pip wheel --no-cache-dir --no-deps --wheel-dir /wheels -r requirements.txt # stage FROM python:3.8.3-slim RUN apt-get update && \ apt-get -y install netcat && \ apt-get clean WORKDIR /usr/src/app COPY --from = base /wheels /wheels COPY --from = base requirements.txt . RUN pip install --no-cache /wheels/* COPY . /usr/src/app CMD gunicorn -b 0 .0.0.0:5000 manage:app

Travis CI:

# _config-examples/multi-stage/.travis.yml os : linux dist : bionic env : global : CACHE_IMAGE : mjhea0/docker-ci-cache DOCKER_BUILDKIT : 1 before_install : - set -e - echo '{"experimental":true}' | sudo tee /etc/docker/daemon.json - export DOCKER_CLI_EXPERIMENTAL=enabled - docker run --rm --privileged docker/binfmt:a7996909642ee92942dcd6cff44b9b95f08dad64 - curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - - sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" - sudo apt-get update - sudo apt-get -y -o Dpkg::Options::="--force-confnew" install docker-ce=5:19.03.8~3-0~ubuntu-bionic - docker info before_script : - docker login -u $REGISTRY_USER -p $REGISTRY_PASS script : - docker build --target base --cache-from $CACHE_IMAGE:base --tag $CACHE_IMAGE:base --file ./Dockerfile.multi --build-arg BUILDKIT_INLINE_CACHE=1 "." - docker build --cache-from $CACHE_IMAGE:base --cache-from $CACHE_IMAGE:stage --tag $CACHE_IMAGE:stage --file ./Dockerfile.multi --build-arg BUILDKIT_INLINE_CACHE=1 "." after_success : - docker push $CACHE_IMAGE:base - docker push $CACHE_IMAGE:stage

CircleCI:

# _config-examples/multi-stage/circle.yml version : 2.1 jobs : build : machine : image : ubuntu-1604:202004-01 environment : CACHE_IMAGE : mjhea0/docker-ci-cache DOCKER_BUILDKIT : 1 steps : - checkout - run : name : Log in to docker hub command : docker login -u $REGISTRY_USER -p $REGISTRY_PASS - run : name : Build base from dockerfile command : | docker build \ --target base \ --cache-from $CACHE_IMAGE:base \ --tag $CACHE_IMAGE:base \ --file ./Dockerfile.multi \ --build-arg BUILDKIT_INLINE_CACHE=1 \ "." - run : name : Build stage from dockerfile command : | docker build \ --cache-from $CACHE_IMAGE:base \ --cache-from $CACHE_IMAGE:stage \ --tag $CACHE_IMAGE:stage \ --file ./Dockerfile.multi \ --build-arg BUILDKIT_INLINE_CACHE=1 \ "." - run : name : Push base image to docker hub command : docker push $CACHE_IMAGE:base - run : name : Push stage image to docker hub command : docker push $CACHE_IMAGE:stage

GitLab CI/CD:

# _config-examples/multi-stage/.gitlab-ci.yml image : docker:stable services : - docker:dind variables : DOCKER_DRIVER : overlay2 CACHE_IMAGE : mjhea0/docker-ci-cache DOCKER_BUILDKIT : 1 stages : - build docker-build : stage : build before_script : - docker login -u $REGISTRY_USER -p $REGISTRY_PASS script : - docker build --target base --cache-from $CACHE_IMAGE:base --tag $CACHE_IMAGE:base --file ./Dockerfile.multi --build-arg BUILDKIT_INLINE_CACHE=1 "." - docker build --cache-from $CACHE_IMAGE:base --cache-from $CACHE_IMAGE:stage --tag $CACHE_IMAGE:stage --file ./Dockerfile.multi --build-arg BUILDKIT_INLINE_CACHE=1 "." after_script : - docker push $CACHE_IMAGE:stage

GitHub Actions:

# _config-examples/multi-stage/github.yml name : Docker Build on : [ push ] env : CACHE_IMAGE : mjhea0/docker-ci-cache DOCKER_BUILDKIT : 1 jobs : build : name : Build Docker Image runs-on : ubuntu-latest steps : - name : Checkout master uses : actions/ [email protected] - name : Log in to docker hub run : docker login -u ${{ secrets.REGISTRY_USER }} -p ${{ secrets.REGISTRY_PASS }} - name : Build base from dockerfile run : | docker build \ --target base \ --cache-from $CACHE_IMAGE:base \ --tag $CACHE_IMAGE:base \ --file ./Dockerfile.multi \ --build-arg BUILDKIT_INLINE_CACHE=1 \ "." - name : Build stage from dockerfile run : | docker build \ --cache-from $CACHE_IMAGE:base \ --cache-from $CACHE_IMAGE:stage \ --tag $CACHE_IMAGE:stage \ --file ./Dockerfile.multi \ --build-arg BUILDKIT_INLINE_CACHE=1 \ "." - name : Push base image to docker hub run : docker push $CACHE_IMAGE:base - name : Push stage image to docker hub run : docker push $CACHE_IMAGE:stage

Compose

Example compose file:

version : '3.7' services : web : build : context : . cache_from : - mjhea0/docker-ci-cache:stage image : mjhea0/docker-ci-cache:stage

Travis CI:

# _config-examples/multi-stage/compose/.travis.yml os : linux dist : bionic env : global : DOCKER_COMPOSE_VERSION : 1.25.4 CACHE_IMAGE : mjhea0/docker-ci-cache DOCKER_BUILDKIT : 1 COMPOSE_DOCKER_CLI_BUILD : 1 before_install : - set -e # install docker - echo '{"experimental":true}' | sudo tee /etc/docker/daemon.json - export DOCKER_CLI_EXPERIMENTAL=enabled - docker run --rm --privileged docker/binfmt:a7996909642ee92942dcd6cff44b9b95f08dad64 - curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - - sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" - sudo apt-get update - sudo apt-get -y -o Dpkg::Options::="--force-confnew" install docker-ce=5:19.03.8~3-0~ubuntu-bionic - docker info # install docker compose - sudo rm /usr/local/bin/docker-compose - curl -L https://github.com/docker/compose/releases/download/${DOCKER_COMPOSE_VERSION}/docker-compose-`uname -s`-`uname -m` > docker-compose - chmod +x docker-compose - sudo mv docker-compose /usr/local/bin before_script : - docker login -u $REGISTRY_USER -p $REGISTRY_PASS script : - docker build --target base --cache-from $CACHE_IMAGE:base --tag $CACHE_IMAGE:base --file ./Dockerfile.multi --build-arg BUILDKIT_INLINE_CACHE=1 "." - docker-compose -f docker-compose.multi.yml build - docker-compose build --build-arg BUILDKIT_INLINE_CACHE=1 after_success : - docker push $CACHE_IMAGE:base - docker push $CACHE_IMAGE:stage

CircleCI:

# _config-examples/multi-stage/compose/circle.yml version : 2.1 jobs : build : machine : image : ubuntu-1604:202004-01 environment : CACHE_IMAGE : mjhea0/docker-ci-cache DOCKER_BUILDKIT : 1 COMPOSE_DOCKER_CLI_BUILD : 1 steps : - checkout - run : name : Log in to docker hub command : docker login -u $REGISTRY_USER -p $REGISTRY_PASS - run : name : Build base from dockerfile command : | docker build \ --target base \ --cache-from $CACHE_IMAGE:base \ --tag $CACHE_IMAGE:base \ --file ./Dockerfile.multi \ --build-arg BUILDKIT_INLINE_CACHE=1 \ "." - run : name : Build Docker images command : docker-compose -f docker-compose.multi.yml build --build-arg BUILDKIT_INLINE_CACHE=1 - run : name : Push base image to docker hub command : docker push $CACHE_IMAGE:base - run : name : Push stage image to docker hub command : docker push $CACHE_IMAGE:stage

GitLab CI/CD:

# _config-examples/multi-stage/compose/.gitlab-ci.yml image : docker/compose:latest services : - docker:dind variables : DOCKER_DRIVER : overlay CACHE_IMAGE : mjhea0/docker-ci-cache DOCKER_BUILDKIT : 1 COMPOSE_DOCKER_CLI_BUILD : 1 stages : - build docker-build : stage : build before_script : - docker login -u $REGISTRY_USER -p $REGISTRY_PASS script : - docker build --target base --cache-from $CACHE_IMAGE:base --tag $CACHE_IMAGE:base --file ./Dockerfile.multi --build-arg BUILDKIT_INLINE_CACHE=1 "." - docker-compose -f docker-compose.multi.yml build --build-arg BUILDKIT_INLINE_CACHE=1 after_script : - docker push $CACHE_IMAGE:base - docker push $CACHE_IMAGE:stage

GitHub Actions:

# _config-examples/multi-stage/compose/github.yml name : Docker Build on : [ push ] env : CACHE_IMAGE : mjhea0/docker-ci-cache DOCKER_BUILDKIT : 1 COMPOSE_DOCKER_CLI_BUILD : 1 jobs : build : name : Build Docker Image runs-on : ubuntu-latest steps : - name : Checkout master uses : actions/ [email protected] - name : Log in to docker hub run : docker login -u ${{ secrets.REGISTRY_USER }} -p ${{ secrets.REGISTRY_PASS }} - name : Build base from dockerfile run : | docker build \ --target base \ --cache-from $CACHE_IMAGE:base \ --tag $CACHE_IMAGE:base \ --file ./Dockerfile.multi \ --build-arg BUILDKIT_INLINE_CACHE=1 \ "." - name : Build images run : docker-compose -f docker-compose.multi.yml build --build-arg BUILDKIT_INLINE_CACHE=1 - name : Push base image to docker hub run : docker push $CACHE_IMAGE:base - name : Push stage image to docker hub run : docker push $CACHE_IMAGE:stage

Conclusion

The caching strategies outlined in this post should work well for single-stage builds and multi-stage builds with two or three stages.

Each stage added to a build step requires a new build and push along with the addition of the --cache-from options for each parent stage. Thus, each new stage will add more clutter, making the CI file increasingly more difficult to read. Fortunately, BuildKit supports multi-stage builds with Docker layer caching built using a single stage. Review the following posts for more info on such advanced BuildKit patterns:

Finally, it's important to note that while caching may speed up your CI builds, you should re-build your images without cache from time to time in order to download the latest OS patches and security updates. For more on this, review this thread.

--

The code can be found in the docker-ci-cache repo:

Cheers!