Using Gitlab pipelines to deploy Python packages in production and staging environments

Deploy staging builds to a private Python package index with PyPRI, and production builds to PyPI

Gitlab pipelines are a group of jobs that get executed in stages whenever you push code to your Gitlab repository. In this tutorial we will look at how to create a Gitlab pipeline for a Python repository, that packages our code using setuptools and pushes those packages to a private package index (for staging) or to PyPI (for release).

Prerequisites

In order to complete this tutorial, you will need the following:

A gitlab account and a gitlab project with some Python code in it

A PyPI account, for making production-ready packages public

A PyPRI account, for storing your private Python packages until they are ready to be release publicly. For more information on creating a PyPRI account, see here

PyPRI lets you create a personal cloud repository to store your private or non-production ready code securely. You can sign up for a free account which will allow you to create an unlimited number of public repositories

Structuring the code

If you don’t already have one, create a new Gitlab project and clone it to your local machine. Structure your repository as follows:

/my-repo

/my-awesome-package

--your code lives here--

.gitlab-ci.yml

setup.py

Put all of the code that you want to include into the my-awesome-package folder. For more information on creating a Python package, see https://python-packaging.readthedocs.io/en/latest/

Configuring your gitlab repository

By default, Gitlab pipelines are enabled on all projects which have a file called .gitlab-ci.yml which defines the stages of your pipelines (more on that later). However, in order to manage authentication to PyPI and PyPRI securely, we need to set some environmental variables. To do this, go to Settings > CI / CD, click on Variables then set the following environmental variables:

PYPRI_REPOSITORY_URL : The endpoint of your PyPRI repository, available from within PyPRI. Will be something like https://api.python-private-package-index.com/ XXXXX /

: The endpoint of your PyPRI repository, available from within PyPRI. Will be something like https://api.python-private-package-index.com/ / STAGING_USERNAME : Your PyPRI username (which is your email address)

: Your PyPRI username (which is your email address) STAGING_PASSWORD : Your PyPRI password

: Your PyPRI password PRODUCTION_USERNAME : Your PyPI username

: Your PyPI username PRODUCTION_PASSWORD: Your PyPI password

Setting CI/CD variables in Gitlab

.gitlab-ci.yml

This file is used to configure the stages and jobs of your pipeline. In the case of this example, we will configure 1 stage (deploy) and 2 jobs (deploy_staging, deploy_production).

The idea here is that whenever you push code to your repository, a tagged commit will run the deploy_production task. Any other type of push will run the deploy_staging task

image: python:3.6-alpine



stages:

- deploy



before_script:

- pip install twine

- python setup.py sdist



deploy_staging:

stage: deploy

variables:

TWINE_USERNAME: $STAGING_USERNAME

TWINE_PASSWORD: $STAGING_PASSWORD

script:

- twine upload --repository-url $PYPRI_REPOSITORY_URL dist/*

except:

- tags



deploy_production:

stage: deploy

variables:

TWINE_USERNAME: $PRODUCTION_USERNAME

TWINE_PASSWORD: $PRODUCTION_PASSWORD

script:

- twine upload dist/*

only:

- tags

Here’s a more detailed look at what’s going on above:

Image: Gitlab CI runs pipelines using Docker images. We use Alpine linux here due to its low image size, but if your Python code is more complex, then you may need to install additional dependencies, or try with another image such as python:3.6-slim

Gitlab CI runs pipelines using Docker images. We use Alpine linux here due to its low image size, but if your Python code is more complex, then you may need to install additional dependencies, or try with another image such as python:3.6-slim before_script : the commands in this section are required for both jobs, so we include them outside of the scope of the individual jobs. The two commands here install twine, which pushes packages to a Python package index, and builds your package using setuptools.

: the commands in this section are required for both jobs, so we include them outside of the scope of the individual jobs. The two commands here install twine, which pushes packages to a Python package index, and builds your package using setuptools. deploy_staging: this is the job that pushes our packaged code to PyPRI. By adding the except argument, we ensure that it doesn’t run on a tagged commit. The script argument performs the actual push to our private repository

this is the job that pushes our packaged code to PyPRI. By adding the except argument, we ensure that it doesn’t run on a tagged commit. The script argument performs the actual push to our private repository deploy_production: This job pushes our code to PyPI, where it will be available for download publicly. This job is identical to deploy_staging, apart from the TWINE_USERNAME/PASSWORD variables, the lack of a — repository-url argument (as we now want to use the default), and the addition of the only argument, which ensures that the task only runs from tagged commits

setup.py

Your setup.py controls how your code is packaged. For more information on writing a setup.py file, you can refer to the official packaging documentation here. For a quick start version, here’s the content of my version:

from setuptools import setup

import os



if os.environ.get('CI_COMMIT_TAG'):

version = os.environ['CI_COMMIT_TAG']

else:

version = os.environ['CI_JOB_ID']



setup(

name='my-awesome-package',

version=version,

description='My awesome package',

author='Me',

author_email='info@python-private-package-index.com',

license='MIT',

packages=['my-awesome-package'],

url='https://gitlab.com/pypri/pypri-gitlab-ci',

zip_safe=False

)

A note about versions: when pushing Python packages to either PyPI or PyPRI, the combination of your package name and version number must be unique, otherwise the upload will fail. This setup.py file has been written so as to avoid such conflicts. GitLab CI sets certain environmental variables in the pipelines — in this case, we are checking for the existence of the CI_COMMIT_TAG to see if the git commit has been tagged, and if so, the version is set to the value of this tag. If the tag doesn’t exist, the Gitlab CI job ID is set as the version number (this is not a semantic version, but will only appear in our staging package index)

Putting it all together

Now that we have configured our repository correctly, we can try it all out. First, let’s test that push a non-tagged commit creates a package in our private package repository:

git commit -m "Initial commit"

git push

If all goes well, we should be able to go to CI / CD > Jobs in Gitlab, and see that the deploy_staging job has completed successfully:

Completed job in Gitlab CI

If the job has passed, then you will also be able to see the package in PyPRI. Note that the Gitlab job number and the PyPRI package version are the same value:

Package in PyPRI

Finally, we can try tagging a commit and pushing that tag. This should push that package to PyPI:

git tag '1.2.7'

git push --tags

The production package on PyPI

This can now be installed by anyone with the following command: