We had to raise an old Django application from the limbo for the Mobil ET research group (GDR CNRS 3546). This blog post sums up what we have learnt improving its compatibility with newer Python/Django versions.

Let’s start with a bit of context. In the Django universe, an application is a part of a project like a plugin or a bundle in some other languages/frameworks. It is self-contained and provides a set of features. In our case, the django-tailordev-biblio application provides scientific bibliography management.

This application was written for Python 2.6 to 3.3 and Django 1.5. As expected, everything exploded when we tried to integrate it in a Python 3.6/Django 1.10 project

Captain Obvious

First, you will not be able to check your application compatibility if it is not automatically tested with an acceptable code coverage. This has been extensively covered on the Internets, no need to describe how to do it here. Instead, let’s focus on more interesting points.

1. Use a sandbox for your Django application

When developing a Django application, you will need a sandbox to run both manual & automated tests, which you could reuse to deploy a demo of your application on Heroku for instance.

In our case, a sandbox is a minimal Django project at the root of the application repository:

. ├── my_app │ ├── __init__.py │ ├── admin.py │ ├── apps.py │ ├── migrations │ ├── models.py │ ├── tests.py │ └── views.py ├── requirements │ ├── base.txt │ └── dev.txt ├── requirements.txt └── sandbox ├── __init__.py ├── manage.py ├── settings.py ├── urls.py └── wsgi.py

You can run the commands below to create this sandbox in your project:

# Create & load your virtualenv $ python3 -m venv venv $ source venv/bin/activate # Install your development requirements (at least Django) ( venv ) $ pip install -r requirements/dev.txt # Create the sandbox ( venv ) $ django-admin startproject sandbox . # Move & fix management script in the sandbox ( venv ) $ cat manage.py | sed 's/sandbox.settings/settings/' > sandbox/manage.py ( venv ) $ rm manage.py

As any Django project, you will have to add your application and its URLs to the project:

# sandbox/settings.py INSTALLED_APPS = [ 'django.contrib.admin' , 'django.contrib.auth' , 'django.contrib.contenttypes' , 'django.contrib.sessions' , 'django.contrib.messages' , 'django.contrib.staticfiles' , 'my_app' , # This is your application ] # sandbox/urls.py from django.conf.urls import include , url from django.contrib import admin urlpatterns = [ url ( '^admin/' , admin . site . urls ), # Your application urls url ( '^' , include ( 'my_app.urls' , namespace = 'my_app' )), ]

The last step is to make your application visible in the PYTHONPATH . You have two options:

# If you already have an installation script, use it! ( venv ) $ python setup.py develop # If not, add your app to the PYTHONPATH ( venv ) $ export PYTHONPATH = '.'

Now you should be able to perform database migrations and run the development server

(venv) $ python sandbox/manage.py migrate (venv) $ python sandbox/manage.py runserver

As previously mentioned, this sandbox can be used to run automated tests with pytest . Once installed with the pytest-django plugin, you have to configure it to use your sandbox. Edit the setup configuration file of your project and add a pytest section:

# setup.cfg [tool:pytest] DJANGO_SETTINGS_MODULE = sandbox.settings addopts = -vs --cov=my_app testpaths = my_app/tests

First, we declare the location of the DJANGO_SETTINGS_MODULE to use in the application tests. The path to these tests to be executed is given in the testpaths value. Between, addopts is used to define some nice default pytest options (see pytest -h ).

Nota Bene: if you want to get more insights on a concrete sandbox integration for a Django application, check out django-tailordev-biblio repository.

2. Define the scope of your compatibility matrix

Let’s now explore how your application behaves with different combinations of Python and Django, assuming your tests pass for a given Python/Django couple.

It is entirely up to you to support a wide or restricted range of Django and Python releases but we believe that a Django application should support Python 2.7 and 3.4+ combined with a still-supported Django version, i.e. Django >= 1.8 at the time of writing.

Pro tip: keep your users informed of the compatibility of your application by adding a table like the following into your project README.md :

| | Django 1.7 | ... | Django 1.10 | Django 1.11 | | -- | -- | -- | -- | -- | | Python 2.7 | :heavy_check_mark: | ... | :heavy_check_mark: | :heavy_check_mark: | | Python 3.4 | :heavy_check_mark: | ... | :heavy_check_mark: | :heavy_check_mark: | | Python 3.5 | | ... | :heavy_check_mark: | :heavy_check_mark: | | Python 3.6 | | ... | :heavy_check_mark: | :heavy_check_mark: |

This will nicely render into something like this on GitHub:

So far, we have a working sandbox and a matrix of Python/Django versions to support. Now question is: how do we actually support all of them? I am glad you asked, keep reading!

3. Test your application compatibility with CI

In order to explore the compatibility matrix of your application, the easiest way to proceed is to take advantage of Continuous Integration (CI) platforms like Travis CI, CircleCI or GitLab CI. Once this automation step has been defined, they will do the hard work for you

In our case, Django TailorDev Biblio is an open source application hosted on GitHub and connected to Travis CI. Below is the content of the .travis.yml configuration file:

language : python python : - 2.7 - 3.4 - 3.5 - 3.6 env : - DJANGO_RELEASE='Django>=1.7,<1.8' - DJANGO_RELEASE='Django>=1.8,<1.9' - DJANGO_RELEASE='Django>=1.9,<1.10' - DJANGO_RELEASE='Django>=1.10,<1.11' - DJANGO_RELEASE='Django>=1.11,<1.12' matrix : exclude : # These older Django versions don't support Python 3.5+ - python : 3.5 env : DJANGO_RELEASE='Django>=1.7,<1.8' - python : 3.6 env : DJANGO_RELEASE='Django>=1.7,<1.8' install : - pip install "$DJANGO_RELEASE" - pip install -r requirements/dev.txt script : - flake8 td_biblio/ - PYTHONPATH=$(pwd) DATABASE_URL="sqlite:///:memory:" py.test after_success : coveralls

We start by defining different python versions and environment variables allowing to define different Django releases ( env ). Travis understands that he has to build a matrix combining these conditions to run the tests. We can even define a matrix section to exclude some combinations (e.g. python 3.5 and Django 1.7).

In the install section, the target Django release is first installed. Then, we pull the requirements needed to run our tests. Note: as long as the installed Django release satisfies your requirements file condition (e.g. Django>=1.7 ), pip will not try to re-install it.

The script section defines the commands required to run the test suite ( flake8 and py.test in our case). When all the lights are green (see the after_success section), we send the coverage report to coveralls.io, a third-party service analyzing your application code coverage.

Nota Bene: we did not mention tox , but it’s a robust alternative to test your application with different environments (i.e. Python/Django couples). To see an example of tox integration with Travis, check out our Watson project.

I have not answered my previous question yet. How do we fix all the things?

4. Iterate to fix compatibility issues

The three previous steps allow you to iterate over all the problems that will likely happen given your compatibility matrix. It is a sort of TDD approach. Your job is now to fix these problems one by one until everything goes green. You have to be patient…

After N iterations, your Travis build should look like this one:

Finally, the next section is about what we have actually learnt applying these 4 rules to the TailorDev Biblio application.

Common compatibility issues

Python 2/3 compatibility

Supporting Python 2 and 3 for your application will be the most time-consuming part. You will find many good resources on the web, on how to write Python 2-3 compatible code, so we won’t copy/paste code samples here except for the three problems we have encountered.

Unicode my

You should always write your strings without the u prefix and add the following import statement when needed to make Python 2.7 behave like Python 3 (every string will be a unicode literal):

from __future__ import unicode_literals # unicode FTW reference = 'Franklin et al., 2017' type ( reference ) # -> <type 'unicode'> # \o/

Python 3 is super

Following the DRY principle, you may find the Python 3 super() function more convenient than in Python 2 but, unfortunately, you won’t be allowed to use it. You must use the retro-compatible Python 2 syntax instead:

class AbstractHuman ( models . Model ): """I am an Abstract Human""" def save ( self , * args , ** kwargs ): # Works in Python 2 and 3 super ( AbstractHuman , self ) . save ( * args , ** kwargs ) class AbstractEntity ( models . Model ): """I am an Abstract Entity""" def save ( self , * args , ** kwargs ): # Works only in Python 3 😞 super () . save ( * args , ** kwargs )

Fix dependencies compatibility

If your application has third-party dependencies, they might not fully support Python 2/3 compatibility. Be prepared to contribute to those projects and submit Pull Requests to fix them.

Django versions compatibility

Ensuring Django compatibility of your application will require a certain Django “expertise”. If you are new into it, we invite you to read the different release notes and to explore the Django sources by tags. Then again, below are the problems we had to fix for the Biblio application.

Settings

MIDDLEWARE vs MIDDLEWARE_CLASSES

The MIDDLEWARE_CLASSES setting is deprecated since Django 1.10, you will have to use the MIDDLEWARE setting instead. Hence, to maintain the django<1.10 compatibility of your sandbox (to be able to run your tests), you will have to duplicate these parameters, which is not that problematic since they do not exactly have the same values:

MIDDLEWARE = ( 'django.middleware.security.SecurityMiddleware' , 'whitenoise.middleware.WhiteNoiseMiddleware' , 'django.contrib.sessions.middleware.SessionMiddleware' , 'django.middleware.common.CommonMiddleware' , 'django.middleware.csrf.CsrfViewMiddleware' , 'django.contrib.auth.middleware.AuthenticationMiddleware' , 'django.contrib.messages.middleware.MessageMiddleware' , 'django.middleware.clickjacking.XFrameOptionsMiddleware' , ) # Django < 1.10 compat MIDDLEWARE_CLASSES = ( 'django.contrib.sessions.middleware.SessionMiddleware' , 'django.middleware.common.CommonMiddleware' , 'django.middleware.csrf.CsrfViewMiddleware' , 'django.contrib.auth.middleware.AuthenticationMiddleware' , 'django.contrib.auth.middleware.SessionAuthenticationMiddleware' , 'django.contrib.messages.middleware.MessageMiddleware' , 'django.middleware.clickjacking.XFrameOptionsMiddleware' )

TEMPLATES vs TEMPLATE_DIRS

Just like the middleware management, Django 1.8 introduced support for multiple template engines. As a consequence, the TEMPLATES_DIRS setting has been deprecated in favor of the TEMPLATES setting. Then again, you will have to define both in the sandbox settings:

BASE_DIR = os . path . dirname ( os . path . abspath ( __file__ )) TEMPLATES = [ { 'BACKEND' : 'django.template.backends.django.DjangoTemplates' , 'DIRS' : [ os . path . join ( BASE_DIR , 'templates' ), ], 'APP_DIRS' : True , 'OPTIONS' : { 'context_processors' : [ 'django.template.context_processors.debug' , 'django.template.context_processors.request' , 'django.contrib.auth.context_processors.auth' , 'django.contrib.messages.context_processors.messages' , ], }, }, ] # Django < 1.8 compat TEMPLATE_DIRS = [ os . path . join ( BASE_DIR , 'templates' ), ]

Migrations

Django 1.7 introduced built-in support for schema migrations. In our case, we introduced a BC break by dropping South and legacy migrations in favor of Django migrations.

Adapt/backport Django code to maintain old releases compatibility

When you develop new features for your application, you usually use recent Django mixins or utils without even noticing it. In this case, maintaining compatibility with old releases may not be trivial. In our case, we used the UserPassesTestMixin to restrict one view to super users, but it was only introduced in Django 1.9. Instead of dropping compatibility with Django 1.7 and 1.8, we wrote a custom mixin adapted to our needs, problem solved!

Conclusion

We had to fix this Django application for one of our clients. Instead of just hacking on the code to make it work, we took the time to ensure compatibility with many different Python and Django versions, without billing the client for that. It is some extra work we were pleased to do as open source contributors.

Managing scientific bibliography is quite common in Academia and every research team, group or even individual needs a website nowadays. If you like Python and want to write your own website with Django, you may want to consider Django TailorDev Biblio (demo here).

What about you? Have you maintained a Django application for multiple Django/Python releases? Is there something we are not aware? Please, let us know!