Testing a Flask Application using pytest

Introduction

I recently started using pytest and it is an incredible test framework for python! After reading Brian Okken’s book titled “Python Testing with pytest“, I was convinced that I wanted to start using pytest instead of the built-in unittest module that comes with python. In this blog post, I’ll explain how to test a Flask application using pytest.

pytest is a test framework for python that you use to write test cases, but also to run the test cases. After setting up your test structure, pytest makes it really easy to write tests and provides so much flexibility for running the tests. Using pytest, I’ve found that is satisfies the key aspects of a good test environment:

tests are fun to write

tests can be written quickly by using helper functions (fixtures)

tests can be executed with a single command

tests run quickly

For reference, the Flask application that is referenced in this blog post can be found on GitLab: https://gitlab.com/patkennedy79/flask_user_management_example

Structure

Within a Flask project, I like to keep all the test cases in a separate ‘tests’ folder that is at the same level as the application files:

$ tree -L 3 . ├── main.py ├── project ├── requirements.txt ├── tests │ ├── conftest.py │ ├── functional │ │ ├── __init__.py │ │ └── test_users.py │ ├── pytest.ini │ └── unit │ ├── __init__.py │ └── test_models.py └── venv

Additionally, I really like differentiating between unit tests and functional (integration) tests by splitting them out as separate sub-folders within the ‘tests’ folder. Full disclosure: This is the structure that Brian Okken uses in his book (Python Testing with pytest) and I really like this structure as it allows you the flexibility to just run unit tests or just run functional tests.

Writing Unit Tests

In the Flask User Management project, there is logic in the models.py file that should be tested to provide confidence that it works properly.

When running pytest, it is able to automatically run any files that start with test*.py or end with *test.py. Therefore, the file to test out the logic in models.py should be called test_models.py and it should be created in …/tests/unit_tests/ directory. Here’s the first test case to write:

from project.models import User def test_new_user(): """ GIVEN a User model WHEN a new User is created THEN check the email, hashed_password, authenticated, and role fields are defined correctly """ user = User('patkennedy79@gmail.com', 'FlaskIsAwesome') assert new_user.email == 'patkennedy79@gmail.com' assert new_user.hashed_password != 'FlaskIsAwesome' assert not new_user.authenticated assert new_user.role == 'user'

This test creates a new instance of the User class and checks that the user is initialized correctly. Once again, I’m stealing directly from Brian Okken’s book (Python Testing with pytest) with the comment for each unit test (GIVEN, WHEN, THEN sequence). I really like this comment structure for identifying what the test case is doing. This might seem like a lot of extra work, but I’ve really found that well-commented test cases make their maintenance so much easier.

To run this test case, navigate to the top-level directory of your project and run:

$ pytest

That’s it! pytest will discover all the test cases for you and run these tests. Super easy!

Fixtures

As someone that has worked with the xUnit type of test frameworks, I’m familiar with the idea of:

Setup()

…run the test case…

Teardown()

This structure feels very familiar, but I have come to prefer the concept of fixtures in pytest. Fixtures allow you greater flexibility than Setup()/Teadown() as you can have your fixture execute with different scope:

function

class

module

session

For this situation where a single class (User) is being testing, a good option for a fixture would be to create a new User. This new User can then be created just once and used multiple times in different test cases.

All fixtures are added to the …/tests/conftest.py:

import pytest from project.models import User @pytest.fixture(scope='module') def new_user(): user = User('patkennedy79@gmail.com', 'FlaskIsAwesome') return user

This fixture creates an instance of the User class and returns it for test cases within the module scope to use. To use this fixture, change the original test case to:

def test_new_user(new_user): """ GIVEN a User model WHEN a new User is created THEN check the email, hashed_password, authenticated, and role fields are defined correctly """ assert new_user.email == 'patkennedy79@gmail.com' assert new_user.hashed_password != 'FlaskIsAwesome' assert not new_user.authenticated assert new_user.role == 'user'

The fixture is run prior to this test case, because it is specified as an argument (new_user). To illustrate what this fixture is doing, let’s let pytest tell us how it is using the fixture by using the ‘–setup-show’ flag (I’ve added two additional test cases in the test_models.py):

$ pytest --setup-show tests/unit/ =================================================== test session starts =================================================== platform darwin -- Python 3.6.3, pytest-3.4.1, py-1.5.2, pluggy-0.6.0 rootdir: .../flask_user_management_example/tests, inifile: pytest.ini collected 3 items tests/unit/test_models.py SETUP M new_user unit/test_models.py::test_new_user (fixtures used: new_user). unit/test_models.py::test_setting_password (fixtures used: new_user). unit/test_models.py::test_user_id (fixtures used: new_user). TEARDOWN M new_user ================================================ 3 passed in 2.02 seconds =================================================

This output is great, because it shows that the fixture (new_user) runs before any of the test cases in unit/test_models.py. This fixture is then used by each test case in unit/test_models.py. Finally, the fixture is torn down. I love this flexibility to do the standard setup/teardown at different scopes!

Writing Functional Tests

Writing functional tests are a bit more complicated, as they require more setup steps. Instead of diving into a new test case, I want to start by writing the fixtures that will help with setting up the functional tests.

The functional tests that I’m going to create are testing how a user can register, log in, and log out. In order to be able to test this functionality, we need a fully functioning Flask application running with a database. This is where fixtures really shine, in my opinion…

Here is the sequence of how the functional tests should run:

Create a new Flask application

Initialize a database

…run the functional tests…

Destroy the database

Stop the Flask application

I created two fixtures in …/tests/conftest.py to implement this sequence…

(1) Fixture for Creating the Flask Application

One of the key aspects of this sequence is having an application factory that you can use to create your Flask application (see my blog post on Structuring a Flask Application). The first fixture creates the Flask application:

@pytest.fixture(scope='module') def test_client(): flask_app = create_app('flask_test.cfg') # Flask provides a way to test your application by exposing the Werkzeug test Client # and handling the context locals for you. testing_client = flask_app.test_client() # Establish an application context before running the tests. ctx = flask_app.app_context() ctx.push() yield testing_client # this is where the testing happens! ctx.pop()

This fixture starts by creating a new Flask application via the create_app() function and a custom configuration file (flask_test.cfg). Next, a testing client is created which will be used in the functional tests for responding to GETs and POSTs. In order for the flask application to be able to respond to GETs and POSTs, the application context needs to be pushed to be able to handle the GETs and POSTs.

At this point, the ‘yield testing_client’ is called to allow all the functional tests to run with the testing client that was created in this fixture. Finally, the application context is popped to clean up the test environment.

Test Configuration

One key point to be aware of is that the parameters specified in the flask_test.cfg file are really important, especially the following:

# Bcrypt algorithm hashing rounds (reduced for testing purposes only!) BCRYPT_LOG_ROUNDS = 4 # Enable the TESTING flag to disable the error catching during request handling # so that you get better error reports when performing test requests against the application. TESTING = True # Disable CSRF tokens in the Forms (only valid for testing purposes!) WTF_CSRF_ENABLED = False

The first parameter (BCRYPT_LOG_ROUNDS) specifies the number of hashing rounds to run when doing the password hashing of the users. This number can range from 4-15, but using the minimum value for testing purposes greatly reduces the execution time of the tests.

The next parameter (TESTING) is recommended for testing a Flask application.

The last parameter (WTF_CSRF_ENABLED) is disabled, which should only ever be done during testing. This parameter must be disabled for the tests to run, but CSRF protection is absolutely critical when running the actual application.

(2) Fixture for Initializing the Database

The second fixture creates and initializes the database:

@pytest.fixture(scope='module') def init_database(): # Create the database and the database table db.create_all() # Insert user data user1 = User(email='patkennedy79@gmail.com', plaintext_password='FlaskIsAwesome') user2 = User(email='kennedyfamilyrecipes@gmail.com', plaintext_password='PaSsWoRd') db.session.add(user1) db.session.add(user2) # Commit the changes for the users db.session.commit() yield db # this is where the testing happens! db.drop_all()

This fixture creates the database (db.create_all()) and then adds two users to the database to use during functional testing. Once again, there is a ‘yield db’ statement to allow the function tests to run while using this database instance. After all the functional tests run, the database is destroyed (db.drop_all()).

Now that these fixtures are in place, it’s time to write a functional test (in …/tests/functional/test_users):

def test_home_page(test_client): """ GIVEN a Flask application WHEN the '/' page is requested (GET) THEN check the response is valid """ response = test_client.get('/') assert response.status_code == 200 assert b"Welcome to the Flask User Management Example!" in response.data assert b"Need an account?" in response.data assert b"Existing user?" in response.data

This functional test is requesting the home page (at the base URL of ‘/’) and checking that the status code returned is valid (200) and the html returned contains the key statements of a the home page.

This functional test uses the test client for the GET request, so the ‘test_client’ fixture is specified as an argument. Since this functional test isn’t using the database, the ‘init_database’ fixture is not specified.

To run this individual test case with the fixture execution explained, run:

$ pytest --setup-show tests/functional/test_users.py::test_home_page =================================================== test session starts =================================================== platform darwin -- Python 3.6.3, pytest-3.4.1, py-1.5.2, pluggy-0.6.0 rootdir: .../flask_user_management_example/tests, inifile: pytest.ini collected 1 item tests/functional/test_users.py SETUP S test_client functional/test_users.py::test_home_page (fixtures used: test_client). TEARDOWN S test_client ================================================ 1 passed in 0.13 seconds =================================================

Here is a more complex test case that checks logging in and then logging out:

def test_valid_login_logout(test_client, init_database): """ GIVEN a Flask application WHEN the '/login' page is posted to (POST) THEN check the response is valid """ response = test_client.post('/login', data=dict(email='patkennedy79@gmail.com', password='FlaskIsAwesome'), follow_redirects=True) assert response.status_code == 200 assert b"Thanks for logging in, patkennedy79@gmail.com!" in response.data assert b"Welcome patkennedy79@gmail.com!" in response.data assert b"Flask User Management" in response.data assert b"Logout" in response.data assert b"Login" not in response.data assert b"Register" not in response.data """ GIVEN a Flask application WHEN the '/logout' page is requested (GET) THEN check the response is valid """ response = test_client.get('/logout', follow_redirects=True) assert response.status_code == 200 assert b"Goodbye!" in response.data assert b"Flask User Management" in response.data assert b"Logout" not in response.data assert b"Login" in response.data assert b"Register" in response.data

Since this functional test is logging in a user, it requires using the database to access if the specified user is a registered user. Therefore, both the ‘test_client’ and ‘init_database’ fixtures are specified.

To illustrate the use of these fixtures, here is the command to run all of the functional tests with the use of the fixtures explained:

$ pytest --setup-show tests/functional/ =================================================== test session starts =================================================== platform darwin -- Python 3.6.3, pytest-3.4.1, py-1.5.2, pluggy-0.6.0 rootdir: .../flask_user_management_example/tests, inifile: pytest.ini collected 7 items tests/functional/test_users.py SETUP S test_client functional/test_users.py::test_home_page (fixtures used: test_client). functional/test_users.py::test_home_page_post (fixtures used: test_client). functional/test_users.py::test_login_page (fixtures used: test_client). SETUP S init_database functional/test_users.py::test_valid_login_logout (fixtures used: init_database, test_client). functional/test_users.py::test_invalid_login (fixtures used: init_database, test_client). functional/test_users.py::test_valid_registration (fixtures used: init_database, test_client). functional/test_users.py::test_invalid_registration (fixtures used: init_database, test_client). TEARDOWN S init_database TEARDOWN S test_client ================================================ 7 passed in 0.36 seconds =================================================

This output is really impressive, as it shows that pytest starts by running the ‘test_client’ fixture to create the Flask application. The three test cases that just rely on the ‘test_client’ fixture are then run. At this point, the remaining tests also require the ‘init_database’ fixture, so pytest runs this fixture and then the remaining test cases. At the end, both fixtures are completed.

This usage of fixtures allows for the setup of the Flask application and database to just be done once at the ‘session’ scope and then have each functional test utilize this configuration.

To wrap up the use of pytest, here is my favorite command for running the unit and functional tests:

$ pytest -v

Conclusion

Yes, pytest is amazing! This blog post shows how to use pytest for testing a Flask application, but it just skims the surface of the full power of pytest. To continue learning about pytest, I highly recommend Brian Okken’s book: Python Testing with pytest.