Fedora Modularity is a project within Fedora with the goal of Building a modular operating system with multiple versions of components on different lifecycles. Fedora 26 features the first look of the modularity vision: the Fedora 26 Boltron Server. However, if you are jumping into the modularity world, creating and deploying your own modules and containers — your next question may be how to test these artifacts. The Modularity Testing Framework (MTF) has been designed for testing artifacts such as modules, RPM base repos, containers, and other artifact types. It helps you to write tests easily, and the tests are also independent of the type of the module.

MTF is a minimalistic library built on the existing avocado and behave testing frameworks. enabling developers to enable test automation for various module aspects and requirements quickly. MTF adds basic support and abstraction for testing various module artifact types: RPM based, docker images, ISOs, and more. For detailed information about the framework, and how to use it check out the MTF Documentation.

Installing MTF

The Modularity Testing Framework is available in the official Fedora repositories. Install MTF using the command:

dnf install -y modularity-testing-framework

A COPR is available if you want to use the untested, unstable version. Install via COPR with the commands:

dnf copr enable phracek/Modularity-testing-framework dnf install -y modularity-testing-framework

Writing a simple test

Creating a testing directory structure

First, create the tests/ directory in the root directory of the module. In the tests/ directory, create a Makefile file:

MODULE_LINT=/usr/share/moduleframework/tools/modulelint/*.py TESTS=*.py (try not to use “*.py”, but use the test files with names such as sanity1.py sanity2.py... separated by spaces) CMD=python -m avocado run $(MODULE_LINT) $(TESTS) # all: generator # use it in case that tests are defined also in config.yaml file (described below) $(CMD)

In the root directory of the module, create a Makefile file containing a section test. For example:

.PHONY: build run default IMAGE_NAME = memcached MODULEMDURL=file://memcached.yaml default: run build: docker build --tag=$(IMAGE_NAME) . run: build docker run -d $(IMAGE_NAME) test: build # used for testing docker image available on Docker Hub. Dockerfile cd tests; MODULE=docker MODULEMD=$(MODULEMDURL) URL="docker.io/modularitycontainers/memcached" make all # used for testing docker image available on locally. # Dockerfile and relavant files has to be stored in root directory of the module. cd tests; MODULE=docker MODULEMD=$(MODULEMDURL) URL="docker=$(IMAGE_NAME)" make all # This tests "modules" on local system. cd tests; MODULE=rpm MODULEMD=$(MODULEMDURL) URL="https://kojipkgs.fedoraproject.org/compose/latest-Fedora-Modular-26/compose/Server/x86_64/os/" make all

In the tests/ directory, place the config.yaml configuration file for module testing(Do adresare tests umisti configuracni soubor config.yaml pro testovani modulu) . See minimal-config.yaml. For example:

document: modularity-testing version: 1 name: memcached modulemd-url: http://raw.githubusercontent.com/container-images/memcached/master/memcached.yaml service: port: 11211 packages: rpms: - memcached - perl-Carp testdependecies: rpms: - nc module: docker: start: "docker run -it -e CACHE_SIZE=128 -p 11211:11211" labels: description: "memcached is a high-performance, distributed memory" io.k8s.description: "memcached is a high-performance, distributed memory" source: https://github.com/container-images/memcached.git container: docker.io/modularitycontainers/memcached rpm: start: /usr/bin/memcached -p 11211 & repo: - https://kojipkgs.fedoraproject.org/compose/latest-Fedora-Modular-26/compose/Server/x86_64/os/ test: processrunning: - 'ls /proc/*/exe -alh | grep memcached' testhost: selfcheck: - 'echo errr | nc localhost 11211' - 'echo set AAA 0 4 2 | nc localhost 11211' - 'echo get AAA | nc localhost 11211' selcheckError: - 'echo errr | nc localhost 11211 |grep ERROR'

Add the simpleTest.py python file, which tests a service or an application, into the tests/ directory:

#!/usr/bin/python # -*- coding: utf-8 -*- # # This Modularity Testing Framework helps you to write tests for modules # Copyright (C) 2017 Red Hat, Inc. import socket from avocado import main from avocado.core import exceptions from moduleframework import module_framework class SanityCheck1(module_framework.AvocadoTest): """ :avocado: enable """ def testSettingTestVariable(self): self.start() s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.connect(('localhost', self.getConfig()['service']['port'])) s.sendall('set Test 0 100 4\r



') #data = s.recv(1024) # print data s.sendall('get Test\r

') #data = s.recv(1024) # print data s.close() def testBinExistsInRootDir(self): self.start() self.run("ls / | grep bin") def test3GccSkipped(self): module_framework.skipTestIf("gcc" not in self.getActualProfile()) self.start() self.run("gcc -v") if __name__ == '__main__': main()

Running tests

To execute tests from the root directory of the module, type

# run tests from a module root directory $ sudo make test

The result looks like:

docker build --tag=memcached . Sending build context to Docker daemon 268.3 kB Step 1 : FROM baseruntime/baseruntime:latest ---> 0cbcd55844e4 Step 2 : ENV NAME memcached ARCH x86_64 ---> Using cache ---> 16edc6a5f7b6 Step 3 : LABEL MAINTAINER "Petr Hracek" <phracek@redhat.com> ---> Using cache ---> 693d322beab2 Step 4 : LABEL summary "High Performance, Distributed Memory Object Cache" name "$FGC/$NAME" version "0" release "1.$DISTTAG" architecture "$ARCH" com.redhat.component $NAME usage "docker run -p 11211:11211 f26/memcached" help "Runs memcached, which listens on port 11211. No dependencies. See Help File below for more details." description "memcached is a high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load." io.k8s.description "memcached is a high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load." io.k8s.diplay-name "Memcached 1.4 " io.openshift.expose-services "11211:memcached" io.openshift.tags "memcached" ---> Using cache ---> eea936c1ae23 Step 5 : COPY repos/* /etc/yum.repos.d/ ---> Using cache ---> 920155da88d9 Step 6 : RUN microdnf --nodocs --enablerepo memcached install memcached && microdnf -y clean all ---> Using cache ---> c83e613f0806 Step 7 : ADD files /files ---> Using cache ---> 7ec5f42c0064 Step 8 : ADD help.md README.md / ---> Using cache ---> 34702988730f Step 9 : EXPOSE 11211 ---> Using cache ---> 577ef9f0d784 Step 10 : USER 1000 ---> Using cache ---> 671ac91ec4e5 Step 11 : CMD /files/memcached.sh ---> Using cache ---> 9c933477acc1 Successfully built 9c933477acc1 cd tests; MODULE=docker MODULEMD=file://memcached.yaml URL="docker=memcached" make all make[1]: Entering directory '/home/phracek/work/FedoraModules/memcached/tests' Added test (runmethod: run): processrunning Added test (runmethod: runHost): selfcheck Added test (runmethod: runHost): selcheckError python -m avocado run --filter-by-tags=-WIP /usr/share/moduleframework/tools/modulelint.py *.py JOB ID : 9ba3a3f9fd982ea087f4d4de6708b88cee15cbab JOB LOG : /root/avocado/job-results/job-2017-06-14T16.25-9ba3a3f/job.log (01/20) /usr/share/moduleframework/tools/modulelint.py:DockerfileLinter.testDockerFromBaseruntime: PASS (1.52 s) (02/20) /usr/share/moduleframework/tools/modulelint.py:DockerfileLinter.testDockerRunMicrodnf: PASS (1.53 s) (03/20) /usr/share/moduleframework/tools/modulelint.py:DockerfileLinter.testArchitectureInEnvAndLabelExists: PASS (1.63 s) (04/20) /usr/share/moduleframework/tools/modulelint.py:DockerfileLinter.testNameInEnvAndLabelExists: PASS (1.61 s) (05/20) /usr/share/moduleframework/tools/modulelint.py:DockerfileLinter.testReleaseLabelExists: PASS (1.60 s) (06/20) /usr/share/moduleframework/tools/modulelint.py:DockerfileLinter.testVersionLabelExists: PASS (1.45 s) (07/20) /usr/share/moduleframework/tools/modulelint.py:DockerfileLinter.testComRedHatComponentLabelExists: PASS (1.64 s) (08/20) /usr/share/moduleframework/tools/modulelint.py:DockerfileLinter.testIok8sDescriptionExists: PASS (1.51 s) (09/20) /usr/share/moduleframework/tools/modulelint.py:DockerfileLinter.testIoOpenshiftExposeServicesExists: PASS (1.50 s) (10/20) /usr/share/moduleframework/tools/modulelint.py:DockerfileLinter.testIoOpenShiftTagsExists: PASS (1.53 s) (11/20) /usr/share/moduleframework/tools/modulelint.py:DockerLint.testBasic: PASS (13.75 s) (12/20) /usr/share/moduleframework/tools/modulelint.py:DockerLint.testContainerIsRunning: PASS (14.19 s) (13/20) /usr/share/moduleframework/tools/modulelint.py:DockerLint.testLabels: PASS (1.57 s) (14/20) /usr/share/moduleframework/tools/modulelint.py:ModuleLintPackagesCheck.test: PASS (14.03 s) (15/20) generated.py:GeneratedTestsConfig.test_processrunning: PASS (13.77 s) (16/20) generated.py:GeneratedTestsConfig.test_selfcheck: PASS (13.85 s) (17/20) generated.py:GeneratedTestsConfig.test_selcheckError: PASS (14.32 s) (18/20) sanity1.py:SanityCheck1.testSettingTestVariable: PASS (13.86 s) (19/20) sanity1.py:SanityCheck1.testBinExistsInRootDir: PASS (13.81 s) (20/20) sanity1.py:SanityCheck1.test3GccSkipped: ERROR (13.84 s) RESULTS : PASS 19 | ERROR 1 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 144.85 s JOB HTML : /root/avocado/job-results/job-2017-06-14T16.25-9ba3a3f/html/results.html Makefile:6: recipe for target 'all' failed make[1]: *** [all] Error 1 Makefile:14: recipe for target 'test' failed make: *** [test] Error 2 $

To execute tests from the tests/ directory, type:

# run Python tests from the tests/ directory $ sudo MODULE=docker avocado run ./*.py

The result looks like:

$ sudo MODULE=docker avocado run ./*.py [sudo] password for phracek: JOB ID : 2a171b762d8ab2c610a89862a88c015588823d29 JOB LOG : /root/avocado/job-results/job-2017-06-14T16.43-2a171b7/job.log (1/6) ./generated.py:GeneratedTestsConfig.test_processrunning: PASS (24.79 s) (2/6) ./generated.py:GeneratedTestsConfig.test_selfcheck: PASS (18.18 s) (3/6) ./generated.py:GeneratedTestsConfig.test_selcheckError: ERROR (24.16 s) (4/6) ./sanity1.py:SanityCheck1.testSettingTestVariable: PASS (18.88 s) (5/6) ./sanity1.py:SanityCheck1.testBinExistsInRootDir: PASS (17.87 s) (6/6) ./sanity1.py:SanityCheck1.test3GccSkipped: ERROR (19.30 s) RESULTS : PASS 4 | ERROR 2 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 124.19 s JOB HTML : /root/avocado/job-results/job-2017-06-14T16.43-2a171b7/html/results.html