Build and test a CI environment for multiple versions of Python

My name is @giginet and I am in charge of the 25th day of Python Advent Calendar. It's the last day, so I managed to write it so as not to be late.

The other day, I developed a simple plugin for Django. So, when I built a CI environment to absorb the difference between Python 2 and 3 and test the implemented plug-in in multiple environments, there were many addictions, so I will summarize it at this opportunity.

This article will show you how to use Travis CI to run tests across multiple Python versions to measure coverage.

Implement the module

First, implement the simple module you want to test. This time I put calculator.py in the calculator module and implemented the Calculator class.

This time, it is assumed that all the code will be implemented in Python3.

calculator/calculator.py

class Calculator(object):
    @staticmethod
    def power(number):
        return number * number

Implement test case

Next, implement the test case. Here we are simply testing if Calculator # power works.

tests/test_calculator.py

from unittest import TestCase
from calculator.calculator import Calculator


class CalculatorTestCase(TestCase):
    def setUp(self):
        self.calculator = Calculator()

    def test_power(self):
        self.assertEqual(self.calculator.power(12), 144)

Implement Test Runner

Now it's time to explore the test cases you've written and write a script to run the tests.

If you have Python 2.7 or above, you can easily search for test cases and execute tests as follows.

$ python -m unittest discover

If you're also testing Python 2.6 and below, the standard ʻunittest module doesn't include adiscover to search for test cases, so you'll need to install a package called ʻunittest2 instead.

requirements-test.txt

unittest2

$ pip install -r requirements-test.txt

Using ʻunittest2`, I implemented a script that searches for and executes test cases as follows.

runtest.py

#!/usr/bin/env python

import unittest2 as unittest

if __name__ == "__main__":
    suite = unittest.TestLoader().discover('tests', pattern = "test_*.py")
    unittest.TextTestRunner().run(suite)

When you do this, the test will run as shown below.

$ chmod +x runtest.py
$ ./runtest.py
.
----------------------------------------------------------------------
Ran 1 tests in 0.001s

OK

Test with multiple versions of Python using tox

Next, let's run this test case in multiple environments.

** tox ** is a well-known easy way to test a source with multiple versions of Python or Django.

Welcome to the tox automation project — tox 2.3.1 documentation

tox is a tool that allows you to build a test environment by changing various conditions and execute tests in multiple environments at once.

You can build a test environment by defining a configuration file called tox.ini, so this time we will define it as follows.

tox.ini

[tox]
skipsdist = True
envlist =
    py26,
    py27,
    py33,
    py34,
    py35

[testenv]
deps=
    -rrequirements-test.txt
commands=
  ./runtest.py

Please refer to the documentation for details.

tox configuration specification — tox 2.3.1 documentation

Finally, running tox allows you to run all test environments at the same time.

This time I will run the test only with Python 3.5 to check the operation.

$ tox -r -e 'py35'

You have now built a test environment that supports multiple Python versions.

Run tox on Travis CI with tox-travis

Now let's run this on Travis CI.

This time we will use a module called tox-travis.

ryanhiebert/tox-travis

With this, you can omit the settings for running tox on Travis CI.

For example, you can run the test in multiple environments just by preparing .travis.yml as shown below.

yaml:.travis.yml

sudo: false
language: python
python:
  - 2.6
  - 2.7
  - 3.3
  - 3.4
  - 3.5

install:
  - pip install tox tox-travis
  - pip install coverage coveralls

script:
  - tox -r

When you do this on Travis CI, multiple containers will be launched for easy testing, as shown below. It's convenient.

Keep Python 2 and 3 compatible with the __future__ module

In order to run the code written for Python3 on Python2, you have to use the __future__ module etc. and write it in a backward compatible manner.

Please refer to the documentation for more details, as this article does not go into detail.

Easy, clean, reliable Python 2/3 compatibility — Python-Future documentation

Mock test with Python2

If you want to test with mock or stub, Python 3 includes a mock module in the standard library.

26.5. unittest.mock — mock object library — Python 3.5.1 documentation

On the other hand, mock cannot be used as standard in the environment before Python2.

There is a package that backports the Python3 mock module before 2, so let's introduce it.

requirements-test.txt

mock

First try to import ʻunittest.mock, and if it is not included in the standard library, import it from the mock` library.

from unittest import TestCase

try:
    from unittest.mock import MagicMock
except ImportError:
    from mock import MagicMock


class CalcTestCase(TestCase):
    def test_mock(self):
        mock_object = MagicMock()
        mock_object.__str__.return_value = 'hello'
        self.assertEqual(str(mock_object), 'hello')

This way, you can do mock tests on Python 2 as well as 3.

For details on how to use mock, refer to the following.

Python --Mock Mock ~ unittest.mock, mock.mock revisit ~ --Qiita

Measure coverage

Once the tests pass in all environments, finally measure the coverage and manage it with coverails.

First, measure the coverage by executing coverage in tox.ini as follows.

tox.ini

deps=
    -rrequirements-test.txt
    coverage
commands=
  {envbindir}/coverage run --append --source=calculator runtest.py

At this time, if you define a file called .coveragerc, you can set the rules for coverage measurement. The following defines files and lines for which coverage measurement is not performed.

See below for details.

Configuration files — Coverage.py 4.0.3 documentation

.coveragerc

[report]
include =
    calculator/*.py
omit =
    calculator/__init__.py
exclude_lines =
    pragma: no cover
    if __name__ == .__main__.:

Finally, add the following to .travis.yml so that the log will be sent to coversalls after the test is completed.

yaml:.travis.yml

install:
  - pip install tox tox-travis
  - pip install coverage coveralls

after_success:
  - coverage report
  - coveralls

You can now record coverage in Coveralls.

Summary

In this article, I've briefly summarized how to test your Python library with multiple versions.

The Test Suite introduced this time is briefly summarized in the following repository.

https://github.com/giginet/python-ci-suite

You can also use tox well to test a library with multiple versions.

The following Django plugins are testing with multiple versions of Python and a combination of Django. If you are interested, please refer to it.

giginet/django-debug-toolbar-vcs-info