Today I thought I’d share my experience with various test tools for the Python programming language.

I was maintaining a Python tool that did a lot of subprocess calling, reading and writing files. This kind of code is not easy to test. My solution to this problem was to create a lot of temporary directories, so that I could exercise the code under test safely, without interfering with the rest of my filesystem. Also, temporary directories are created empty, which is a good way to isolate tests from each other. (No files read or created by test_one in /tmp/test_one can interfere with test_two running in /tmp/test_two)

Using unittest

I naturally started with unittest, which is in the standard library. (Note: this was a long time ago, unittest did many progress since then)

Basic test with a temporary directory

This is what the code looked like:

# in test_one.py

import unnitest

class TestOne(unittest.TestCase):
    def setUp(self):
        self.tmpdir = tmpfile.mkdtemp("test-one-")

    def test_one(self):
        # do stuff in self.tmpdir
        rc = ...
        self.assertEquals(rc, 0)

    def tearDown(self):
        shutil.rmtree(self.tmpdir)


# in run_tests.py

import unittest

TESTCASES = [
    TestOne,
    # ....
]

suite = unittest.TestSuite()
for test_case in TEST_CASES:
    suite.addTests(unittest.makeSuite(test_case))
runner = unittest.TextTestRunner()
result = runner.run(suite)
if not result.wasSuccessful():
    sys.exit(1)

Sharing fixtures

Also, I had tests that shared a common set up and tear down. I found two solutions, but neither was really satisfying:

  • Write some helper methods like setup_foo in a test_helper module
  • Or write a class containing setup_foo and subclass it in the other tests

So in the end there was a lot of code duplication among tests…

Problems with unittest

  • A lot of boilerplate.
  • The API is taken from JUnit, a framework written for the Java programming language
  • The setup and the tear down of the test are in two different places, so it’s easy to forget to cleanup the temp directory in the tearDown() method
  • The code is not PEP8
  • There’s no way to skip tests (This was fixed in Python 3.1)
  • There’s now way to discover tests (fixed in Python 3.2)

The last two points illustrate a fundamental problem with unittest: since it’s part of the standard library, you are stuck with the version coming with your Python installation, and you cannot get new features without upgrading Python too.

Yes, I know unittest2 exists, but if you’re going to use an external package to run the tests, why stick with unittest?

Switching to pytest

Basic test

Here’s what the same code looks like when using pytest

def test_one(tmpdir):
      # do stuff in tmpdir
      rc = ....
      assert rc == 0

Well, that’s nicer isn’t it?

  • No boiler plate: tests functions are automatically discovered.
  • No special methods for asserting: assertEquals, assertTrue, assertContains and the like are all replaced by a simple assert. But then pytest does some black magic and you still get nice error messages:
file test_foo.py, line 1
    def test_foo():
        actual = "foo" + "bar"
        expected = "fooBar"
>       assert actual == expected
E       assert 'foobar' == 'fooBar'
E         - foobar
E         ?    ^
E         + fooBar
E         ?    ^

test_foo.py:4: AssertionError
  • tmpdir is already a predifined fixture. The whole list is here

Sharing fixtures

The nice thing about pytestis that the code of the “fixtures” (the setup / tear down) is completely separated from the code that exercise the production code.

pytest encourages you to write them in a special file called conftest.py. Sharing fixture is then as easy as writing a function, decorate it with @pytest.fixture and then pass it as parameter to whatever function needs it.

Here’s an example:

# in conftest.py

import pytest

@pytest.fixture
def db(request):
    connection = DataBaseConnection("...")
    @request.addfinalizer
    def disconnect():
         connection.close()
    return connection

# in test_one

def test_one(db):
    # ...

# in test_two

def test_two(db):
    # ...

Note how the code that deals with closing the connection to the database is right next to the code that opens it.

Also note how the tests do not care where the database come from: they just use it as a parameter.

Finally, by default fixtures have a scope of “function” (meaning the database will be opened and then closed for each test function), but you can chose to have a “module scope” or even a “session scope”.

(This is quite hard to do with unittest)

You can even have fixtures that are always implicitly called, by using autouse=True in the fixture definition.

Customizing pytest

You can also extend the command line API: this is especially useful if you need some kind of token to run your tests.

Here’s an example:

# in conftest.py
import pytest

def pytest_addoption(parser):
    parser.addoption("--token", action="store", help="secret token")

# in test_foo.py

def test_foo(request):
     token = request.config.getoption("--token")

Again, this is quite hard to do with unittest

Awesome plugins

Last but not least, there are a lot of plugins available to use with pytest.

Here are a few:

  • pytest-sugar prettier output, show failures instantly
  • pytest-cache allow to run only the tests that failed in the previous run with --lf (note: included in pytest core since 2.8)
  • pytest-xdist run tests in parallel, or even distribute them over the network
  • pytest-cov measure code coverage

You can even use pytest with tests written in C++ using gtest or boost::test thanks to the pytest-cpp plugin