Today I thought I’d share my experience with various test tools for the Python programming language.
I was maintaining a Python tool that did a lot of
subprocess calling, reading
and writing files. This kind of code is not easy to test. My solution to this
problem was to create a lot of temporary directories, so that I could exercise
the code under test safely, without interfering with the rest of my filesystem.
Also, temporary directories are created empty, which is a good way to isolate
tests from each other.
(No files read or created by
/tmp/test_one can interfere with
test_two running in
I naturally started with
unittest, which is in the standard library. (Note:
this was a long time ago,
unittest did many progress since then)
Basic test with a temporary directory
This is what the code looked like:
# in test_one.py import unnitest class TestOne(unittest.TestCase): def setUp(self): self.tmpdir = tmpfile.mkdtemp("test-one-") def test_one(self): # do stuff in self.tmpdir rc = ... self.assertEquals(rc, 0) def tearDown(self): shutil.rmtree(self.tmpdir) # in run_tests.py import unittest TESTCASES = [ TestOne, # .... ] suite = unittest.TestSuite() for test_case in TEST_CASES: suite.addTests(unittest.makeSuite(test_case)) runner = unittest.TextTestRunner() result = runner.run(suite) if not result.wasSuccessful(): sys.exit(1)
Also, I had tests that shared a common set up and tear down. I found two solutions, but neither was really satisfying:
- Write some helper methods like
- Or write a class containing
setup_fooand subclass it in the other tests
So in the end there was a lot of code duplication among tests…
Problems with unittest
- A lot of boilerplate.
- The API is taken from
JUnit, a framework written for the Java programming language
- The setup and the tear down of the test are in two different places, so it’s
easy to forget to cleanup the temp directory in the
- The code is not PEP8
- There’s no way to skip tests (This was fixed in Python 3.1)
- There’s now way to discover tests (fixed in Python 3.2)
The last two points illustrate a fundamental problem with
it’s part of the standard library, you are stuck with the version coming with
your Python installation, and you cannot get new features without upgrading
Yes, I know
unittest2 exists, but if you’re going to use an external package
to run the tests, why stick with
Switching to pytest
Here’s what the same code looks like when using
def test_one(tmpdir): # do stuff in tmpdir rc = .... assert rc == 0
Well, that’s nicer isn’t it?
- No boiler plate: tests functions are automatically discovered.
- No special methods for asserting:
assertContainsand the like are all replaced by a simple
assert. But then
pytestdoes some black magic and you still get nice error messages:
file test_foo.py, line 1 def test_foo(): actual = "foo" + "bar" expected = "fooBar" > assert actual == expected E assert 'foobar' == 'fooBar' E - foobar E ? ^ E + fooBar E ? ^ test_foo.py:4: AssertionError
tmpdiris already a predifined fixture. The whole list is here
The nice thing about
pytestis that the code of the “fixtures” (the setup /
tear down) is completely separated from the code that exercise the production
pytest encourages you to write them in a special file called
Sharing fixture is then as easy as writing a function, decorate it with
@pytest.fixture and then pass it as parameter to whatever function needs it.
Here’s an example:
# in conftest.py import pytest @pytest.fixture def db(request): connection = DataBaseConnection("...") @request.addfinalizer def disconnect(): connection.close() return connection # in test_one def test_one(db): # ... # in test_two def test_two(db): # ...
Note how the code that deals with closing the connection to the database is right next to the code that opens it.
Also note how the tests do not care where the database come from: they just use it as a parameter.
Finally, by default fixtures have a scope of “function” (meaning the database will be opened and then closed for each test function), but you can chose to have a “module scope” or even a “session scope”.
(This is quite hard to do with
You can even have fixtures that are always implicitly called, by using
autouse=True in the fixture definition.
You can also extend the command line API: this is especially useful if you need some kind of token to run your tests.
Here’s an example:
# in conftest.py import pytest def pytest_addoption(parser): parser.addoption("--token", action="store", help="secret token") # in test_foo.py def test_foo(request): token = request.config.getoption("--token")
Again, this is quite hard to do with
Last but not least, there are a lot of plugins available to use with
Here are a few:
- pytest-sugar prettier output, show failures instantly
- pytest-cache allow to run only
the tests that failed in the previous run with
--lf(note: included in
pytest coresince 2.8)
- pytest-xdist run tests in parallel, or even distribute them over the network
- pytest-cov measure code coverage
You can even use
pytest with tests written in C++ using
boost::test thanks to the pytest-cpp plugin