38

How can I make it so unit tests in Python (using unittest) are run in the order in which they are specified in the file?

Improve this question
4
  • 1
    It also precludes running the tests in parallel on multicores, or distributed across a cluster. Not a good idea. Commented Oct 24, 2010 at 3:50
  • 11
    It would matter if 1. You need to debug and want to run the tests in order of increasing complexity and 2. If a small percentage of the tests are computationally heavy, but there are errors that can be spotted with the simpler tests, then having them ordered will make your work much, much faster. Commented Oct 22, 2012 at 11:31
  • Sounds like integration tests, not unit tests. Commented Mar 1, 2013 at 20:04
  • 16
    What is the Python standard for integration testing then? Is it not...unittest? Commented Apr 21, 2014 at 18:10

8 Answers 8

Reset to default
29

You can change the default sorting behavior by setting a custom comparison function. In unittest.py you can find the class variable unittest.TestLoader.sortTestMethodsUsing which is set to the builtin function cmp by default.

For example you can revert the execution order of your tests with doing this:

import unittest
unittest.TestLoader.sortTestMethodsUsing = lambda _, x, y: cmp(y, x)
Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

This does not answer the question above
This does answer the question above
This does seem to do the trick. So is the default order always to run opposite the order of declaration?
The documentation indicates it depends on the default sorting of strings:--- Note that the order in which the various test cases will be run is determined by sorting the test function names with respect to the built-in ordering for strings.
26

Clever Naming.

class Test01_Run_Me_First( unittest.TestCase ):
    def test010_do_this( self ):
        assertTrue( True )
    def test020_do_that( self ):
        etc.

Is one way to force a specific order.

Improve this answer

2 Comments

Slightly more clever name is Test010_Run_Me_First so if you decided to push a new one between two, you could.
Indeed, clever :).
9

As said above, normally tests in test cases should be tested in any (i.e. random) order.

However, if you do want to order the tests in the test case, apparently it is not trivial. Tests (method names) are retrieved from test cases using dir(MyTest), which returns a sorted list of members. You can use a clever (?) hack to order methods by their line numbers. This will work for one test case:

if __name__ == "__main__":
    loader = unittest.TestLoader()
    ln = lambda f: getattr(MyTestCase, f).im_func.func_code.co_firstlineno
    lncmp = lambda a, b: cmp(ln(a), ln(b))
    loader.sortTestMethodsUsing = lncmp
    unittest.main(testLoader=loader, verbosity=2)
Improve this answer

2 Comments

Nice trick, but there's an error in your code. It should be lncomp = lambda _, a, b: cmp(ln(a), ln(b))
How can I use this on multiple test cases?
2

There are also test runners which do that by themselves – I think py.test does it.

Improve this answer

Comments

2

Use proboscis library as I mentioned already (please see short description there).

Improve this answer

1 Comment

Tried proboscis... it's worth mentioning that it hijacks the test loading process so if you're using nose, any nose plugins that take advantage of that (e.g. testid) will stop working
1

The default order is alphabetical. You can put test_<int> before every test to specify the order of execution.

Also you can set unittest.TestLoader.sortTestMethodsUsing = None to eliminate the sort.

Check out Unittest Documentation for more details.

Improve this answer

Comments

0

I found a solution for it using PyTest ordering plugin provided here.

Try py.test YourModuleName.py -vv in CLI and the test will run in the order they have appeared in your module.

I did the same thing and works fine for me.

Note: You need to install PyTest package and import it.

Improve this answer

1 Comment

The question states directly: "using unittest." There are plenty of valid reasons for not wanting to use a different test framework; granted, pytest does do a lot of things much nicer than the built-in one.
0

hacky way (run this file in pycharm or other unit test runner)

#!/usr/bin/env python
# -*- coding: UTF-8 -*-
import unittest


def make_suite():
    class Test(unittest.TestCase):
        def test_32(self):
            print "32"
        def test_23(self):
            print "23"

    suite = unittest.TestSuite()
    suite.addTest(Test('test_32'))
    suite.addTest(Test('test_23'))
    return suite

suite = make_suite()

class T(unittest.TestCase):
    counter = 0
    def __call__(self, *args, **kwargs):
        res = suite._tests[T.counter](*args, **kwargs)
        T.counter += 1
        return res

for t in suite._tests:
    name = "{}${}".format(t._testMethodName, t.__class__.__name__)
    setattr(T, name, t)
Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.