Generating py.test tests in Python

Question

Question first, then an explanation if you're interested.

In the context of py.test, how do I generate a large set of test functions from a small set of test-function templates?

Something like:

models = [model1,model2,model3]
data_sets = [data1,data2,data3]

def generate_test_learn_parameter_function(model,data):
    def this_test(model,data):
        param = model.learn_parameters(data)
        assert((param - model.param) < 0.1 )
    return this_test

for model,data in zip(models,data_sets):
    # how can py.test can see the results of this function?
    generate_test_learn_parameter_function(model,data)

Explanation:

The code I'm writing takes a model structure, some data, and learns the parameters of the model. So my unit testing consists of a bunch of model structures and pre-generated data sets, and then a set of about 5 machine learning tasks to complete on each structure+data.

So if I hand code this I need one test per model per task. Every time I come up with a new model I need to then copy and paste the 5 tasks, changing which pickled structure+data I'm pointing at. This feels like bad practice to me. Ideally what I'd like is 5 template functions that define each of my 5 tasks and then to just spit out test functions for a list of structures that I specify.

Googling about brings me to either a) factories or b) closures, both of which addle my brain and suggest to me that there must be an easier way, as this problem must be faced regularly by proper programmers. So is there?

EDIT: So here's how to solve this problem!

def pytest_generate_tests(metafunc):
    if "model" in metafunc.funcargnames:
        models = [model1,model2,model3]
        for model in models:
            metafunc.addcall(funcargs=dict(model=model))

def test_awesome(model):
    assert model == "awesome"

This will apply the test_awesome test to each model in my list of models! Thanks @dfichter!

(NOTE: that assert always passes, btw)

Generally it's a bad idea to generate test code dynamically like that. Because then you have to write tests for the test code etc. copy and paste the 5 tasks I think this shows that instead of generating new code or copy-pasting, you can find commonalities that your functions can test without knowing exactly what they're testing. — Falmarri
– Falmarri, Commented Feb 7, 2011 at 17:07
At the moment I'm writing tests that look like def test_learn: for model in models: assert(error < threshold). But this means that the test could fail on any one of the models, whereas I'd like to have a test that fails for one model and a seperate test that passes for another model, even though it's the same machine learning function. — Mike Dewar
– Mike Dewar, Commented Feb 7, 2011 at 17:11
@Falmarri: en.wikipedia.org/wiki/Copy_and_paste_programming Please avoid it at almost all costs. — László Papp
– László Papp, Commented Sep 1, 2014 at 17:57

Community · Accepted Answer · 2019-08-14 10:49:49Z

21

Good instincts. py.test supports exactly what you're talking about with its pytest_generate_tests() hook. They explain it here.

edited Aug 14, 2019 at 10:49

CommunityBot

11 silver badge

answered Feb 8, 2011 at 4:05

dfichter

1,0888 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Mike Dewar Over a year ago

This is awesome. I'll edit my question to add what I've done in the end, in case anyone comes here looking for similar things..

asaf92 Over a year ago

Link is broken. Is it still supported?

delucasvb Over a year ago

I think this is it: docs.pytest.org/en/latest/how-to/…

Gill Bates · Accepted Answer · 2015-08-11 14:13:29Z

8

You also could do that using parametrized fixtures. While hooks, is an API to build plugins for Py.test, parametrized fixtures is a generalized way to make a fixtures that outputs multiple values and generates additional test cases for them.

Plugins are meant to be some project-wide (or package-wide) features, not test case specific features and parametrized fixtures are exactly what's needed to parametrize some resource for test case(s).

So your solution could be rewritten as that:

@pytest.fixture(params=[model1, model2, model3])
def model(request):
    return request.param

def test_awesome(model):
    assert model == "awesome"

answered Aug 11, 2015 at 14:13

Gill Bates

15.5k25 gold badges81 silver badges144 bronze badges

Comments

Zbyněk Winkler · Accepted Answer · 2022-05-31 12:24:51Z

1

The simplest solution today is to use pytest.mark.parametrize:

@pytest.mark.parametrize('model', ["awesome1", "awesome2", "awesome3"])
def test_awesome(model):
    assert model.startswith("awesome")

answered May 31, 2022 at 12:24

Zbyněk Winkler

1,5151 gold badge14 silver badges13 bronze badges

Collectives™ on Stack Overflow

Generating py.test tests in Python

3 Answers 3

3 Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related