20

I've got a python program which is tested by running it several times with different inputs, and comparing the outputs against reference results.

I'd like to get code coverage of all the tests combined, so I can see if there are extra sets of inputs I should be using to get complete coverage. I've looked at the coverage module but can't work out how I can make it do this.

Any clues?

Improve this question

2 Answers 2

Reset to default
32

If running on the same machine, run it with the -a option, which accumulates coverage data across multiple calls.

Example:

coverage erase

coverage run -a <command> [arguments, ...]

coverage run -a <command> ... # repeat as many times as needed.

coverage report

coverage html

doc: http://coverage.readthedocs.org/en/latest/cmd.html#data-file

Hope this helps.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

see also coverage combine
4

Ned Batchelder's coverage.py has a feature to combine the results of multiple runs, which seems to be exactly what you are looking for.

Improve this answer

4 Comments

Yes, this is exactly right. You can use this to run across multiple data sets, across multiple versions of Python, or multiple dependency choices (e.g. different versions of middleware or support modules). You can "orchestrate" multiple test runs in a shell script, but you can do that one better: The tox test runner will auto-create pristine virtual environments and run tests in those; highly recommended for anything beyond the simplest testing setup.
@opa The OP selected this answer because it apparently helped them. The other answer was added two years later. I'm a bit puzzled why this got you so angry – I'm happy to improve my answers based on friendly suggestions, but when being shouted at I don't really feel like volunteering more time to help people.
@opa I did not complain about your issue with this answer. You do have a point, and I never questioned that. I was rather complaining about the tone of your comment, which is unnecessarily rude. It would have been sufficient to say "You should include the gist of the linked page in this answer, since otherwise it will become useless if that page goes away" or similar. Instead you chose to say that the answer is "godawful", that it is unfortunate that the OP selected it etc, which sounds rather hostile. I will improve the answer when I find some time to look into this.
@SvenMarnach You're right, my hostility wasn't productive. I get particularly frustrated when I see experienced users have made the same mistakes that lead to new users producing poor quality content.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.