16

Let's say that I've got my code base to as high a degree of unit test coverage as makes sense. (Beyond a certain point, increasing coverage doesn't have a good ROI.)

Next I want to test performance. To benchmark code to make sure that new commits aren't slowing things down needlessly. I was very intrigued by Safari's zero tolerance policy for slowdowns from commits. I'm not sure that level of commitment to speed has a good ROI for most projects, but I'd at least like to be alerted that a speed regression has happened, and be able to make a judgment call about it.

Environment is Python on Linux, and a suggestion that was also workable for BASH scripts would make me very happy. (But Python is the main focus.)

Improve this question
1
  • It seems this question still lacks a good answer. Of course - a general solution may not exist, but there are a lot of possibilities - e.g. measure time with time.clock() before and after, then self.assertLess(elapsed, some_value) - not portable across space (machines with different specs) and time (Moore's Law), but it's a start. Commented Sep 23, 2019 at 12:33

4 Answers 4

Reset to default
7

You will want to do performance testing at a system level if possible - test your application as a whole, in context, with data and behaviour as close to production use as possible.

This is not easy, and it will be even harder to automate it and get consistent results.

Moreover, you can't use a VM for performance testing (unless your production environment runs in VMs, and even then, you'd need to run the VM on a host with nothing else on).

When you say doing performance unit-testing, that may be valuable, but only if it is being used to diagnose a problem which really exists at a system level (not just in the developer's head).

Also, performance of units in unit testing sometimes fails to reflect their performance in-context, so it may not be useful at all.

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

+1: AND... Performance testing only makes sense when you have absolute performance objectives. Rare in general; essential in military and ebedded control systems.
Sorry @S.Lott - I strongly disagree with you on that. Anytime more than 2 or 3 people are working on a project and there is an end user with expectations of responsiveness, teams will do well to automatically monitor their performance against objectives. Otherwise, feature after feature gets added without true understanding of its effects on performance and customer experience.
4

While I agree that testing performance at a system level is ultimately more relevant, if you'd like to do UnitTest style load testing for Python, FunkLoad http://funkload.nuxeo.org/ does exactly that.

Micro benchmarks have their place when you're trying to speed up a specific action in your codebase. And getting subsequent performance unit tests done is a useful way to ensure that this action that you just optimized does not unintentionally regress in performance upon future commits.

Improve this answer

1 Comment

For people who, like me, stumble upon this years after it was written: Note that the link is dead, and the last PyPI upload (as of me writing this) is from 2015. I suppose it's fair to assume that the project is dead or in hibernation.
3

MarkR is right... doing real-world performance testing is key, and may be somewhat dodgey in unit tests. Having said that, have a look at the cProfile module in the standard library. It will at least be useful for giving you a relative sense from commit-to-commit of how fast things are running, and you can run it within a unit test, though of course you'll get results in the details that include the overhead of the unit test framework itself.

In all, though, if your objective is zero-tolerance, you'll need something much more robust than this... cProfile in a unit test won't cut it at all, and may be misleading.

Improve this answer

Comments

2

When I do performance testing, I generally have a test suite of data inputs, and measure how long it takes the program to process each one.

You can log the performance on a daily or weekly basis, but I don't find it particularly useful to worry about performance until all the functionality is implemented.

If performance is too poor, then I break out cProfile, run it with the same data inputs, and try to see where the bottlenecks are.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.