2

For example I have a list:

L=[-13, -24, -21, -3, -23, -15, -14, -27, -13, -12]
  1. if type in %timeit -n 10 myList = [item for item in L if item < 15] The output is 10 loops, best of 3: 1.25 µs per loop

  2. if I type myGen = (item for item in L if item < 15) The output is 1000000 loops, best of 3: 561 ns per loop

I don't understand in case 2, why a generator takes 1000000 loops rather than 10 loops? And what does "best of 3" mean? And how can I work out the total seconds it takes for each commond, like 10*1.25=12.5 us for case 1?

1
  • 1
    "best of 3" is from the number of repeats: -r N, --repeat=N "how many times to repeat the timer (default 3)" Commented Aug 24, 2016 at 13:46

1 Answer 1

6

You didn't include the -n argument to %timeit in your second example, so ipython varies the number of repetitions based on how long a trial-run takes; the faster the piece of code being tested, the more iterations are done to get a more accurate per-iteration time value.

Moreover, the tests are run several times to try and minimise external factors (for example, when your OS just happened to schedule a disk buffer flush and everything else becomes a little bit slower). This is where the 'best of 3' comes in; the tests were run 3 times in a row and the best timings were picked.

See the %timeit magic command documentation, which includes these options and their default behaviour:

-n<N>: execute the given statement <N> times in a loop. If this value is not given, a fitting value is chosen.

-r<R>: repeat the loop iteration <R> times and take the best result. Default: 3

Your first example does use -n 10 so it was run just 10 times.

Because creating a generator object with a generator expression is near-instant, ipython can execute the loop way more often than executing a list comprehension (which has to execute the for loop and produce a list object with all the results there and then). Remember that a generator expression does not do any work until you drive iteration.

If you wanted to compare how long a generator expression takes to produce the same results as a list comprehension, you'd have to actually iterate. You could pass the expression to a list() call to actually produce a list too:

%timeit -n 10 myGen = (item for item in L if item < 15); list(myGen)

This'll be slower as a generator has a little more overhead than a list comprehension:

In [1]: L=[-13, -24, -21, -3, -23, -15, -14, -27, -13, -12]

In [2]: %timeit -n 10 myList = [item for item in L if item < 15]
10 loops, best of 3: 1.29 µs per loop

In [3]: %timeit -n 10 myGen = (item for item in L if item < 15); list(myGen)
10 loops, best of 3: 1.72 µs per loop

Note that you have to re-create the generator each test iteration because generators can produce their output just once.

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.