Python nested loop with generators does not work (in some cases)?

Question

Would somebody please explain the behavior of a nested loop using generators? Here is an example.

a = (x for x in range(3))
b = (x for x in range(2))
for i in a:
    for j in b:
        print (i,j)

The outer loop is not evaluated after the first iteration for some reason. The result is,

(0, 0)
(0, 1)

On the other hand, if generators are directly inserted into the loops, it does what I expect.

for i in (x for x in range(3)):
    for j in (x for x in range(2)):
        print (i,j)

giving all 3x2 pairs.

(0, 0)
(0, 1)
(1, 0)
(1, 1)
(2, 0)
(2, 1)

Lauritz V. Thaulow · Accepted Answer · 2012-07-19 21:46:58Z

25

It's because the b generator is exhausted during the first iteration of the outer for loop. Subsequent iterations will in effect have an empty inner loop (like for x in ()) so what's inside is never executed. This gives the false impression that it's the outer loop that fails, somehow.

Your second example works because there the inner generator is created anew for each outer loop. To fix your first example, you have to do the same:

a = (x for x in range(3))
for i in a:
    b = (x for x in range(2))
    for j in b:
        print (i,j)

edited Jul 19, 2012 at 21:46

answered Jul 19, 2012 at 21:26

Lauritz V. Thaulow

51.3k13 gold badges76 silver badges94 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

phantomile Over a year ago

Aha! I did't notice the exhaustion of the generator. Thank you so much.

Jon Clements · Accepted Answer · 2012-07-19 21:37:53Z

9

@lazyr has answered this brilliantly, but I would point out for reference that when using nested generators it's worth knowing about itertools.product...

for i, j in itertools.product(range(3), range(2)):
    print (i, j)

or (if you have lots of vals):

for vals in itertools.product(range(45), range(12), range(3)):
    print (sum(vals))

It's (IMHO) readable and avoids excessive indentation.

answered Jul 19, 2012 at 21:37

Jon Clements

143k34 gold badges254 silver badges288 bronze badges

Comments

parity3 · Accepted Answer · 2015-06-05 14:56:00Z

itertools.product is the best fit for this example. But you may want more options during iterations. Here is one way to still get the product in your example without using the product method:

a = (range(2) for x in range(3))
for i in a:
    for j in i:
        print (i,j)

Also, I use itertoolz.concat from the pytoolz functional helper library to streamline/flatten cases like this. concat is just like itertools.chain but instead takes a single argument which yields iterators that get unraveled:

from pytoolz import itertoolz
a = (((x,y) for y in range(2)) for x in range(3))
for i,j in itertoolz.concat(a):
    print (i,j)

So, the above looks less readable than the product method but allows for finer-grained transformations / filtering at each loop level. And of course, you don't have nested for-loops during the final iteration logic which can be nice.

Also, if you use pytoolz you should probably be using cytoolz, it's the same library compiled into C.

Collectives™ on Stack Overflow

Python nested loop with generators does not work (in some cases)?

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related