python generators for concurrency

Question

I am following slides from Python's guru David Beazley. It states "Generators are also used for concurrency. Here is an example:

from collections import deque

def countdown(n):
    while n > 0:
        print("T-minus", n)
        yield
        n -=1

def countup(n):
    x = 0
    while x > n:
        print("Up we go", x)
        yield
        x +=1

# instantiate some tasks in a queue
tasks = deque([countdown(10),
               countdown(5),
               countup(20)
               ])

# run a little scheduler
while tasks:
    t = tasks.pop()  # get a task
    try:
        next(t)   # run it until it yields
        tasks.appendleft(t) # reschedule
    except StopIteration:
        pass

Here is the output:

T-minus 5
T-minus 10
T-minus 4
T-minus 9
T-minus 3
T-minus 8
T-minus 2
T-minus 7
T-minus 1
T-minus 6
T-minus 5
T-minus 4
T-minus 3
T-minus 2
T-minus 1

The question is how is concurrency introduced by generators and how is it manifesting?

The concurrency is plain as day (to me) in that example. Perhaps the issue is with your understanding of the term concurrency? — user395760
– user395760, Commented May 16, 2014 at 21:29
Please explain Concurrency in layman's terms so i and others like me can better understand. I see this as one task after the other as they are removed and then put back into the queue — user2979872
– user2979872, Commented May 16, 2014 at 21:37
If you haven't already, you could try the PyCon talk that these slides are from: youtube.com/watch?v=MCs5OvhV9S4 — rbrtl
– rbrtl, Commented Jul 4, 2016 at 21:38

SingleNegationElimination · Accepted Answer · 2014-05-16 21:36:44Z

2

This bit of code implements the concept of "green threads", cooperative, userland (as opposed to Preemptive, kernel) threading.

The "threads" are the generators, each function with yeild or yield from in it. The scheduler lives, obviously, inside the if __name__ == '__main__': bit.

So, lets imagine we have not generators but regular lists, and each list has in it, a sequence of functions.

def doThis(): pass
def sayThis(): pass
def doThat(): pass
...

myThread = [doThis, doThat, doAnother]
yourThread = [sayThis, sayThat, sayAnother]

We could run all of the functions in order:

for thread in [myThread, yourThread]:
    for stmt in thread:
        stmt()

Or we could do them in some other order:

for myStmt, yourStmt in zip(myThread, yourThread):
    myStmt()
    yourStmt()

In the first "scheduler", we exhaust the first thread, and then proceed to the second thread. In the second scheduler, we interleave statements out of both threads, first mine, then yours, then back to mine.

It's because we are interleaving "statements" across multiple "threads" before exausting those threads that we can say that the second scheduler gives concurrency.

Note that concurrency doesn't neccesarily mean parallelism. It's not simultaneous execution, just overlapping.

answered May 16, 2014 at 21:36

SingleNegationElimination

157k35 gold badges269 silver badges306 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

user2979872 Over a year ago

Just as another question. If generators can help with concurrency, are threads really needed? If yes, why? if not, why not? Thanks a lot

SingleNegationElimination Over a year ago

Many things in programming are not strictly neccessary, for example: variables, but they are convenient, so we keep them around anyway.

Benyamin Jafari · Accepted Answer · 2019-11-02 13:52:26Z

0

Here is an example to clarify:

from collections import deque

def coro1():
    for i in range(1, 10):
        yield i

def coro2():
    for i in range(1, 10):
        yield i*10

print('Async behaviour'.center(60, '#'))
tasks = deque()
tasks.extend([coro1(), coro2()])

while tasks:
    task = tasks.popleft()  # select and remove a task (coro1/coro2).
    try:
        print(next(task))
        tasks.append(task)  # add the removed task (coro1/coro2) for permutation.
    except StopIteration:
        pass

Out:

######################Async behaviour#######################
1
10
2
20
3
30
4
40
5
50
6
60
7
70
8
80
9
90

answered Nov 2, 2019 at 13:52

Benyamin Jafari

35.8k35 gold badges161 silver badges184 bronze badges

Collectives™ on Stack Overflow

python generators for concurrency

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related