9

I am studying coroutines and generators in various programming languages.

I was wondering if there is a cleaner way to combine together two coroutines implemented via generators than yielding back at the caller whatever the callee yields?

Let's say that we are using the following convention: all yields apart from the last one return null, while the last one returns the result of the coroutine. So, for example, we could have a coroutine that invokes another:

def A():
  # yield until a certain condition is met
  yield result

def B():
  # do something that may or may not yield
  x = bind(A())
  # ...
  return result

in this case I wish that through bind (which may or may not be implementable, that's the question) the coroutine B yields whenever A yields until A returns its final result, which is then assigned to x allowing B to continue.

I suspect that the actual code should explicitly iterate A so:

def B():
  # do something that may or may not yield
  for x in A(): ()
  # ...
  return result

which is a tad ugly and error prone...

PS: it's for a game where the users of the language will be the designers who write scripts (script = coroutine). Each character has an associated script, and there are many sub-scripts which are invoked by the main script; consider that, for example, run_ship invokes many times reach_closest_enemy, fight_with_closest_enemy, flee_to_allies, and so on. All these sub-scripts need to be invoked the way you describe above; for a developer this is not a problem, but for a designer the less code they have to write the better!

Improve this question
15
  • I thought the examples clearly demonstrated idiomatic. Since I'm unable to understand what's wrong with the examples, could you state which examples you found to be unclear? Which examples were confusing? Can you be more specific on how all those examples where not able to show idiomatic Python? Commented May 10, 2011 at 13:04
  • 3
    I've read precisely those articles, and the PEP-342 leaves me somewhat confused: is it some actual extension that is currently working in Python? Is the Trampoline class shown there part of the standard libraries of the language? BTW, my question was very precise, and it was about the IDIOMATIC way to pass control around coroutines. The fact that I can read about a ton of ways to do so really does not help. Neither does your snarkiness... Commented May 10, 2011 at 13:09
  • Idiomatic is about the "standard" way to perform some function; there is absolutely nothing wrong with iterating the results of a nested coroutine, but there are examples in the literature of programming languages where yielding automatically climbs down the call stack and so you do not need to re-yield at each caller, hence my curiosity if this pattern is covered by sintactic sugar in Python or not! Commented May 10, 2011 at 13:11
  • @Giuseppe Maggiore: "programming languages where yielding automatically climbs down the call stack" That doesn't sound like the same question. Are you asking for idiomatic Python -- as shown by numerous examples -- or are you asking for some other feature that's not shown in the Python examples but is shown in other languages? I'm afraid that I can't understand your question at all. Can you please clarify what you're really looking for? Commented May 10, 2011 at 13:19
  • You might want to check out what "idiomatic" means, because I'm getting the feeling that this is where we are having a problem (thefreedictionary.com/idiomatic). I know that I can build coroutines in Python. I know how to do so with generators. I want to know if (A) there is an accepted "right way" to do so, which maybe uses some syntactic sugar or other technique, or (B) if there is no recommendation and any way that works is acceptable. (A) is what I mean by idiomatic. In comparison, writing in a functional language with lots of mutable references would NOT be idiomatic. Cheers Commented May 10, 2011 at 13:30

2 Answers 2

Reset to default
19

Edit: I recommend using Greenlet. But if you're interested in a pure Python approach, read on.

This is addressed in PEP 342, but it's somewhat tough to understand at first. I'll try to explain simply how it works.

First, let me sum up what I think is the problem you're really trying to solve.

Problem

You have a callstack of generator functions calling other generator functions. What you really want is to be able to yield from the generator at the top, and have the yield propagate all the way down the stack.

The problem is that Python does not (at a language level) support real coroutines, only generators. (But, they can be implemented.) Real coroutines allow you to halt an entire stack of function calls and switch to a different stack. Generators only allow you to halt a single function. If a generator f() wants to yield, the yield statement has to be in f(), not in another function that f() calls.

The solution that I think you're using now, is to do something like in Simon Stelling's answer (i.e. have f() call g() by yielding all of g()'s results). This is very verbose and ugly, and you're looking for syntax sugar to wrap up that pattern. Note that this essentially unwinds the stack every time you yield, and then winds it back up again afterwards.

Solution

There is a better way to solve this problem. You basically implement coroutines by running your generators on top of a "trampoline" system.

To make this work, you need to follow a couple patterns: 1. When you want to call another coroutine, yield it. 2. Instead of returning a value, yield it.

so

def f():
    result = g()
    # …
    return return_value

becomes

def f():
    result = yield g()
    # …
    yield return_value

Say you're in f(). The trampoline system called f(). When you yield a generator (say g()), the trampoline system calls g() on your behalf. Then when g() has finished yielding values, the trampoline system restarts f(). This means that you're not actually using the Python stack; the trampoline system manages a callstack instead.

When you yield something other than a generator, the trampoline system treats it as a return value. It passes that value back to the caller generator through the yield statement (using .send() method of generators).

Comments

This kind of system is extremely important and useful in asynchronous applications, like those using Tornado or Twisted. You can halt an entire callstack when it's blocked, go do something else, and then come back and continue execution of the first callstack where it left off.

The drawback of the above solution is that it requires you to write essentially all your functions as generators. It may be better to use an implementation of true coroutines for Python - see below.

Alternatives

There are several implementations of coroutines for Python, see: http://en.wikipedia.org/wiki/Coroutine#Implementations_for_Python

Greenlet is an excellent choice. It is a Python module that modifies the CPython interpreter to allow true coroutines by swapping out the callstack.

Python 3.3 should provide syntax for delegating to a subgenerator, see PEP 380.

Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

Very useful and clear answer, thank's! However, when you say that standard Python coroutines essentially require to write all functions as generators, did you mean only first level functions or really all functions? As you said above, when yielding something other than a generator, the trampoline system still works, so theoretically we can just yield at the first-layer functions any other functions that may or may not be generators themselves. Am I right?
All "functions" between the trampoline system and a yield must be written as generators. You can call regular functions normally, but then you can't effectively "yield" from that function or any functions it calls. Does that make sense / answer your question?
I highly recommend using Greenlet - it's a true implementation of coroutines for Python, and you don't have to use any of these patterns I've described. The trampoline stuff is for people who are interested in how you can do it in pure Python.
To anyone reading this in 2015 or later, the new syntax is 'yield from' (PEP 380) and it allows true coroutines in Python >3.3.
2

Are you looking for something like this?

def B():
   for x in A():
     if x is None:
       yield
     else:
       break

   # continue, x contains value A yielded
Improve this answer

4 Comments

Yes, that is what I am doing. My question is if this is the idiomatic way or if there is some syntactic construct that is capable of hiding this pattern which recurs very often in my application.
@Guiseppe Maggiore: I'm not aware of any such constructs. However, it seems strange that you need this pattern often... I can't think of many valid used cases off the top of my head. If you give more context information, maybe we can propose an alternative solution which is more elegant over all?
It's for a game where the users of the language will be the designers who write scripts (script = coroutine). Each character has an associated script, and there are many sub-scripts which are invoked by the main script; consider that, for example, run_ship invokes many times reach_closest_enemy, fight_with_closest_enemy, flee_to_allies, and so on. All these sub-scripts need to be invoked the way you describe above; for a developer this is not a problem, but for a designer the less code they have to write the better!
@Guiseppe Maggiore: I'd propose you add that last comment to the question so that other get a chance of answering it, too...

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.