Part 17: Just Another Way to Spell “Callback”
In this Part we’re going to return to the subject of callbacks. We’ll introduce another technique for writing callbacks in Twisted that uses generators. We’ll show how the technique works and contrast it with using “pure” Deferreds. Finally we’ll rewrite one of our poetry clients using this technique. But first let’s review how generators work so we can see why they are a candidate for creating callbacks.
A Brief Review of Generators
As you probably know, a Python generator is a “restartable function” that you create by using the
yield expression in the body of your function. By doing so, the function becomes a “generator function” that returns an iterator you can use to run the function in a series of steps. Each cycle of the iterator restarts the function, which proceeds to execute until it reaches the next
Generators (and iterators) are often used to represent lazily-created sequences of values. Take a look at the example code in inline-callbacks/gen-1.py:
def my_generator(): print 'starting up' yield 1 print "workin'" yield 2 print "still workin'" yield 3 print 'done' for n in my_generator(): print n
Here we have a generator that creates the sequence 1, 2, 3. If you run the code, you will see the
for loop as the loop cycles through the generator.
We can make this code more explicit by creating the generator ourselves (inline-callbacks/gen-2.py):
def my_generator(): print 'starting up' yield 1 print "workin'" yield 2 print "still workin'" yield 3 print 'done' gen = my_generator() while True: try: n = gen.next() except StopIteration: break else: print n
Considered as a sequence, the generator is just an object for getting successive values. But we can also view things from the point of view of the generator itself:
- The generator function doesn’t start running until “called” by the loop (using the
- Once the generator is running, it keeps running until it “returns” to the loop (using
- When the loop is running other code (like the
- When the generator is running, the loop is not running (it’s “blocked” waiting for the generator).
- Once a generator
yields control to the loop, an arbitrary amount of time may pass (and an arbitrary amount of other code may execute) until the generator runs again.
This is very much like the way callbacks work in an asynchronous system. We can think of the
while loop as the reactor, and the generator as a series of callbacks separated by
yield statements, with the interesting fact that all the callbacks share the same local variable namespace, and the namespace persists from one callback to the next.
Furthermore, you can have multiple generators active at once (see the example in inline-callbacks/gen-3.py), with their “callbacks” interleaved with each other, just as you can have independent asynchronous tasks running in a system like Twisted.
Something is still missing, though. Callbacks aren’t just called by the reactor, they also receive information. When part of a deferred’s chain, a callback either receives a result, in the form of a single Python value, or an error, in the form of a
Starting with Python 2.5, generators were extended in a way that allows you to send information to a generator when you restart it, as illustrated in inline-callbacks/gen-4.py:
class Malfunction(Exception): pass def my_generator(): print 'starting up' val = yield 1 print 'got:', val val = yield 2 print 'got:', val try: yield 3 except Malfunction: print 'malfunction!' yield 4 print 'done' gen = my_generator() print gen.next() # start the generator print gen.send(10) # send the value 10 print gen.send(20) # send the value 20 print gen.throw(Malfunction()) # raise an exception inside the generator try: gen.next() except StopIteration: pass
In Python 2.5 and later versions, the
yield statement is an expression that evaluates to a value. And the code that restarts the generator can determine that value using the
send method instead of
next (if you use
next the value is
None). What’s more, you can actually raise an arbitrary exception inside the generator using the
throw method. How cool is that?
Given what we just reviewed about
throwing values and exceptions into a generator, we can envision a generator as a series of callbacks, like the ones in a deferred, which receive either results or failures. The callbacks are separated by
yields and the value of each
yield expression is the result for the next callback (or the
yield raises an exception and that’s the failure). Figure 35 shows the correspondence:
Now when a series of callbacks is chained together in a deferred, each callback receives the result from the one prior. That’s easy enough to do with a generator — just
send the value you got from the previous run of the generator (the value it
yielded) the next time you restart it. But that also seems a bit silly. Since the generator computed the value to begin with, why bother sending it back? The generator could just save the value in a variable for the next time it’s needed. So what’s the point?
Recall the fact we learned in Part 13, that the callbacks in a deferred can return deferreds themselves. And when that happens, the outer deferred is paused until the inner deferred fires, and then the next callback (or errback) in the outer deferred’s chain is called with the result (or failure) from the inner deferred.
So imagine that our generator
yields a deferred object instead of an ordinary Python value. The generator is now “paused”, and that’s automatic; generators always pause after every
yield statement until they are explicitly restarted. So we can delay restarting the generator until the deferred fires, at which point we either
send the value (if the deferred succeeds) or
throw the exception (if the deferred fails). That would make our generator a genuine sequence of asynchronous callbacks and that’s the idea behind the
inlineCallbacks function in
Consider the example program in inline-callbacks/inline-callbacks-1.py:
from twisted.internet.defer import inlineCallbacks, Deferred @inlineCallbacks def my_callbacks(): from twisted.internet import reactor print 'first callback' result = yield 1 # yielded values that aren't deferred come right back print 'second callback got', result d = Deferred() reactor.callLater(5, d.callback, 2) result = yield d # yielded deferreds will pause the generator print 'third callback got', result # the result of the deferred d = Deferred() reactor.callLater(5, d.errback, Exception(3)) try: yield d except Exception, e: result = e print 'fourth callback got', repr(result) # the exception from the deferred reactor.stop() from twisted.internet import reactor reactor.callWhenRunning(my_callbacks) reactor.run()
Run the example and you will see the generator execute to the end and then stop the reactor. The example illustrates several aspects of the
inlineCallbacks function. First,
inlineCallbacks is a decorator and it always decorates generator functions, i.e., functions that use
yield. The whole purpose of
inlineCallbacks is turn a generator into a series of asynchronous callbacks according to the scheme we outlined before.
Second, when we invoke an
inlineCallbacks-decorated function, we don’t need to call
throw ourselves. The decorator takes care of those details for us and ensures the generator will run to the end (assuming it doesn’t raise an exception).
Third, if we
yield a non-deferred value from the generator, it is immediately restarted with that same value as the result of the
And finally, if we
yield a deferred from the generator, it will not be restarted until that deferred fires. If the deferred succeeds, the result of the
yield is just the result from the deferred. And if the deferred fails, the
yield statement raises the exception. Note the exception is just an ordinary
Exception object, rather than a
Failure, and we can catch it with a
except statement around the
In the example we are just using
callLater to fire the deferreds after a short period of time. While that’s a handy way to put in a non-blocking delay into our callback chain, normally we would be
yielding a deferred returned by some other asynchronous operation (i.e.,
get_poetry) invoked from our generator.
Ok, now we know how an
inlineCallbacks-decorated function runs, but what return value do you get if you actually call one? As you might have guessed, you get a deferred. Since we can’t know exactly when that generator will stop running (it might
yield one or more deferreds), the decorated function itself is asynchronous and a deferred is the appropriate return value. Note the deferred that is returned isn’t one of the deferreds the generator may
yield. Rather, it’s a deferred that fires only after the generator has completely finished (or throws an exception).
If the generator throws an exception, the returned deferred will fire its errback chain with that exception wrapped in a
Failure. But if we want the generator to return a normal value, we must “return” it using the
defer.returnValue function. Like the ordinary
return statement, it will also stop the generator (it actually raises a special exception). The inline-callbacks/inline-callbacks-2.py example illustrates both possibilities.
inlineCallbacks to work with a new version of our poetry client. You can see the code in twisted-client-7/get-poetry.py. You may wish to compare it to client 6.0 in twisted-client-6/get-poetry.py. The relevant changes are in
def poetry_main(): addresses = parse_args() xform_addr = addresses.pop(0) proxy = TransformProxy(*xform_addr) from twisted.internet import reactor results =  @defer.inlineCallbacks def get_transformed_poem(host, port): try: poem = yield get_poetry(host, port) except Exception, e: print >>sys.stderr, 'The poem download failed:', e raise try: poem = yield proxy.xform('cummingsify', poem) except Exception: print >>sys.stderr, 'Cummingsify failed!' defer.returnValue(poem) def got_poem(poem): print poem def poem_done(_): results.append(_) if len(results) == len(addresses): reactor.stop() for address in addresses: host, port = address d = get_transformed_poem(host, port) d.addCallbacks(got_poem) d.addBoth(poem_done) reactor.run()
In our new version the
inlineCallbacks generator function
get_transformed_poem is responsible for both fetching the poem and then applying the transformation (via the transform service). Since both operations are asynchronous, we yield a deferred each time and then (implicitly) wait for the result. As in client 6.0, if the transformation fails we just return the original poem. Notice we can use
except statements to handle asynchronous errors inside the generator.
We can test the new client out in the same way as before. First start up a transform server:
python twisted-server-1/transformedpoetry.py --port 10001
Then start a couple of poetry servers:
python twisted-server-1/fastpoetry.py --port 10002 poetry/fascination.txt python twisted-server-1/fastpoetry.py --port 10003 poetry/science.txt
Now you can run the new client:
python twisted-client-7/get-poetry.py 10001 10002 10003
Try turning off one or more of the servers to see how the client handles errors.
Deferred object, the
inlineCallbacks function gives us a new way of organizing our asynchronous callbacks. And, as with deferreds,
inlineCallbacks doesn’t change the rules of the game. Specifically, our callbacks still run one at a time, and they are still invoked by the reactor. We can confirm that fact in our usual way by printing out a traceback from an inline callback, as in the example script inline-callbacks/inline-callbacks-tb.py. Run that code and you will get a traceback with
reactor.run() at the top, lots of helper functions in between, and our callback at the bottom.
We can adapt Figure 29, which explains what happens when one callback in a deferred returns another deferred, to show what happens when an
yields a deferred. See Figure 36:
The same figure works in both cases because the idea being illustrated is the same — one asynchronous operation is waiting for another.
inlineCallbacks and deferreds solve many of the same problems, why choose one over the other? Here are some potential advantages of
- Since the callbacks share a namespace, there is no need to pass extra state around.
- The callback order is easier to see, as they just execute from top to bottom.
- With no function declarations for individual callbacks and implicit flow-control, there is generally less typing.
- Errors are handled with the familiar
And here are some potential pitfalls:
- The callbacks inside the generator cannot be invoked individually, which could make code re-use difficult. With a deferred, the code constructing the deferred is free to add arbitrary callbacks in an arbitrary order.
- The compact form of a generator can obscure the fact that an asynchronous callback is even involved. Despite its visually similar appearance to an ordinary sequential function, a generator behaves in a very different manner. The
inlineCallbacksfunction is not a way to avoid learning the asynchronous programming model.
As with any technique, practice will provide the experience necessary to make an informed choice.
In this Part we learned about the
inlineCallbacks decorator and how it allows us to express a sequence of asynchronous callbacks in the form of a Python generator.
In Part 18 we will learn a technique for managing a set of “parallel” asynchronous operations.
- Why is the
- Study the implementation of
inlineCallbacksand its helper function
_inlineCallbacks. Ponder the phrase “the devil is in the details”.
- How many callbacks are contained in a generator with N
yieldstatements, assuming it has no loops or
- Poetry client 7.0 might have three generators running at once. Conceptually, how many different ways might they be interleaved with one another? Considering the way they are invoked in the poetry client and the implementation of
inlineCallbacks, how many ways do you think are actually possible?
- Move the
got_poemcallback in client 7.0 inside the generator.
- Then move the
poem_donecallback inside the generator. Be careful! Make sure to handle all the failure cases so the reactor gets shutdown no matter what. How does the resulting code compare to using a deferred to shutdown the reactor?
- A generator with
yieldstatements inside a
whileloop can represent a conceptually infinite sequence. What does such a generator decorated with