An Introduction to Asynchronous Programming and Twisted

Part 9: A Second Interlude, Deferred

This continues the introduction started here. You can find an index to the entire series here.

More Consequence of Callbacks

We’re going to pause for a moment to think about callbacks again. Although we now know enough about deferreds to write simple asynchronous programs in the Twisted style, the Deferred class provides more features that only come into play in more complex settings. So we’re going to think up some more complex settings and see what sort of challenges they pose when programming with callbacks. Then we’ll investigate how deferreds address those challenges.

To motivate our discussion we’re going to add a hypothetical feature to our poetry client. Suppose some hard-working Computer Science professor has invented a new poetry-related algorithm, the Byronification Engine. This nifty algorithm takes a single poem as input and produces a new poem like the original, but written in the style of Lord Byron. What’s more, our professor has kindly provided a reference implementation in Python, with this interface:

class IByronificationEngine(Interface):

    def byronificate(poem):
        """
        Return a new poem like the original, but in the style of Lord Byron.

        Raises GibberishError if the input is not a genuine poem.
        """

Like most bleeding-edge software, the implementation has some bugs. This means that in addition to the documented exception, the byronificate method sometimes throws random exceptions when it hits a corner-case the professor forgot to handle.

We’ll also assume the engine runs fast enough that we can just call it in the main thread without worrying about tying up the reactor. This is how we want our program to work:

  1. Try to download the poem.
  2. If the download fails, tell the user we couldn’t get the poem.
  3. If we do get the poem, transform it with the Byronification Engine.
  4. If the engine throws a GibberishError, tell the user we couldn’t get the poem.
  5. If the engine throws another exception, just keep the original poem.
  6. If we have a poem, print it out.
  7. End the program.

The idea here is that a GibberishError means we didn’t get an actual poem after all, so we’ll just tell the user the download failed. That’s not so useful for debugging, but our users just want to know whether we got a poem or not. On the other hand, if the engine fails for some other reason then we’ll use the poem we got from the server. After all, some poetry is better than none at all, even if it’s not in the trademark Byron style.

Here’s the synchronous version of our code:

try:
    poem = get_poetry(host, port) # synchronous get_poetry
except:
    print >>sys.stderr, 'The poem download failed.'
else:
    try:
        poem = engine.byronificate(poem)
    except GibberishError:
        print >>sys.stderr, 'The poem download failed.'
    except:
        print poem # handle other exceptions by using the original poem
    else:
        print poem

sys.exit()

This sketch of a program could be make simpler with some refactoring, but it illustrates the flow of logic pretty clearly. We want to update our most recent poetry client (which uses deferreds) to implement this same scheme. But we won’t do that until Part 10. For now, instead, let’s imagine how we might do this with client 3.1, our last client that didn’t use deferreds at all. Suppose we didn’t bother handling exceptions, but instead just changed the got_poem callback like this:

def got_poem(poem):
    poems.append(byron_engine.byronificate(poem))
    poem_done()

What happens when the byronificate method raises a GibberishError or some other exception? Looking at Figure 11 from Part 6, we can see that:

  1. The exception will propagate to the poem_finished callback in the factory, the method that actually invokes the callback.
  2. Since poem_finished doesn’t catch the exception, it will proceed to poemReceived on the protocol.
  3. And then on to connectionLost, also on the protocol.
  4. And then up into the core of Twisted itself, finally ending up at the reactor.

As we have learned, the reactor will catch and log the exception instead of crashing. But what it certainly won’t do is tell the user we couldn’t download a poem. The reactor doesn’t know anything about poems or GibberishErrors, it’s a general-purpose piece of code used for all kinds of networking, even non-poetry-related networking.

Notice how, at each step in the list above, the exception moves to a more general-purpose piece of code than the one before. And at no step after got_poem is the exception in a piece of code that could be expected to handle an error in the specific way we want for this client. This situation is basically the exact opposite of the way exceptions propagate in synchronous code.

Take a look at Figure 15, an illustration of a call stack we might see with a synchronous poetry client :

Figure 15: exceptions in synchronous code
Figure 15: synchronous code and exceptions

The main function is “high-context”, meaning it knows a lot about the whole program, why it exists, and how it’s supposed to behave overall. Typically, main would have access to the command-line options that indicate just how the user wants the program to work (and perhaps what to do if something goes wrong). It also has a very specific purpose: running the show for a command-line poetry client.

The socket connect method, on the other hand, is “low-context”. All it knows is that it’s supposed to connect to some network address. It doesn’t know what’s on the other end or why we need to connect right now. But connect is quite general-purpose — you can use it no matter what sort of service you are connecting to.

And get_poetry is in the middle. It knows it’s getting some poetry (and that’s the only thing it’s really good at), but not what should happen if it can’t.

So an exception thrown by connect will  move up the stack, from low-context and general-purpose code to high-context and special-purpose code, until it reaches some code with enough context to know what to do when something goes wrong (or it hits the Python interpreter and the program crashes).

Of course the exception is really just moving up the stack no matter what rather than literally seeking out high-context code. It’s just that in a typical synchronous program “up the stack” and “towards higher-context” are the same direction.

Now recall our hypothetical modification to client 3.1 above. The call stack we analyzed is pictured in Figure 16, abbreviated to just a few functions:

Figure 16: asynchronous callbacks and exceptions
Figure 16: asynchronous callbacks and exceptions

The problem is now clear: during a callback, low-context code (the reactor) is calling higher-context code which may in turn call even higher-context code, and so on. So if an exception occurs and it isn’t handled immediately, close to the same stack frame where it occurred, it’s unlikely to be handled at all. Because each time the exception moves up the stack it moves to a piece of lower-context code that’s even less likely to know what to do.

Once an exception crosses over into the Twisted core the game is up. The exception will not be handled, it will only be noted (when the reactor finally catches it). So when we are programming with “plain old” callbacks (without using deferreds), we must be careful to catch every exception before it gets back into Twisted proper, at least if we want to have any chance of handling errors according to our own rules. And that includes exceptions caused by our own bugs!

Since a bug can exist anywhere in our code, we would need to wrap every callback we write in an extra “outer layer” of try/except statements so the exceptions from our fumble-fingered typos can be handled as well. And the same goes for our errbacks because code to handle errors can have bugs too.

Well that’s not so nice.

The Fine Structure of Deferreds

It turns out the Deferred class helps us solve this problem. Whenever a deferred invokes a callback or errback, it catches any exception that might be raised. In other words, a deferred acts as the “outer layer” of try/except statements so we don’t need to write that layer after all, as long as we use deferreds. But what does a deferred do with an exception it catches? Simple — it passes the exception (in the form of a Failure) to the next errback in the chain.

So the first errback we add to a deferred is there to handle whatever error condition is signaled when the deferred’s .errback(err) method is called. But the second errback will handle any exception raised by either the first callback or the first errback, and so on down the line.

Recall Figure 12, a visual representation of a deferred with some callbacks and errbacks in the chain. Let’s call the first callback/errback pair stage 0, the next pair stage 1, and so on.

At a given stage N, if either the callback or the errback (whichever was executed) fails, then the errback in stage N+1 is called with the appropriate Failure object and the callback in stage N+1 is not called.

By passing exceptions raised by callbacks “down the chain”, a deferred moves exceptions in the direction of “higher context”. This also means that invoking the callback and errback methods of a deferred will never result in an exception for the caller (as long as you only fire the deferred once!), so lower-level code can safely fire a deferred without worrying about catching exceptions. Instead, higher-level code catches the exception by adding errbacks to the deferred (with addErrback, etc.).

Now in synchronous code, an exception stops propagating as soon as it is caught. So how does an errback signal the fact that it “caught” the error? Also simple — by not raising an exception. And in that case, the execution switches over to the callback line. So at a given stage N, if either the callback or errback succeeds (i.e., doesn’t raise an exception) then the callback in stage N+1 is called with the return value from stage N, and the errback in stage N+1 is not called.

Let’s summarize what we know about the deferred firing pattern:

  1. A deferred contains a chain of ordered callback/errback pairs (stages). The pairs are in the order they were added to the deferred.
  2. Stage 0, the first callback/errback pair, is invoked when the deferred is fired. If the deferred is fired with the callback method, then the stage 0 callback is called. If the deferred is fired with the errback method, then the stage 0 errback is called.
  3. If stage N fails, then the stage N+1 errback is called with the exception (wrapped in a Failure) as the first argument.
  4. If stage N succeeds, then the stage N+1 callback is called with the stage N return value as the first argument.

This pattern is illustrated in Figure 17:

Figure 17: control flow in a deferred
Figure 17: control flow in a deferred

The green lines indicate what happens when a callback or errback succeeds and the red lines are for failures. The lines show both the flow of control and the flow of exceptions and return values down the chain. Figure 17 shows all possible paths a deferred might take, but only one path will be taken in any particular case. Figure 18 shows one possible path for a “firing”:

Figure 18: one possible deferred firing pattern
Figure 18: one possible deferred firing pattern

In figure 18, the deferred’s callback method is called, which invokes the callback in stage 0. That callback succeeds, so control (and the return value from stage 0) passes to the stage 1 callback. But that callback fails (raises an exception), so control switches to the errback in stage 2. The errback “handles” the error (it doesn’t raise an exception) so control moves back to the callback chain and the callback in stage 3 is called with the result from the stage 2 errback.

Notice that any path you can make with Figure 17 will pass through every stage in the chain, but only one member of the callback/errback pair at any stage will be called.

In Figure 18, we’ve indicated that the stage 3 callback succeeds by drawing a green arrow out of it, but since there aren’t any more stages in that deferred, the result of stage 3 doesn’t really go anywhere. If the callback succeeds, that’s not really a problem, but what if it had failed? If the last stage in a deferred fails, then we say the failure is unhandled, since there is no errback to “catch” it.

In synchronous code an unhandled exception will crash the interpreter, and in plain-old-callbacks asynchronous code an unhandled exception is caught by the reactor and logged. What happens to unhandled exceptions in deferreds? Let’s try it out and see. Look at the sample code in twisted-deferred/defer-unhandled.py. That code is firing a deferred with a single callback that always raises an exception. Here’s the output of the program:

Finished
Unhandled error in Deferred:
Traceback (most recent call last):
  ...
--- <exception caught here> ---
  ...
exceptions.Exception: oops

Some things to notice:

  1. The last print statement runs, so the program is not “crashed” by the exception.
  2. That means the Traceback is just getting printed out, it’s not crashing the interpreter.
  3. The text of the traceback tells us where the deferred itself caught the exception.
  4. The “Unhandled” message gets printed out after “Finished”.

So when you use deferreds, unhandled exceptions in callbacks will still be noted, for debugging purposes, but as usual they won’t crash the program (in fact they won’t even make it to the reactor, the deferred will catch them first). By the way, the reason that “Finished” comes first is because the “Unhandled” message isn’t actually printed until the deferred is garbage collected. We’ll see the reason for that in a future Part.

Now, in synchronous code we can “re-raise” an exception using the raise keyword without any arguments. Doing so raises the original exception we were handling and allows us to take some action on an error without completely handling it. It turns out we can do the same thing in an errback. A deferred will consider a callback/errback to have failed if:

  • The callback/errback raises any kind of exception, or
  • The callback/errback returns a Failure object.

Since an errback’s first argument is always a Failure, an errback can “re-raise” the exception by returning its first argument, after performing whatever action it wants to take.

Callbacks and Errbacks, Two by Two

One thing that should be clear from the above discussion is that the order you add callbacks and errbacks to a deferred makes a big difference in how the deferred will fire. What should also be clear is that, in a deferred, callbacks and errbacks always occur in pairs. There are four methods on the Deferred class you can use to add pairs to the chain:

  1. addCallbacks
  2. addCallback
  3. addErrback
  4. addBoth

Obviously, the first and last methods add a pair to the chain. But the middle two methods also add a callback/errback pair. The addCallback method adds an explicit callback (the one you pass to the method) and an implicit “pass-through” errback. A pass-through function is a dummy function that just returns its first argument. Since the first argument to an errback is always a Failure, a pass-through errback will always “fail” and send its error to the next errback in the chain.

As you’ve no doubt guessed, the addErrback function adds an explicit errback and an implicit pass-through callback. And since the first argument to a callback is never a Failure, a pass-through callback sends its result to the next callback in the chain.

The Deferred Simulator

It’s a good idea to become familiar with the way deferreds fire their callbacks and errbacks. The python script in twisted-deferred/deferred-simulator.py is a “deferred simulator”, a little python program that lets you explore how deferreds fire. When you run the script it will ask you to enter list of callback and errback pairs, one per line. For each callback or errback, you specify that either:

  • It returns a given value (succeds), or
  • It raises a given exception (fails), or
  • It returns its argument (passthru).

After you’ve entered all the pairs you want to simulate, the script will print out, in high-resolution ASCII art, a diagram showing the contents of the chain and the firing patterns for the callback and errback methods. You will want to use a terminal window that is as wide as possible to see everything correctly. You can also use the --narrow option to print the diagrams one after the other, but it’s easier to see their relationships when you print them side-by-side.

Of course, in real code a callback isn’t going to return the same value every time, and a given function might sometimes succeed and other times fail. But the simulator can give you a picture of what will happen for a given combination of normal results and failures, in a given arrangement of callbacks and errbacks.

Summary

After thinking some more about callbacks, we realize that letting callback exceptions bubble up the stack isn’t going to work out so well, since callback programming inverts the usual relationship between low-context and high-context code. And the Deferred class tackles this problem by catching exceptions and sending them down the chain instead of up into the reactor.

We’ve also learned that ordinary results (return values) move down the chain as well. Combining both facts together results in a kind of criss-cross firing pattern as the deferred switches back and forth between the callback and errback lines, depending on the result of each stage.

Armed with this knowledge, in Part 10 we will update our poetry client with some poetry transformation logic.

Suggested Exercises

  1. Inspect the implementation of each of the four methods on the Deferred which add callbacks and errbacks. Verify that all methods add a callback/errback pair.
  2. Use the deferred simulator to investigate the difference between this code:
    deferred.addCallbacks(my_callback, my_errback)

    and this code:

    deferred.addCallback(my_callback)
    deferred.addErrback(my_errback)

    Recall that the last two methods add implicit pass-through functions as one member of the pair.

27 thoughts on “An Introduction to Asynchronous Programming and Twisted”

  1. Wow, your article is so cute! I like your style very much.
    I am from China, I don’t know if I can express my problem exactly.
    But I think you will teach us newbie what happens when you return a deferred from your code.
    How does the reactor bind the deferred you just returned to my code?
    When my nonblocking function finished, where can I find my deferred?
    I am not sure that if other newbies have this puzzle.

    1. Hey there, glad you like the articles. It’s probably easier to see that in Part 8.

      Check out the get-poetry.py client we write there. The get_poetry function
      creates a new deferred and returns it to the caller. After that, the caller
      attaches its own code to the deferred with addCallbacks and addBoth.

      That’s how user code gets connected to deferreds.

      Meanwhile, the get_poetry function also gave the same deferred to the PoetryClientFactory.
      That’s where the reactor comes into play. The reactor (via the transport) is calling the dataReceived
      and connectionLost functions on the PoetryProtocol. Once the connection is closed, the protocol
      tells the factory, and the factory fires the deferred. This all happens during a reactor callback which
      invokes connectionLost.

      Hope that helps,
      dave

  2. hello,Dave.
    Can u give me detailed explaination of this sentence?
    “Doing so raises the original exception we were handling and allows us to take some action on an error without completely handling it.”

    1. Sorry I missed this earlier, some of my wordpress email seems to suddenly be going into my spam folder.

      What I meant was that you can use the raise keyword to re-raise the current exception you are handling.
      This causes the original exception to continue propagating up the stack, as if it were never handled. Does that make sense?

  3. Hi,Dave.
    Maybe this is not about this artical.But I think you could help me.
    when i read Twisted API,i always see the return values or parameters with L{****} or C{***}.
    what’s mean about that?

      1. At first,i guess it indicates class type.
        By the way,i has finished 14th of this articals.And for some reason,i can’t complete the rest ones some days later.And this is the index of my translation.[http://blog.sina.com.cn/s/blog_704b6af70100py9n.html]
        By this,i realized that translation is not a piece of cake,hehe!

        Thank you ,dave!
        I believe you are a kind man!Wish the best to you!

        1. Translation is a lot of work isn’t it? Thanks very much for doing this. Can you tell me the English
          transliteration of your name so I can put it on the index page together with the link?

  4. yeah,a lot of work.It always happens that i has got what you want to mean,but can’t describe them perfectly in chinese .So it needs much time to select better translation again and again.
    yang xiaowei is the chinese spell of 杨晓伟.
    I think,if there many people read my translation,i will complete the rest ones with my rest periods,or i will do it in one holiday in the future.This is my thought,what do you think about?

    1. Sounds good to me! I added the link to your translation to the index page and added a new post about it, so hopefully you will get some traffic.

  5. Is it right to check your condition as the following , Dave :
    r good, p
    f bad, p , this is for d.addCallback(someCallback); d.addErrback(…)

    and
    r good f bad for d.addCallback(someCallback, someErrback).
    What i noticed ,that

  6. Hi!
    Your tutorial is amazing, great work!

    Sorry if I bother you here, I have a question that’s driving me crazy.
    In twisted.mail.smtp, method do_RCPT(), there is this deferred:
    d = defer.maybeDeferred(self.validateTo, user)
    d.addCallbacks(
    self._cbToValidate,
    self._ebToValidate,
    callbackArgs=(user,)
    )

    The method doesn’t return anything, and I couldn’t be able to find any caller of d.callback().
    Despite of this, the callback/errorback functions are indeed called properly.
    What am I missing?

    Sorry, newbie here :)

    Thanks a lot, your guide is invaluable!

    1. Thank you! So maybeDeferred is a helper function for calling a function or method that might return a deferred and might not. If a deferred is returned, then maybeDeferred just returns it. If some other value is returned, then the function was really synchronous and maybeDeferred creates an already-fired deferred with the result and returns that. In the latter case, maybeDeferred actually fires the deferred. In the former, the deferred is fired by some other mechanism set up by the original function (in this case validateTo). So to see why that deferred is firing you will need to start looking at validateTo.

  7. Hi, thanks for your reply! I’ve found what I was missing: the call d.callback(result) in the succeed(result) method. Thanks again!

  8. I tried to learn twisted by looking at examples but finally gave in and used tutorial. I don’t think I could’ve connected the dots otherwise. Thanks for your effort. Took me a bit to understand that there are 3 main components that work together, the protocol, factory, and reactor. Might be good to just lay that out there from the start and their purposes so that as the pieces are introduced it makes obvious sense.

    I don’t quite understand this part though…

    def message_completed(self, message):
    if self.deferred is not None:
    d = self.deferred
    d, self.deferred = self.deferred, None
    d.callback(message)

    In creating my own code I’m opening a connection that just stays open until the program terminates. So first time through a get the message passed back from the server as expected. However, the assignment under the if sets self.deferred = None so on second message the if evaluates false and I get no message passed… why are you assigning None to self.defferred? i know there’s some concept I’m missing here. Please enlighten!

    1. So a Deferred object represents a single asynchronous result. For that reason, Deferreds are single-use objects. They can only be fired once, so once they are fired there is no use keeping them around. For a long-running connection that is going to be sending multiple messages over time, you need another solution, like a message_received callback. Does that make sense?

  9. I think so. So your saying just explicitly pass a callback and errback as you did in the poetry 3.1 example? I’m guessing there’s a good reason but why would deferreds be limited to a single use? Seems kind of short sighted when connections so commonly remain open.

    1. I think because it is a common use case that is easy to get wrong with callbacks. But even with long-running connections, Deferreds are still useful. Think RPC calls, they might re-use the same connection but for a given call there should only be one response.

      Instead of a pair of callbacks I would recommend an object with multiple methods.

    1. Deferreds do not actually cause anything to be asynchronous. “Asynchronous” code is simply code that never makes a blocking I/O call, though in
      Twisted and other async Python frameworks that restriction usually only applies to networking I/O calls since non-blocking disk I/O is not as well supported.

      Since you cannot wait for I/O to finish before returning, single-threaded async systems like Twisted need to use callbacks to signify when I/O is complete. And the Deferred class is simply an abstraction for managing callbacks, nothing more. You can always use callbacks directly and still be asynchronous.

Leave a Reply