Poetry Transformed

Part 10: Poetry Transformed

This continues the introduction started here. You can find an index to the entire series here.

Client 5.0

Now we’re going to add some transformation logic to our poetry client, along the lines suggested in Part 9. But first, I have a shameful and humbling confession to make: I don’t know how to write the Byronification Engine. It is beyond my programming abilities. So instead, I’m going to implement a simpler transformation, the Cummingsifier. The Cummingsifier is an algorithm that takes a poem and returns a new poem like the original but written in the style of e.e. cummings. Here is the Cummingsifier algorithm in its entirety:

def cummingsify(poem)
    return poem.lower()

Unfortunately, this algorithm is so simple it never actually fails, so in client 5.0, located in twisted-client-5/get-poetry.py, we use a modified version of cummingsify that randomly does one of the following:

1. Return a cummingsified version of the poem.
2. Raise a GibberishError.
3. Raise a ValueError.

In this way we simulate a more complicated algorithm that sometimes fails in unexpected ways.

The only other changes in client 5.0 are in the poetry_main function:

def poetry_main():
    addresses = parse_args()

    from twisted.internet import reactor

    poems = []
    errors = []

    def try_to_cummingsify(poem):
        try:
            return cummingsify(poem)
        except GibberishError:
            raise
        except:
            print 'Cummingsify failed!'
            return poem

    def got_poem(poem):
        print poem
        poems.append(poem)

    def poem_failed(err):
        print >>sys.stderr, 'The poem download failed.'
        errors.append(err)

    def poem_done(_):
        if len(poems) + len(errors) == len(addresses):
            reactor.stop()

    for address in addresses:
        host, port = address
        d = get_poetry(host, port)
        d.addCallback(try_to_cummingsify)
        d.addCallbacks(got_poem, poem_failed)
        d.addBoth(poem_done)

    reactor.run()

So when the program downloads a poem from the server, it will either:

1. Print the cummingsified (lower-cased) version of the poem.
2. Print “Cummingsify failed!” followed by the original poem.
3. Print “The poem download failed.”

Although we have retained the ability to download from multiple servers, when you are testing out client 5.0 it’s easier to just use a single server and run the program multiple times, until you see all three different outcomes. Also try running the client on a port with no server.

Let’s draw the callback/errback chain we create on each Deferred we get back from get_poetry:

Note the pass-through errback that gets added by addCallback. It passes whatever Failure it receives onto the next errback (poem_failed). Thus, poem_failed can handle failures from both get_poetry (i.e., the deferred is fired with the errback method) and the cummingsify function.

Also note the hauntingly beautiful drop-shadow around the border of the deferred in Figure 19. It doesn’t signify anything other than me discovering how to do it in Inkscape. Expect more drop-shadows in the future.

Let’s analyze the different ways our deferred can fire. The case where we get a poem and the cummingsify function works correctly is shown in Figure 20:

In this case no callback fails, so control flows down the callback line. Note that poem_done receives None as its result, since got_poem doesn’t actually return a value. If we wanted subsequent callbacks to have access to the poem, we would modify got_poem to return the poem explicitly.

Figure 21 shows the case where we get a poem, but cummingsify raises a GibberishError:

Since the try_to_cummingsify callback re-raises a GibberishError, control switches to the errback line and poem_failed is called with the exception as its argument (wrapped in a Failure, of course).

And since poem_failed doesn’t raise an exception, or return a Failure, after it is done control switches back to the callback line. If we want poem_failed to handle the error completely, then returning None is a reasonable behavior. On the other hand, if we wanted poem_failed to take some action, but still propagate the error, we could change poem_failed to return its err argument and processing would continue down the errback line.

Note that in the current code neither got_poem nor poem_failed ever fail themselves, so the poem_done errback will never be called. But it’s safe to add it in any case and doing so represents an instance of “defensive” programming, as either got_poem or poem_failed might have bugs we don’t know about. Since the addBoth method ensures that a particular function will run no matter how the deferred fires, using addBoth is analogous to adding a finally clause to a try/except statement.

Now examine the case where we download a poem and the cummingsify function raises a ValueError, displayed in Figure 22:

This is the same as figure 20, except got_poem receives the original version of the poem instead of the transformed version. The switch happens entirely inside the try_to_cummingsify callback, which traps the ValueError with an ordinary try/except statement and returns the original poem instead. The deferred object never sees that error at all.

Lastly, we show the case where we try to download a poem from a non-existent server in Figure 23:

As before, poem_failed returns None so afterwards control switches to the callback line.

Client 5.1

In client 5.0 we are trapping exceptions from cummingsify in our try_to_cummingsify callback using an ordinary try/except statement, rather than letting the deferred catch them first. There isn’t necessarily anything wrong with this strategy, but it’s instructive to see how we might do this differently.

Let’s suppose we wanted to let the deferred catch both GibberishError and ValueError exceptions and send them to the errback line. To preserve the current behavior our subsequent errback needs to check to see if the error is a ValueError and, if so, handle it by returning the original poem, so that control goes back to the callback line and the original poem gets printed out.

But there’s a problem: the errback wouldn’t get the original poem, it would get the Failure-wrapped ValueError raised by the cummingsify function. To let the errback handle the error, we need to arrange for it to receive the original poem.

One way to do that is to modify the cummingsify function so the original poem is included in the exception. That’s what we’ve done in client 5.1, located in twisted-client-5/get-poetry-1.py. We changed the ValueError exception into a custom CannotCummingsify exception which takes the original poem as the first argument.

If cummingsify were a real function in an external module, then it would probably be best to wrap it with another function that trapped any exception that wasn’t GibberishError and raise a CannotCummingsify exception instead. With this new setup, our poetry_main function looks like this:

def poetry_main():
    addresses = parse_args()

    from twisted.internet import reactor

    poems = []
    errors = []

    def cummingsify_failed(err):
        if err.check(CannotCummingsify):
            print 'Cummingsify failed!'
            return err.value.args[0]
        return err

    def got_poem(poem):
        print poem
        poems.append(poem)

    def poem_failed(err):
        print >>sys.stderr, 'The poem download failed.'
        errors.append(err)

    def poem_done(_):
        if len(poems) + len(errors) == len(addresses):
            reactor.stop()

    for address in addresses:
        host, port = address
        d = get_poetry(host, port)
        d.addCallback(cummingsify)
        d.addErrback(cummingsify_failed)
        d.addCallbacks(got_poem, poem_failed)
        d.addBoth(poem_done)

And each deferred we create has the structure pictured in Figure 24:

Examine the cummingsify_failed errback:

    def cummingsify_failed(err):
        if err.check(CannotCummingsify):
            print 'Cummingsify failed!'
            return err.value.args[0]
        return err

We are using the check method on Failure objects to test whether the exception embedded in the Failure is an instance of CannotCummingsify. If so, we return the first argument to the exception (the original poem) and thus handle the error. Since the return value is not a Failure, control returns to the callback line. Otherwise, we return the Failure itself and send (re-raise) the error down the errback line. As you can see, the exception is available as the value attribute on the Failure.

Figure 25 shows what happens when we get a CannotCummingsify exception:

So when we are using a deferred, we can sometimes choose whether we want to use try/except statements to handle exceptions, or let the deferred re-route errors to an errback.

Summary

In Part 10 we updated our poetry client to make use of the Deferred‘s ability to route errors and results down the chain. Although the example was rather artificial, it did illustrate how control flow in a deferred switches back and forth between the callback and errback line depending on the result of each stage.

So now we know everything there is to know about deferreds, right? Not yet! We’re going to explore some more features of deferreds in a future Part. But first we’ll take a little detour and, in Part 11, implement a Twisted version of our poetry server.

Suggested Exercises

1. Figure 25 shows one of the four possible ways the deferreds in client 5.1 can fire. Draw the other three.
2. Use the deferred simulator to simulate all possible firings for clients 5.0 and 5.1. To get you started, this simulator program can represent the case where the try_to_cummingsify function succeeds in client 5.0:

   r poem p
   r None r None
   r None r None