An Introduction to Asynchronous Programming and Twisted

Part 12: A Poetry Transformation Server

This continues the introduction started here. You can find an index to the entire series here.

One More Server

Alright, we’ve written one Twisted server so let’s write another, and then we’ll get back to learning some more about Deferreds.

In Parts 9 and 10 we introduced the idea of a poetry transformation engine. The one we eventually implemented, the cummingsifier, was so simple we had to add random exceptions to simulate a failure. But if the transformation engine was located on another server, providing a network “poetry transformation service”, then there is a much more realistic failure mode: the transformation server is down.

So in Part 12 we’re going to implement a poetry transformation server and then, in the next Part, we’ll update our poetry client to use the external transformation service and learn a few new things about Deferreds in the process.

Designing the Protocol

Up till now the interactions between client and server have been strictly one-way. The server sends a poem to the client while the client never sends anything at all to the server. But a transformation service is two-way — the client sends a poem to the server and then the server sends a transformed poem back. So we’ll need to use, or invent, a protocol to handle that interaction.

While we’re at it, let’s allow the server to support multiple kinds of transformations and allow the client to select which one to use. So the client will send two pieces of information: the name of the transformation and the complete text of the poem. And the server will return a single piece of information, namely the text of the transformed poem. So we’ve got a very simple sort of Remote Procedure Call.

Twisted includes support for several protocols we could use to solve this problem, including XML-RPC, Perspective Broker, and AMP.

But introducing any of these full-featured protocols would require us to go too far afield, so we’ll roll our own humble protocol instead. Let’s have the client send a string of the form (without the angle brackets):

<transform-name>.<text of the poem>

That’s just the name of the transform, followed by a period, followed by the complete text of the poem itself. And we’ll encode the whole thing in the form of a netstring. And the server will send back the text of the transformed poem, also in a netstring. Since netstrings use length-encoding, the client will be able to detect the case where the server fails to send back a complete result (maybe it crashed in the middle of the operation). If you recall, our original poetry protocol has trouble detecting aborted poetry deliveries.

So much for the protocol design. It’s not going to win any awards, but it’s good enough for our purposes.

The Code

Let’s look at the code of our transformation server, located in twisted-server-1/transformedpoetry.py. First, we define a TransformService class:

class TransformService(object):

    def cummingsify(self, poem):
        return poem.lower()

The transform service currently implements one transformation, cummingsify, via a method of the same name. We could add additional algorithms by adding additional methods. Here’s something important to notice: the transformation service is entirely independent of the particular details of the protocol we settled on earlier. Separating the protocol logic from the service logic is a common pattern in Twisted programming. Doing so makes it easy to provide the same service via multiple protocols without duplicating code.

Now let’s look at the protocol factory (we’ll look at the protocol right after):

class TransformFactory(ServerFactory):

    protocol = TransformProtocol

    def __init__(self, service):
        self.service = service

    def transform(self, xform_name, poem):
        thunk = getattr(self, 'xform_%s' % (xform_name,), None)

        if thunk is None: # no such transform
            return None

        try:
            return thunk(poem)
        except:
            return None # transform failed

    def xform_cummingsify(self, poem):
        return self.service.cummingsify(poem)

This factory provides a transform method which a protocol instance can use to request a poetry transformation on behalf of a connected client. The method returns None if there is no such transformation or if the transformation fails. And like the TransformService, the protocol factory is independent of the wire-level protocol, the details of which are delegated to the protocol class itself.

One thing to notice is the way we guard access to the service though the xform_-prefixed methods. This is a pattern you will find in the Twisted sources, although the prefixes vary and they are usually on an object separate from the factory. It’s one way of preventing client code from executing an arbitrary method on the service object, since the client can send any transform name they want. It also provides a place to perform protocol-specific adaptation to the API provided by the service object.

Now we’ll take a look at the protocol implementation:

class TransformProtocol(NetstringReceiver):

    def stringReceived(self, request):
        if '.' not in request: # bad request
            self.transport.loseConnection()
            return

        xform_name, poem = request.split('.', 1)

        self.xformRequestReceived(xform_name, poem)

    def xformRequestReceived(self, xform_name, poem):
        new_poem = self.factory.transform(xform_name, poem)

        if new_poem is not None:
            self.sendString(new_poem)

        self.transport.loseConnection()

In the protocol implementation we take advantage of the fact that Twisted supports netstrings via the NetstringReceiver protocol. That base class takes care of decoding (and encoding) the netstrings and all we have to do is implement the stringReceived method. In other words, stringReceived is called with the content of a netstring sent by the client, without the extra bytes added by the netstring encoding. The base class also takes care of buffering the incoming bytes until we have enough to decode a complete string.

If everything goes ok (and if it doesn’t we just close the connection) we send the transformed poem back to the client using the sendString method provided by NetstringReceiver (and which ultimately calls transport.write()). And that’s all there is to it. We won’t bother listing the main function since it’s similar to the ones we’ve seen before.

Notice how we continue the Twisted pattern of translating the incoming byte stream to higher and higher levels of abstraction by defining the xformRequestReceived method, which is passed the name of the transform and the poem as two separate arguments.

A Simple Client

We’ll implement a Twisted client for the transformation service in the next Part. For now we’ll just make do with a simple script located in twisted-server-1/transform-test. It uses the netcat program to send a poem to the server and then prints out the response (which will be encoded as a netstring). Let’s say you run the transformation server on port 11000 like this:

python twisted-server-1/transformedpoetry.py --port 11000

Then you could run the test script against that server like this:

./twisted-server-1/transform-test 11000

And you should see some output like this:

15:here is my poem,

That’s the netstring-encoded transformed poem (the original is in all upper case).

Discussion

We introduced a few new ideas in this Part:

  1. Two-way communication.
  2. Building on an existing protocol implementation provided by Twisted.
  3. Using a service object to separate functional logic from protocol logic.

The basic mechanics of two-way communication are simple. We used the same techniques for reading and writing data in previous clients and servers; the only difference is we used them both together. Of course, a more complex protocol will require more complex code to process the byte stream and format outgoing messages. And that’s a great reason to use an existing protocol implementation like we did today.

Once you start getting comfortable writing basic protocols, it’s a good idea to take a look at the different protocol implementations provided by Twisted. You might start by perusing the twisted.protocols.basic module and going from there. Writing simple protocols is a great way to familiarize yourself with the Twisted style of programming, but in a “real” program it’s probably a lot more common to use a ready-made implementation, assuming there is one available for the protocol you want to use.

The last new idea we introduced, the use of a Service object to separate functional and protocol logic, is a really important design pattern in Twisted programming. Although the service object we made today is trivial, you can imagine a more realistic network service could be quite complex. And by making the Service independent of protocol-level details, we can quickly provide the same service on a new protocol without duplicating code.

Figure 27 shows a transformation server that is providing poetry transformations via two different protocols (the version of the server we presented above only has one protocol):

Figure 27: a transformation server with two protocols

Figure 27: a transformation server with two protocols

Although we need two separate protocol factories in Figure 27, they might differ only in their protocol class attribute and would be otherwise identical. The factories would share the same Service object and only the Protocols themselves would require separate implementations. Now that’s code re-use!

Looking Ahead

So much for our transformation server. In Part 13, we’ll update our poetry client to use the transform server instead of implementing transformations in the client itself.

Suggested Exercises

  1. Read the source code for the NetstringReceiver class. What happens if the client sends a malformed netstring? What happens if the client tries to send a huge netstring?
  2. Invent another transformation algorithm and add it to the transformation service and the protocol factory. Test it out by modifying the netcat client.
  3. Invent another protocol for requesting poetry transformations and modify the server to handle both protocols (on two different ports). Use the same instance of the TransformService for both.
  4. How would the code need to change if the methods on the TransformService were asynchronous (i.e., they returned Deferreds)?
  5. Write a synchronous client for the transformation server.
  6. Update the original client and server to use netstrings when sending poetry.

21 thoughts on “An Introduction to Asynchronous Programming and Twisted”

  1. Hi,

    Great tutorial, it should be first part of Twisted documentation ;) I have a question about code re-use when implementing UDP services. There are no factories for DatagramProtocol, so how should it be organized if, let’s say, we’d like to implement Poetry Transformation Server based on UDP protocol?

    Thanks in advance!

    1. Thank you! That’s an intriguing question :) I think you would only be able to re-use the Service object for a UDP protocol. As you pointed out, there are no factories involved when you listen on UDP ports with Twisted (since there are no connections, just datagram packets). So the Protocol object you pass to listenUDP would have a reference to the Service object. And the Protocol itself would have to be rewritten entirely, as you would need to re-implement some of the features that TCP gives you, like order preservation and guaranteed delivery.

  2. I was not able to get any output from the netcat script ./twisted-server-1/transform-test

    Using Wireshark I could see the poem was actually returned in lower case.

    It seems that netcat is exiting before the transformed poem has been received.

    Using netcat with option -q -1 (-q secs quit after EOF on stdin and delay of secs (-1 to not quit)) it works.

    This is on Ubuntu 10.10 / Intel Atom CPU 330@1.60GHz.

    Thanks for the excellent tutorials!

  3. Is it possible to change protocol class during connection? My scenario is something like this: video player connects to 554 port. On this port there might be normal RTSP service or RTSP-over-HTTP thingy. I can discover this after first request from client (he will call me with HTTP GET or RTSP DESCRIBE). Handling both protocols in single class kinda sux. Any ideas? ;)

    1. What you can do is provide a protocol that starts forwarding its calls to another protocol after the original
      protocol decides what to do. See twisted.protocols.policies.ProtocolWrapper for an example of what I mean,
      though that’s doing a little more than I think you need (providing ITransport calls in addition to IProtocol) and
      also a little less (it’s not switching to the wrapped protocol after it first figures out what the next protocol will be).

      See twisted.protocols.ftp.ProtocolWrapper for a similar idea.

  4. Hi Dave,
    Great tutorial. I have managed to reach part 12. I am trying to implement the transform server on my own. In my transform server, every time there is a protocol error i.e. ‘.’ not sent in the request, I call transport.loseConnection(). The reactor returns an error code: “The connection was closed in n non-clean fashion”. Am I forgetting to do something here? Is there a clean way to close the socket from the server?

    1. I call the disconnect code after I make the check for a ‘.’ in the received message.
      def disconnect(self):
      self.logger.warning(“Disconnecting”)
      self.sessionState = ‘UNBOUND’
      self.transport.loseConnection()

      1. Ok, thanks, but I actually meant the whole thing, so I can run it myself to recreate
        the error message. Could you post it as a github gist perhaps?

        1. It actually turned out to be something unrelated to the code. There was a firewall interfering with the code. Resolved when I tested on a personal computer.

  5. I saw this PyCon demo on cooperative multitasking using twisted and it made me wonder:
    Let’s say the cummingsify() method actually has a lot of lines of code and takes longer to execute (but no IO involved). Does it make sense for the server app to yield to the reactor in the middle of execution or return only after transformation is finished? In a single threaded context, is there any use case where it is better to let the reactor service another transform request before the current one is complete?

    I am trying to understand if it is okay to have synchronous CPU bound code run to completion? If the performance is unacceptable what are the ways out in the reactor model?

    1. Yes, you raise a very good point. A cpu-bound task is going to
      block the reactor just like a blocking I/O call would. If such
      tasks are long and frequent enough then they could definitely
      reduce the performance of your server. At that point you have
      a few options:

      • Break up the computation into multiple steps, as you said above.
      • Run the computation in another thread. Twisted has some support for this in the deferToThread utility API.
      • Run the computation in another process.
  6. Excellent tutorial. I was cruising along pretty well but I’m having a heck of a time with example #4. I’m not wrapping my brain around how to make the methods on TransformService (ex: cummingsify()) return a Deferred and also actually do the transformation work, too, asynchronously. I’m failing to understand how to kick off the transformation but return the Deferred before it’s complete. The only solutions I can come up with using Deferreds are, I think, effectively the same in terms of synchronicity as the original. Can you give me any guidance on this? Thanks!

    1. Hey, glad you like the tutorial. So #4 is really more of a though experiment (just imagine .transform() returned a deferred — how would the main service need to change?). But one way to make a function return a deferred that
      fires later is with a nifty Twisted utility api: twisted.internet.task.deferLater. You call it like:

      d = deferLater(reactor, delay, function, *args, **kw)

      The first argument is the reactor object, and the second is the number of seconds to delay.
      After the delay, Twisted will call the given function with the given arguments and fire the
      deferred you got back from deferLater with the result.

      Make sense?

  7. That does make sense. I’m still having some trouble putting it together into the transformation server the way I thought I might, but that’s okay. What I’m doing with the exercise at this point is fairly contrived and pointless anyway; just seeing if I could do it. Your future Parts and the project I ultimately will use these skills on will guide me when it really counts. Thanks!

  8. As of GNU netcat 0.7, -q option is not available.
    -n option for echo seems to prevent transform-test from sending correct data. I’m using Twisted 12.3.0 and bash 4 on Mac OS 10.8.2

Leave a Reply