krondo

Part 17: Just Another Way to Spell “Callback”

This continues the introduction started here. You can find an index to the entire series here.

Introduction

In this Part we’re going to return to the subject of callbacks. We’ll introduce another technique for writing callbacks in Twisted that uses generators. We’ll show how the technique works and contrast it with using “pure” Deferreds. Finally we’ll rewrite one of our poetry clients using this technique. But first let’s review how generators work so we can see why they are a candidate for creating callbacks.

A Brief Review of Generators

As you probably know, a Python generator is a “restartable function” that you create by using the yield expression in the body of your function. By doing so, the function becomes a “generator function” that returns an iterator you can use to run the function in a series of steps. Each cycle of the iterator restarts the function, which proceeds to execute until it reaches the next yield.

Generators (and iterators) are often used to represent lazily-created sequences of values. Take a look at the example code in inline-callbacks/gen-1.py:

def my_generator():
    print 'starting up'
    yield 1
    print "workin'"
    yield 2
    print "still workin'"
    yield 3
    print 'done'

for n in my_generator():
    print n

Here we have a generator that creates the sequence 1, 2, 3. If you run the code, you will see the print statements in the generator interleaved with the print statement in the for loop as the loop cycles through the generator.

We can make this code more explicit by creating the generator ourselves (inline-callbacks/gen-2.py):

def my_generator():
    print 'starting up'
    yield 1
    print "workin'"
    yield 2
    print "still workin'"
    yield 3
    print 'done'

gen = my_generator()

while True:
    try:
        n = gen.next()
    except StopIteration:
        break
    else:
        print n

Considered as a sequence, the generator is just an object for getting successive values. But we can also view things from the point of view of the generator itself:

1. The generator function doesn’t start running until “called” by the loop (using the next method).
2. Once the generator is running, it keeps running until it “returns” to the loop (using yield).
3. When the loop is running other code (like the print statement), the generator is not running.
4. When the generator is running, the loop is not running (it’s “blocked” waiting for the generator).
5. Once a generator yields control to the loop, an arbitrary amount of time may pass (and an arbitrary amount of other code may execute) until the generator runs again.

This is very much like the way callbacks work in an asynchronous system. We can think of the while loop as the reactor, and the generator as a series of callbacks separated by yield statements, with the interesting fact that all the callbacks share the same local variable namespace, and the namespace persists from one callback to the next.

Furthermore, you can have multiple generators active at once (see the example in inline-callbacks/gen-3.py), with their “callbacks” interleaved with each other, just as you can have independent asynchronous tasks running in a system like Twisted.

Something is still missing, though. Callbacks aren’t just called by the reactor, they also receive information. When part of a deferred’s chain, a callback either receives a result, in the form of a single Python value, or an error, in the form of a Failure.

Starting with Python 2.5, generators were extended in a way that allows you to send information to a generator when you restart it, as illustrated in inline-callbacks/gen-4.py:

class Malfunction(Exception):
    pass

def my_generator():
    print 'starting up'

    val = yield 1
    print 'got:', val

    val = yield 2
    print 'got:', val

    try:
        yield 3
    except Malfunction:
        print 'malfunction!'

    yield 4

    print 'done'

gen = my_generator()

print gen.next() # start the generator
print gen.send(10) # send the value 10
print gen.send(20) # send the value 20
print gen.throw(Malfunction()) # raise an exception inside the generator

try:
    gen.next()
except StopIteration:
    pass

In Python 2.5 and later versions, the yield statement is an expression that evaluates to a value. And the code that restarts the generator can determine that value using the send method instead of next (if you use next the value is None). What’s more, you can actually raise an arbitrary exception inside the generator using the throw method. How cool is that?

Inline Callbacks

Given what we just reviewed about sending and throwing values and exceptions into a generator, we can envision a generator as a series of callbacks, like the ones in a deferred, which receive either results or failures. The callbacks are separated by yields and the value of each yield expression is the result for the next callback (or the yield raises an exception and that’s the failure). Figure 35 shows the correspondence:

Now when a series of callbacks is chained together in a deferred, each callback receives the result from the one prior. That’s easy enough to do with a generator — just send the value you got from the previous run of the generator (the value it yielded) the next time you restart it. But that also seems a bit silly. Since the generator computed the value to begin with, why bother sending it back? The generator could just save the value in a variable for the next time it’s needed. So what’s the point?

Recall the fact we learned in Part 13, that the callbacks in a deferred can return deferreds themselves. And when that happens, the outer deferred is paused until the inner deferred fires, and then the next callback (or errback) in the outer deferred’s chain is called with the result (or failure) from the inner deferred.

So imagine that our generator yields a deferred object instead of an ordinary Python value. The generator is now “paused”, and that’s automatic; generators always pause after every yield statement until they are explicitly restarted. So we can delay restarting the generator until the deferred fires, at which point we either send the value (if the deferred succeeds) or throw the exception (if the deferred fails). That would make our generator a genuine sequence of asynchronous callbacks and that’s the idea behind the inlineCallbacks function in twisted.internet.defer.

inlineCallbacks

Consider the example program in inline-callbacks/inline-callbacks-1.py:

from twisted.internet.defer import inlineCallbacks, Deferred

@inlineCallbacks
def my_callbacks():
    from twisted.internet import reactor

    print 'first callback'
    result = yield 1 # yielded values that aren't deferred come right back

    print 'second callback got', result
    d = Deferred()
    reactor.callLater(5, d.callback, 2)
    result = yield d # yielded deferreds will pause the generator

    print 'third callback got', result # the result of the deferred

    d = Deferred()
    reactor.callLater(5, d.errback, Exception(3))

    try:
        yield d
    except Exception, e:
        result = e

    print 'fourth callback got', repr(result) # the exception from the deferred

    reactor.stop()

from twisted.internet import reactor
reactor.callWhenRunning(my_callbacks)
reactor.run()

Run the example and you will see the generator execute to the end and then stop the reactor. The example illustrates several aspects of the inlineCallbacks function. First, inlineCallbacks is a decorator and it always decorates generator functions, i.e., functions that use yield. The whole purpose of inlineCallbacks is turn a generator into a series of asynchronous callbacks according to the scheme we outlined before.

Second, when we invoke an inlineCallbacks-decorated function, we don’t need to call next or send or throw ourselves. The decorator takes care of those details for us and ensures the generator will run to the end (assuming it doesn’t raise an exception).

Third, if we yield a non-deferred value from the generator, it is immediately restarted with that same value as the result of the yield.

And finally, if we yield a deferred from the generator, it will not be restarted until that deferred fires. If the deferred succeeds, the result of the yield is just the result from the deferred. And if the deferred fails, the yield statement raises the exception. Note the exception is just an ordinary Exception object, rather than a Failure, and we can catch it with a try/except statement around the yield expression.

In the example we are just using callLater to fire the deferreds after a short period of time. While that’s a handy way to put in a non-blocking delay into our callback chain, normally we would be yielding a deferred returned by some other asynchronous operation (i.e., get_poetry) invoked from our generator.

Ok, now we know how an inlineCallbacks-decorated function runs, but what return value do you get if you actually call one? As you might have guessed, you get a deferred. Since we can’t know exactly when that generator will stop running (it might yield one or more deferreds), the decorated function itself is asynchronous and a deferred is the appropriate return value. Note the deferred that is returned isn’t one of the deferreds the generator may yield. Rather, it’s a deferred that fires only after the generator has completely finished (or throws an exception).

If the generator throws an exception, the returned deferred will fire its errback chain with that exception wrapped in a Failure. But if we want the generator to return a normal value, we must “return” it using the defer.returnValue function. Like the ordinary return statement, it will also stop the generator (it actually raises a special exception). The inline-callbacks/inline-callbacks-2.py example illustrates both possibilities.

Client 7.0

Let’s put inlineCallbacks to work with a new version of our poetry client. You can see the code in twisted-client-7/get-poetry.py. You may wish to compare it to client 6.0 in twisted-client-6/get-poetry.py. The relevant changes are in poetry_main:

def poetry_main():
    addresses = parse_args()

    xform_addr = addresses.pop(0)

    proxy = TransformProxy(*xform_addr)

    from twisted.internet import reactor

    results = []

    @defer.inlineCallbacks
    def get_transformed_poem(host, port):
        try:
            poem = yield get_poetry(host, port)
        except Exception, e:
            print >>sys.stderr, 'The poem download failed:', e
            raise

        try:
            poem = yield proxy.xform('cummingsify', poem)
        except Exception:
            print >>sys.stderr, 'Cummingsify failed!'

        defer.returnValue(poem)

    def got_poem(poem):
        print poem

    def poem_done(_):
        results.append(_)
        if len(results) == len(addresses):
            reactor.stop()

    for address in addresses:
        host, port = address
        d = get_transformed_poem(host, port)
        d.addCallbacks(got_poem)
        d.addBoth(poem_done)

    reactor.run()

In our new version the inlineCallbacks generator function get_transformed_poem is responsible for both fetching the poem and then applying the transformation (via the transform service). Since both operations are asynchronous, we yield a deferred each time and then (implicitly) wait for the result. As in client 6.0, if the transformation fails we just return the original poem. Notice we can use try/except statements to handle asynchronous errors inside the generator.

We can test the new client out in the same way as before. First start up a transform server:

python twisted-server-1/tranformedpoetry.py --port 10001

Then start a couple of poetry servers:

python twisted-server-1/fastpoetry.py --port 10002 poetry/fascination.txt
python twisted-server-1/fastpoetry.py --port 10003 poetry/science.txt

Now you can run the new client:

python twisted-client-7/get-poetry.py 10001 10002 10003

Try turning off one or more of the servers to see how the client handles errors.

Discussion

Like the Deferred object, the inlineCallbacks function gives us a new way of organizing our asynchronous callbacks. And, as with deferreds, inlineCallbacks doesn’t change the rules of the game. Specifically, our callbacks still run one at a time, and they are still invoked by the reactor. We can confirm that fact in our usual way by printing out a traceback from an inline callback, as in the example script inline-callbacks/inline-callbacks-tb.py. Run that code and you will get a traceback with reactor.run() at the top, lots of helper functions in between, and our callback at the bottom.

We can adapt Figure 29, which explains what happens when one callback in a deferred returns another deferred, to show what happens when an inlineCallbacks generator yields a deferred. See Figure 36:

The same figure works in both cases because the idea being illustrated is the same — one asynchronous operation is waiting for another.

Since inlineCallbacks and deferreds solve many of the same problems, why choose one over the other? Here are some potential advantages of inlineCallbacks:

• Since the callbacks share a namespace, there is no need to pass extra state around.
• The callback order is easier to see, as they just execute from top to bottom.
• With no function declarations for individual callbacks and implicit flow-control, there is generally less typing.
• Errors are handled with the familiar try/except statement.

And here are some potential pitfalls:

• The callbacks inside the generator cannot be invoked individually, which could make code re-use difficult. With a deferred, the code constructing the deferred is free to add arbitrary callbacks in an arbitrary order.
• The compact form of a generator can obscure the fact that an asynchronous callback is even involved. Despite its visually similar appearance to an ordinary sequential function, a generator behaves in a very different manner. The inlineCallbacks function is not a way to avoid learning the asynchronous programming model.

As with any technique, practice will provide the experience necessary to make an informed choice.

Summary

In this Part we learned about the inlineCallbacks decorator and how it allows us to express a sequence of asynchronous callbacks in the form of a Python generator.

In Part 18 we will learn a technique for managing a set of “parallel” asynchronous operations.

Suggested Exercises

1. Why is the inlineCallbacks function plural?
2. Study the implementation of inlineCallbacks and its helper function _inlineCallbacks. Ponder the phrase “the devil is in the details”.
3. How many callbacks are contained in a generator with N yield statements, assuming it has no loops or if statements?
4. Poetry client 7.0 might have three generators running at once. Conceptually, how many different ways might they be interleaved with one another? Considering the way they are invoked in the poetry client and the implementation of inlineCallbacks, how many ways do you think are actually possible?
5. Move the got_poem callback in client 7.0 inside the generator.
6. Then move the poem_done callback inside the generator. Be careful! Make sure to handle all the failure cases so the reactor gets shutdown no matter what. How does the resulting code compare to using a deferred to shutdown the reactor?
7. A generator with yield statements inside a while loop can represent a conceptually infinite sequence. What does such a generator decorated with inlineCallbacks represent?

Packer reviews the last century or so of liberalism through the biographies of his father and grandfather.

Today I made my blog my main website, since the old one was getting kind of crufty. I moved the stuff I wanted to keep into WordPress, including my collection of links to programmer’s editors. The old /blog URLs will continue to work.

Wonderful.

This book is my favorite sort of history: an exploration of the minds and viewpoints of the people who lived it. The medieval world-view is so alien to us, and Huizinga lets us glimpse it.

I watched a 1-D film yesterday. Good acting, but I thought the plot was too linear.

Pirenne’s posthumous work where he advances the thesis that Medieval Europe was made possible by the rise of Islam, which severed many of the East-West ties that kept the West under the influence of Byzantium.

Part 16: Twisted Daemonologie

This continues the introduction started here. You can find an index to the entire series here.

Introduction

The servers we have written so far have just run in a terminal window, with output going to the screen via print statements. This works alright for development, but it’s hardly a way to deploy services in production. A well-behaved production server ought to:

1. Run as a daemon process, unconnected with any terminal or user session. You don’t want a service to shut down just because the administrator logs out.
2. Send debugging and error output to a set of rotated log files, or to the syslog service.
3. Drop excessive privileges, e.g., switching to a lower-privileged user before running.
4. Record its pid in a file so that the administrator can easily send signals to the daemon.

We can get all of those features by using the twistd script provided by Twisted. But first we’ll have to change our code a bit.

The Concepts

Understanding twistd will require learning a few new concepts in Twisted, the most important being a Service. As usual, several of the new concepts are accompanied by new Interfaces.

IService

The IService interface defines a named service that can be started and stopped. What does the service do? Whatever you like — rather than define the specific function of the service, the interface requires only that it provide a small set of generic attributes and methods.

There are two required attributes: name and running. The name attribute is just a string, like 'fastpoetry'. The running attribute is a Boolean value and is true if the service has been successfully started.

We’re only going to touch on some of the methods of IService. We’ll skip some that are obvious, and others that are more advanced and often go unused in simpler Twisted programs. The two principle methods of IService are startService and stopService:

    def startService():
        """
        Start the service.
        """

    def stopService():
        """
        Stop the service.

        @rtype: L{Deferred}
        @return: a L{Deferred} which is triggered when the service has
            finished shutting down. If shutting down is immediate, a
            value can be returned (usually, C{None}).
        """

Again, what these methods actually do will depend on the service in question. For example, the startService method might:

• Load some configuration data, or
• Initialize a database, or
• Start listening on a port, or
• Do nothing at all.

And the stopService method might:

• Persist some state, or
• Close open database connections, or
• Stop listening on a port, or
• Do nothing at all.

When we write our own custom services we’ll need to implement these methods appropriately. For some common behaviors, like listening on a port, Twisted provides ready-made services we can use instead.

Notice that stopService may optionally return a deferred, which is required to fire when the service has completely shut down. This allows our services to finish cleaning up after themselves before the entire application terminates. If your service shuts down immediately you can just return None instead of a deferred.

Services can be organized into collections that get started and stopped together. The last IService method we’re going to look at, setServiceParent, adds a Service to a collection:

    def setServiceParent(parent):
        """
        Set the parent of the service.

        @type parent: L{IServiceCollection}
        @raise RuntimeError: Raised if the service already has a parent
            or if the service has a name and the parent already has a child
            by that name.
        """

Any service can have a parent, which means services can be organized in a hierarchy. And that brings us to the next Interface we’re going to look at today.

IServiceCollection

The IServiceCollection interface defines an object which can contain IService objects. A service collection is a just plain container class with methods to:

• Look up a service by name (getServiceNamed)
• Iterate over the services in the collection (__iter__)
• Add a service to the collection (addService)
• Remove a service from the collection (removeService)

Note that an implementation of IServiceCollection isn’t automatically an implementation of IService, but there’s no reason why one class can’t implement both interfaces (and we’ll see an example of that shortly).

Application

A Twisted Application is not defined by a separate interface. Rather, an Application object is required to implement both IService and IServiceCollection, as well as a few other interfaces we aren’t going to cover.

An Application is the top-level service that represents your entire Twisted application. All the other services in your daemon will be children (or grandchildren, etc.) of the Application object.

It is rare to actually implement your own Application. Twisted provides an implementation that we’ll use today.

Twisted Logging

Twisted includes its own logging infrastructure in the module twisted.python.log. The basic API for writing to the log is simple, so we’ll just include a short example located in basic-twisted/log.py, and you can skim the Twisted module for details if you are interested.

We won’t bother showing the API for installing logging handlers, since twistd will do that for us.

FastPoetry 2.0

Alright, let’s look at some code. We’ve updated the fast poetry server to run with twistd. The source is located in twisted-server-3/fastpoetry.py. First we have the poetry protocol:

class PoetryProtocol(Protocol):

    def connectionMade(self):
        poem = self.factory.service.poem
        log.msg('sending %d bytes of poetry to %s'
                % (len(poem), self.transport.getPeer()))
        self.transport.write(poem)
        self.transport.loseConnection()

Notice instead of using a print statement, we’re using the twisted.python.log.msg function to record each new connection.
Here’s the factory class:

class PoetryFactory(ServerFactory):

    protocol = PoetryProtocol

    def __init__(self, service):
        self.service = service

As you can see, the poem is no longer stored on the factory, but on a service object referenced by the factory. Notice how the protocol gets the poem from the service via the factory. Finally, here’s the service class itself:

class PoetryService(service.Service):

    def __init__(self, poetry_file):
        self.poetry_file = poetry_file

    def startService(self):
        service.Service.startService(self)
        self.poem = open(self.poetry_file).read()
        log.msg('loaded a poem from: %s' % (self.poetry_file,))

As with many other Interface classes, Twisted provides a base class we can use to make our own implementations, with helpful default behaviors. Here we use the twisted.application.service.Service class to implement our PoetryService.

The base class provides default implementations of all required methods, so we only need to implement the ones with custom behavior. In this case, we just override startService to load the poetry file. Note we still call the base class method (which sets the running attribute for us).

Another point is worth mentioning. The PoetryService object doesn’t know anything about the details of the PoetryProtocol. The service’s only job is to load the poem and provide access to it for any object that might need it. In other words, the PoetryService is entirely concerned with the higher-level details of providing poetry, rather than the lower-level details of sending a poem down a TCP connection. So this same service could be used by another protocol, say UDP or XML-RPC. While the benefit is rather small for our simple service, you can imagine the advantage for a more realistic service implementation.

If this were a typical Twisted program, all the code we’ve looked at so far wouldn’t actually be in this file. Rather, it would be in some other module(s) (perhaps fastpoetry.protocol and fastpoetry.service). But following our usual practice of making these examples self-contained, we’ve including everything we need in a single script.

Twisted tac files

The rest of the script contains what would normally be the entire content — a Twisted tac file. A tac file is a Twisted Application Configuration file that tells twistd how to construct an application. As a configuration file it is responsible for choosing settings (like port numbers, poetry file locations, etc.) to run the application in some particular way. In other words, a tac file represents a specific deployment of our service (serve that poem on this port) rather than a general script for starting any poetry server.

If we were running multiple poetry servers on the same host, we would have a tac file for each one (so you can see why tac files normally don’t contain any general-purpose code). In our example, the tac file is configured to serve poetry/ecstasy.txt run on port 10000 of the loopback interface:

# configuration parameters
port = 10000
iface = 'localhost'
poetry_file = 'poetry/ecstasy.txt'

Note that twistd doesn’t know anything about these particular variables, we just define them here to keep all our configuration values in one place. In fact, twistd only really cares about one variable in the entire file, as we’ll see shortly. Next we begin building up our application:

# this will hold the services that combine to form the poetry server
top_service = service.MultiService()

Our poetry server is going to consist of two services, the PoetryService we defined above, and a Twisted built-in service that creates the listening socket our poem will be served from. Since these two services are clearly related to each other, we’ll group them together using a MultiService, a Twisted class which implements both IService and IServiceCollection.

As a service collection, the MultiService will group our two poetry services together. And as a service, the MultiService will start both child services when the MultiService itself is started, and stop both child services when it is stopped. Let’s add the first poetry service to the collection:

# the poetry service holds the poem. it will load the poem when it is
# started
poetry_service = PoetryService(poetry_file)
poetry_service.setServiceParent(top_service)

This is pretty simple stuff. We just create the PoetryService and then add it to the collection with setServiceParent, a method we inherited from the Twisted base class. Next we add the TCP listener:

# the tcp service connects the factory to a listening socket. it will
# create the listening socket when it is started
factory = PoetryFactory(poetry_service)
tcp_service = internet.TCPServer(port, factory, interface=iface)
tcp_service.setServiceParent(top_service)

Twisted provides the TCPServer service for creating a TCP listening socket connected to an arbitrary factory (in this case our PoetryFactory). We don’t call reactor.listenTCP directly because the job of a tac file is to get our application ready to start, without actually starting it. The TCPServer will create the socket after it is started by twistd.

You might have noticed we didn’t bother to give any of our services names. Naming services is not required, but only an optional feature you can use if you want to ‘look up’ services at runtime. Since we don’t need to do that in our little application, we don’t bother with it here.

Ok, now we’ve got both our services combined into a collection. Now we just make our Application and add our collection to it:

# this variable has to be named 'application'
application = service.Application("fastpoetry")

# this hooks the collection we made to the application
top_service.setServiceParent(application)

The only variable in this script that twistd really cares about is the application variable. That is how twistd will find the application it’s supposed to start (and so the variable has to be named ‘application’). And when the application is started, all the services we added to it will be started as well.

Figure 34 shows the structure of the application we just built:

Running the Server

Let’s take our new server for a spin. As a tac file, we need to start it with twistd. Of course, it’s also just a regular Python file, too. So let’s run it with Python first and see what happens:

python twisted-server-3/fastpoetry.py

If you do this, you’ll find that what happens is nothing! As we said before, the job of a tac file is to get an application ready to run, without actually running it. As a reminder of this special purpose of tac files, some people name them with a .tac extension instead of .py. But the twistd script doesn’t actually care about the extension.

Let’s run our server for real, using twistd:

twistd --nodaemon --python twisted-server-3/fastpoetry.py

After running that command, you should see some output like this:

2010-06-23 20:57:14-0700 [-] Log opened.
2010-06-23 20:57:14-0700 [-] twistd 10.0.0 (/usr/bin/python 2.6.5) starting up.
2010-06-23 20:57:14-0700 [-] reactor class: twisted.internet.selectreactor.SelectReactor.
2010-06-23 20:57:14-0700 [-] __builtin__.PoetryFactory starting on 10000
2010-06-23 20:57:14-0700 [-] Starting factory <__builtin__.PoetryFactory instance at 0x14ae8c0>
2010-06-23 20:57:14-0700 [-] loaded a poem from: poetry/ecstasy.txt

Here’s a few things to notice:

1. You can see the output of the Twisted logging system, including the PoetryFactory‘s call to log.msg. But we didn’t install a logger in our tac file, so twistd must have installed one for us.
2. You can also see our two main services, the PoetryService and the TCPServer starting up.
3. The shell prompt never came back. That means our server isn’t running as a daemon. By default, twistd does run a server as a daemon process (that’s the main reason twistd exists), but if you include the --nodaemon option then twistd will run your server as a regular shell process instead, and will direct the log output to standard output as well. This is useful for debugging your tac files.

Now test out the server by fetching a poem, either with one of our poetry clients or just netcat:

netcat localhost 10000

That should fetch the poem from the server and you should see a new log line like this:

2010-06-27 22:17:39-0700 [__builtin__.PoetryFactory] sending 3003 bytes of poetry to IPv4Address(TCP, '127.0.0.1', 58208)

That’s from the call to log.msg in PoetryProtocol.connectionMade. As you make more requests to the server, you will see additional log entries for each request.

Now stop the server by pressing Ctrl-C. You should see some output like this:

^C2010-06-29 21:32:59-0700 [-] Received SIGINT, shutting down.
2010-06-29 21:32:59-0700 [-] (Port 10000 Closed)
2010-06-29 21:32:59-0700 [-] Stopping factory <__builtin__.PoetryFactory instance at 0x28d38c0>
2010-06-29 21:32:59-0700 [-] Main loop terminated.
2010-06-29 21:32:59-0700 [-] Server Shut Down.

As you can see, Twisted does not simply crash, but shuts itself down cleanly and tells you about it with log messages. Notice our two main services shutting themselves down as well.

Ok, now start the server up once more:

twistd --nodaemon --python twisted-server-3/fastpoetry.py

Then open another shell and change to the twisted-intro directory. A directory listing should show a file called twistd.pid. This file is created by twistd and contains the process ID of our running server. Try executing this alternative command to shut down the server:

kill `cat twistd.pid`

Notice that twistd cleans up the process ID file when our server shuts down.

A Real Daemon

Now let’s start our server as an actual daemon process, which is even simpler to do as it’s twistd‘s default behavior:

twistd --python twisted-server-3/fastpoetry.py

This time we get our shell prompt back almost immediately. And if you list the contents of your directory you will see, in addition to the twistd.pid file for the server we just ran, a twistd.log file with the log entries that were formerly displayed at the shell prompt.

When starting a daemon process, twistd installs a log handler that writes entries to a file instead of standard output. The default log file is twistd.log, located in the same directory where you ran twistd, but you can change that with the --logfile option if you wish. The handler that twistd installs also rotates the log whenever the size exceeds one megabyte.

You should be able to see the server running by listing all the processes on your system. Go ahead and test out the server by fetching another poem. You should see new entries appear in the log file for each poem you request.

Since the server is no longer connected to the shell (or any other process except init), you cannot shut it down with Ctrl-C. As a true daemon process, it will continue to run even if you log out. But we can still use the twistd.pid file to stop the process:

kill `cat twistd.pid`

And when that happens the shutdown messages appear in the log, the twistd.pid file is removed, and our server stops running. Neato.

It’s a good idea to check out some of the other twistd startup options. For example, you can tell twistd to switch to a different user or group account before starting the daemon (typically a way to drop privileges your server doesn’t need as a security precaution). We won’t bother going into those extra options, you can find them using the --help switch to twistd.

The Twisted Plugin System

Ok, now we can use twistd to start up our servers as genuine daemon processes. This is all very nice, and the fact that our “configuration” files are really just Python source files gives us a great deal of flexibility in how we set things up. But we don’t always need that much flexibility. For our poetry servers, we typically only have a few options we might care about:

1. The poem to serve.
2. The port to serve it from.
3. The interface to listen on.

Making new tac files for simple variations on those values seems rather excessive. It would be nice if we could just specify those values as options on the twistd command line. The Twisted plugin system allows us to do just that.

Twisted plugins provide a way of defining named Applications, with a custom set of command-line options, that twistd can dynamically discover and run. Twisted itself comes with a set of built-in plugins. You can see them all by running twistd without any arguments. Try running it now, but outside of the twisted-intro directory. After the help section, you should see some output like this:

    ...
    ftp                An FTP server.
    telnet             A simple, telnet-based remote debugging service.
    socks              A SOCKSv4 proxy service.
    ...

Each line shows one of the built-in plugins that come with Twisted. And you can run any of them using twistd.
Each plugin also comes with its own set of options, which you can discover using --help. Let’s see what the options for the ftp plugin are:

twistd ftp --help

Note that you need to put the --help switch after the ftp command, since you want the options for the ftp plugin rather than for twistd itself.
We can run the ftp server with twistd just like we ran our poetry server. But since it’s a plugin, we just run it by name:

twistd --nodaemon ftp --port 10001

That command runs the ftp plugin in non-daemon mode on port 10001. Note the twistd option nodaemon comes before the plugin name, while the plugin-specific option port comes after the plugin name. As with our poetry server, you can stop that plugin with Ctrl-C.

Ok, let’s turn our poetry server into a Twisted plugin. First we need to introduce a couple of new concepts.

IPlugin

Any Twisted plugin must implement the twisted.plugin.IPlugin interface. If you look at the declaration of that Interface, you’ll find it doesn’t actually specify any methods. Implementing IPlugin is simply a way for a plugin to say “Hello, I’m a plugin!” so twistd can find it. Of course, to be of any use, it will have to implement some other interface and we’ll get to that shortly.

But how do you know if an object actually implements an empty interface? The zope.interface package includes a function called implements that you can use to declare that a particular class implements a particular interface. We’ll see an example of that in the plugin version of our poetry server.

IServiceMaker

In addition to IPlugin, our plugin will implement the IServiceMaker interface. An object which implements IServiceMaker knows how to create an IService that will form the heart of a running application. IServiceMaker specifies three attributes and a method:

1. tapname: a string name for our plugin. The “tap” stands for Twisted Application Plugin. Note: an older version of Twisted also made use of pickled application files called “tapfiles”, but that functionality is deprecated.
2. description: a description of the plugin, which twistd will display as part of its help text.
3. options: an object which describes the command-line options this plugin accepts.
4. makeService: a method which creates a new IService object, given a specific set of command-line options

We’ll see how all this gets put together in the next version of our poetry server.

Fast Poetry 3.0

Now we’re ready to take a look at the plugin version of Fast Poetry, located in twisted/plugins/fastpoetry_plugin.py.

You might notice we’ve named these directories differently than any of the other examples. That’s because twistd requires plugin files to be located in a twisted/plugins directory, located in your Python module search path. The directory doesn’t have to be a package (i.e., you don’t need any __init__.py files) and you can have multiple twisted/plugins directories on your path and twistd will find them all. The actual filename you use for the plugin doesn’t matter either, but it’s still a good idea to name it according to the application it represents, like we have done here.

The first part of our plugin contains the same poetry protocol, factory, and service implementations as our tac file. And as before, this code would normally be in a separate module but we’ve placed it in the plugin to make the example self-contained.

Next comes the declaration of the plugin’s command-line options:

class Options(usage.Options):

    optParameters = [
        ['port', 'p', 10000, 'The port number to listen on.'],
        ['poem', None, None, 'The file containing the poem.'],
        ['iface', None, 'localhost', 'The interface to listen on.'],
        ]

This code specifies the plugin-specific options that a user can place after the plugin name on the twistd command line. We won’t go into details here as it should be fairly clear what is going on. Now we get to the main part of our plugin, the service maker class:

class PoetryServiceMaker(object):

    implements(service.IServiceMaker, IPlugin)

    tapname = "fastpoetry"
    description = "A fast poetry service."
    options = Options

    def makeService(self, options):
        top_service = service.MultiService()

        poetry_service = PoetryService(options['poem'])
        poetry_service.setServiceParent(top_service)

        factory = PoetryFactory(poetry_service)
        tcp_service = internet.TCPServer(int(options['port']), factory,
                                         interface=options['iface'])
        tcp_service.setServiceParent(top_service)

        return top_service

Here you can see how the zope.interface.implements function is used to declare that our class implements both IServiceMaker and IPlugin.

You should recognize the code in makeService from our earlier tac file implementation. But this time we don’t need to make an Application object ourselves, we just create and return the top level service that our application will run and twistd will take care of the rest. Notice how we use the options argument to retrieve the plugin-specific command-line options given to twistd.

After declaring that class, there’s only on thing left to do:

service_maker = PoetryServiceMaker()

The twistd script will discover that instance of our plugin and use it to construct the top level service. Unlike the tac file, the variable name we choose is irrelevant. What matters is that our object implements both IPlugin and IServiceMaker.

Now that we’ve created our plugin, let’s run it. Make sure that you are in the twisted-intro directory, or that the twisted-intro directory is in your python module search path. Then try running twistd by itself. You should now see that “fastpoetry” is one of the plugins listed, along with the description text from our plugin file.

You will also notice that a new file called dropin.cache has appeared in the twisted/plugins directory. This file is created by twistd to speed up subsequent scans for plugins.

Now let’s get some help on using our plugin:

twistd fastpoetry --help

You should see the options that are specific to the fastpoetry plugin in the help text. Finally, let’s run our plugin:

twistd fastpoetry --port 10000 --poem poetry/ecstasy.txt

That will start a fastpoetry server running as a daemon. As before, you should see both twistd.pid and twistd.log files in the current directory. After testing out the server, you can shut it down:

kill `cat twistd.pid`

And that’s how you make a Twisted plugin.

Summary

In this Part we learned about turning our Twisted servers into long-running daemons. We touched on the Twisted logging system and on how to use twistd to start a Twisted application as a daemon process, either from a tac configuration file or a Twisted plugin. In Part 17 we’ll return to the more fundamental topic of asynchronous programming and look at another way of structuring our callbacks in Twisted.

Suggested Exercises

1. Modify the tac file to serve a second poem on another port. Keep the services for each poem separate by using another MultiService object.
2. Create a new tac file that starts a poetry proxy server.
3. Modify the plugin file to accept an optional second poetry file and second port to serve it on.
4. Create a new plugin for the poetry proxy server.