An Introduction to Asynchronous Programming and Twisted

Part 3: Our Eye-beams Begin to Twist

This continues the introduction started here. You can find an index to the entire series here.

Doing Nothing, the Twisted Way

Eventually we are going to re-implement our asynchronous poetry client using Twisted. But first let’s write a few really simple Twisted programs just to get the flavor of things. As I mentioned in Part 2, I developed these examples using Twisted 8.2.0. Twisted APIs do change, but the core APIs we are going to use will likely change slowly, if at all, so I expect these examples to work for many future releases. If you don’t have Twisted installed you can obtain it here.

The absolute simplest Twisted program is listed below, and is also available in basic-twisted/simple.py in the base directory of the twisted-intro example code.

from twisted.internet import reactor
reactor.run()

You can run it like this:

python basic-twisted/simple.py

As we saw in Part 2, Twisted is an implementation of the Reactor Pattern and thus contains an object that represents the reactor, or event loop, that is the heart of any Twisted program. The first line of our program imports the reactor object so we can use it, and the second line tells the reactor to start running the loop.

This program just sits there doing nothing. You’ll have to stop it by pressing Control-C, otherwise it will just sit there forever. Normally we would have given the loop one or more file descriptors (connected to, say, a poetry server) that we want to monitor for I/O. We’ll see how to do that later, but for now our reactor loop is stuck. Note that this is not a busy loop which keeps cycling over and over. If you happen to have a CPU meter on your screen, you won’t see any spikes caused by this technically infinite loop. In fact, our program isn’t using any CPU at all. Instead, the reactor is stuck at the top cycle of Figure 5, waiting for an event that will never come (to be specific, waiting on a select call with no file descriptors).

That might make for a compelling metaphor of Hamletian inaction, but it’s still a pretty boring program. We’re about to make it more interesting, but we can already draw a few conclusions:

  1. Twisted’s reactor loop doesn’t start until told to. You start it by calling reactor.run().
  2. The reactor loop runs in the same thread it was started in. In this case, it runs in the main (and only) thread.
  3. Once the loop starts up, it just keeps going. The reactor is now “in control” of the program (or the specific thread it was started in).
  4. If it doesn’t have anything to do, the reactor loop does not consume CPU.
  5. The reactor isn’t created explicitly, just imported.

That last point is worth elaborating on. In Twisted, the reactor is basically a Singleton. There is only one reactor object and it is created implicitly when you import it. If you open the reactor module in the twisted.internet package you will find very little code. The actual implementation resides in other files (see, for example, twisted.internet.selectreactor).

Twisted actually contains multiple reactor implementations. As mentioned in Part 2, the select call is just one method of waiting on file descriptors. Twisted includes several reactor implementations that use a variety of different methods. For example, twisted.internet.pollreactor uses the poll system call instead of select.

To use a specific reactor, you must install it before importing twisted.internet.reactor. Here is how you install the pollreactor:

from twisted.internet import pollreactor
pollreactor.install()

If you import twisted.internet.reactor without first installing a specific reactor implementation, then Twisted will install the default reactor for you. The particular one you get will depend on the operating system and Twisted version you are using. For that reason, it is general practice not to import the reactor at the top level of modules to avoid accidentally installing the default reactor. Instead, import the reactor in the same scope in which you use it.

Note: as of this writing, Twisted has been moving gradually towards an architecture which would allow multiple reactors to co-exist. In this scheme, a reactor object would be passed around as a reference rather than imported from a module.

Note: not all operating systems support the poll call. If that is the case for your system, this example will not work.

Now we can re-implement our first Twisted program using the pollreactor, as found in basic-twisted/simple-poll.py:

from twisted.internet import pollreactor
pollreactor.install()

from twisted.internet import reactor
reactor.run()

And we have a poll loop that does nothing at all instead of a select loop that does nothing at all. Neato.

We’re going to stick with the default reactor for the rest of this introduction. For the purposes of learning Twisted, all the reactors do the same thing.

Hello, Twisted

Let’s make a Twisted program that at least does something. Here’s one that prints a message to the terminal window, after the reactor loop starts up:

def hello():
    print 'Hello from the reactor loop!'
    print 'Lately I feel like I\'m stuck in a rut.'

from twisted.internet import reactor

reactor.callWhenRunning(hello)

print 'Starting the reactor.'
reactor.run()

This program is in basic-twisted/hello.py. If you run it, you will see this output:

Starting the reactor.
Hello from the reactor loop!
Lately I feel like I'm stuck in a rut.

You’ll still have to kill the program yourself, since it gets stuck again after printing those lines.

Notice the hello function is called after the reactor starts running. That means it is called by the reactor itself, so Twisted code must be calling our function. We arrange for this to happen by invoking the reactor method callWhenRunning with a reference to the function we want Twisted to call. And, of course, we have to do that before we start the reactor.

We use the term callback to describe the reference to the hello function. A callback is a function reference that we give to Twisted (or any other framework) that Twisted will use to “call us back” at the appropriate time, in this case right after the reactor loop starts up. Since Twisted’s loop is separate from our code, most interactions between the reactor core and our business logic will begin with a callback to a function we gave to Twisted using various APIs.

We can see how Twisted is calling our code using this program:

import traceback

def stack():
    print 'The python stack:'
    traceback.print_stack()

from twisted.internet import reactor
reactor.callWhenRunning(stack)
reactor.run()

You can find it in basic-twisted/stack.py and it prints out something like this:

The python stack:
...
  reactor.run() <-- This is where we called the reactor
...
...  <-- A bunch of Twisted function calls
...
  traceback.print_stack() <-- The second line in the stack function

Don’t worry about all the Twisted calls in between. Just notice the relationship between the reactor.run() call and our callback.

What’s the deal with callbacks?

Twisted is not the only reactor framework that uses callbacks. The older asynchronous Python frameworks Medusa and asyncore also use them. As do the GUI toolkits GTK and QT, both based, like many GUI frameworks, on a reactor loop.

The developers of reactive systems sure love callbacks. Maybe they should just marry them. Maybe they already did. But consider this:

  1. The reactor pattern is single-threaded.
  2. A reactive framework like Twisted implements the reactor loop so our code doesn’t have to.
  3. Our code still needs to get called to implement our business logic.
  4. Since it is “in control” of the single thread, the reactor loop will have to call our code.
  5. The reactor can’t know in advance which part of our code needs to be called.

In this situation callbacks are not just one option — they are the only real game in town.

Figure 6 shows what happens during a callback:

Figure 6: the reactor making a callback

Figure 6: the reactor making a callback

Figure 6 illustrates some important properties of callbacks:

  1. Our callback code runs in the same thread as the Twisted loop.
  2. When our callbacks are running, the Twisted loop is not running.
  3. And vice versa.
  4. The reactor loop resumes when our callback returns.

During a callback, the Twisted loop is effectively “blocked” on our code. So we should make sure our callback code doesn’t waste any time. In particular, we should avoid making blocking I/O calls in our callbacks. Otherwise, we would be defeating the whole point of using the reactor pattern in the first place. Twisted will not take any special precautions to prevent our code from blocking, we just have to make sure not to do it. As we will eventually see, for the common case of network I/O we don’t have to worry about it as we let Twisted do the asynchronous communication for us.

Other examples of potentially blocking operations include reading or writing from a non-socket file descriptor (like a pipe) or waiting for a subprocess to finish. Exactly how you switch from blocking to non-blocking operations is specific to what you are doing, but there is often a Twisted API that will help you do it. Note that many standard Python functions have no way to switch to a non-blocking mode. For example, the os.system function will always block until the subprocess is finished. That’s just how it works. So when using Twisted, you will have to eschew os.system in favor of the Twisted API for launching subprocesses.

Goodbye, Twisted

It turns out you can tell the Twisted reactor to stop running by using the reactor’s stop method. But once stopped the reactor cannot be restarted, so it’s generally something you do only when your program needs to exit.

Note: there has been past discussion on the Twisted mailing list about making the reactor “restartable” so it could be started and stopped as you like. But as of version 8.2.0, you can only start (and thus stop) the reactor once.

Here’s a program, listed in basic-twisted/countdown.py, which stops the reactor after a 5 second countdown:

class Countdown(object):

    counter = 5

    def count(self):
        if self.counter == 0:
            reactor.stop()
        else:
            print self.counter, '...'
            self.counter -= 1
            reactor.callLater(1, self.count)

from twisted.internet import reactor

reactor.callWhenRunning(Countdown().count)

print 'Start!'
reactor.run()
print 'Stop!'

This program uses the callLater API to register a callback with Twisted. With callLater the callback is the second argument and the first argument is the number of seconds in the future you would like your callback to run. You can use a floating point number to specify a fractional number of seconds, too.

So how does Twisted arrange to execute the callback at the right time? Since this program doesn’t listen on any file descriptors, why doesn’t it get stuck in the select loop like the others? The select call, and the others like it, also accepts an optional timeout value. If a timeout value is supplied and no file descriptors have become ready for I/O within the specified time then the select call will return anyway. Incidentally, by passing a timeout value of zero you can quickly check (or “poll”) a set of file descriptors without blocking at all.

You can think of a timeout as another kind of event the event loop of Figure 5 is waiting for. And Twisted uses timeouts to make sure any “timed callbacks” registered with callLater get called at the right time. Or rather, at approximately the right time. If another callback takes a really long time to execute, a timed callback may be delayed past its schedule. Twisted’s callLater mechanism cannot provide the sort of guarantees required in a hard real-time system.

Here is the output of our countdown program:

Start!
5 ...
4 ...
3 ...
2 ...
1 ...
Stop!

Note the “Stop!” line at the ends shows us that when the reactor exits, the reactor.run call returns. And we have a program that stops all by itself.

Take That, Twisted

Since Twisted often ends up calling our code in the form of callbacks, you might wonder what happens when a callback raises an exception. Let’s try it out. The program in basic-twisted/exception.py raises an exception in one callback, but behaves normally in another:

def falldown():
    raise Exception('I fall down.')

def upagain():
    print 'But I get up again.'
    reactor.stop()

from twisted.internet import reactor

reactor.callWhenRunning(falldown)
reactor.callWhenRunning(upagain)

print 'Starting the reactor.'
reactor.run()

When you run it at the command line, you will see this output:

Starting the reactor.
Traceback (most recent call last):
  ... # I removed most of the traceback
exceptions.Exception: I fall down.
But I get up again.

Notice the second callback runs after the first, even though we see the traceback from the exception the first raised. And if you comment out the reactor.stop() call, the program will just keep running forever. So the reactor will keep going even when our callbacks fail (though it will report the exception).

Network servers generally need to be pretty robust pieces of software. They’re not supposed to crash whenever any random bug shows its head. That’s not to say we should be lackadaisical when it comes to handling our own errors, but it’s nice to know Twisted has our back.

Poetry, Please

Now we’re ready to grab some poetry with Twisted. In Part 4, we will implement a Twisted version of our asynchronous poetry client.

Suggested Exercises

  1. Update the countdown.py program to have three independently running counters going at different rates. Stop the reactor when all counters have finished.
  2. Consider the LoopingCall class in twisted.internet.task. Rewrite the countdown program above to use LoopingCall. You only need the start and stop methods and you don’t need to use the “deferred” return value in any way. We’ll learn what a “deferred” value is in a later Part.

86 thoughts on “An Introduction to Asynchronous Programming and Twisted”

  1. in the countdown app you are doing import the reactor twice. I don’t think that’s neccesary, the code runs fine without the one in line 7.

    1. Hey Luis, that’s a good point, there’s no actual need for the inline import. I’m going to go ahead
      and leave it, though. Since most reactor imports are done inline, it’s still a good example of typical
      usage, I think.

  2. Thanks for the intro. Maybe a dumb question here, after I install a reactor will subsequent imports of reactor from twisted internet still use my installed reactor?

      1. I was wondering about the same thing and doing line 6 confused me quite a bit. Doing further investigating reveals that it’s not needed. So I’d suggest removing that from the code altogether. It confused me, at least :)

  3. Me again :) Any idea why this, when trying to run basic-twisted/simple-poll.py:

    File “/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/internet/pollreactor.py”, line 19, in
    from select import error as SelectError, poll

    Been beating my head against it for quite a while now. Do pardon whatever obvious thing I’m undoubtedly missing (or operator error, more likely).

  4. Hilarious: I can’t even report the problem correctly.

    Here’s what I was actually wondering about:
    “ImportError: cannot import name poll”

    Where is ‘select’ supposed to be found? The error message seems to imply that ‘error’ itself has been imported correctly, but that ‘poll’ can’t be located, yes? Any insights greatly appreciated.

    1. Hello Felipe,

      I didn’t succeed to run your solution without changing

      line 10 : print “LoopCall # {} down to {}”.format(index, counters[index])

      by

      print “LoopCall # {0} down to {1}”.format(index, counters[index])

      copy/paste problem ?

      regards

      1. probably python versions. If I recall correctly, in 2.6 and earlier, you have to explicitly specify your var indices. My guess is Felipe is using 2.7 or higher…

    1. Glad you like the series!

      I think you’ve got yourself a a solution there, nicely done. I do think it would be better to avoid sub-classing LoopingCall. How would you do that?

  5. Hey Dave,

    I’m quite new to Twisted Programming, and I have a question regarding the NotRestartable thingy: what if the sockets and factories the SelectReactor shall deal with, change during runtime? Say, an IP address changed, or – in my case – you got seperate modules for each Protocol which are not known at the moment the reactor.run()s ?
    As you cannot restart the reactor, I see the following options:
    - Change to poll reactor to have one reactor for each Protocol: not possible in my case (Mac OSX)
    - Change Program Architecture to start Reactor after sockets are known

    Are there any further options?

    Cheers

      1. Hey Fabian, I didn’t have time to respond to your post yesterday. The short answer is that it’s definitely
        possible, just do those same calls to connect to a port while your program is running rather than on startup.
        But now that you have the question out on the Twisted mailing list, you should get some detailed answers.

        1. The trick is to make an initial connection.

          I just did an initial reactor.run() and somewhen later made the first reactor.connectTCP() – it just got stuck then.
          It’ll work if you provide the reactor with a connectTCP() to some kind of Fake Protocol BEFORE calling reactor.run() :-)

          1. Interesting solution :) You can always call reactor.connectTCP after the reactor starts and you don’t need to call it even once
            before it does so. However, you will need to arrange for some callback to be invoked when you are ready to make
            the connection. Consider the reactor.callLater() function, for example.

  6. Here’s my solution for #1. I don’t really believe this is the best approach but since the counters have to communicate somehow between each other in order to decide when to call reactor.stop() the only solution I managed to come up with was global variables.

    http://dl.dropbox.com/u/11996458/countdown.py

    I would greatly appreciate if someone could give me a hint on how this could could be solved in a better way.

    P.S.: I’m not a native speaker, so excuse me for any mistakes I could have made :)

  7. Your solution certainly works! And your English is very good, don’t worry.

    To get rid of the global variable, you could pass a shared object to each
    of the counters and have them invoke a method on it when the counter
    is done. And then that separate object could be in charge of shutting down
    the reactor instead of the counters. You could even just pass a reference
    to the method itself, as a ‘callback’, which is kind of looking forward to the
    next parts.

      1. I did have one question,

        Out of curiosity, I commented out the line that did stopped the looping call. The code still worked. I suspected this was a bad idea, but why?

        1. It still printed out the right things because of that ‘if self.counter:
          guard in your count() method. Although the loops keep running after they
          reach zero, they don’t print anything out. And when you stop the reactor it doesn’t
          wait for LoopingCalls (which are implemented using reactor.callLater()).

          But it’s still much better to stop those loops when you don’t need them. They will
          take up some CPU because they keep looping, and if you accumulate a lot of them then
          that will add up.

  8. Very well explained .
    I am new to python and having some problem in exercise :
    this is link to my solution of exercise1 :http://pastebin.com/T5hihtTM
    this is giving some errors, please give it a look!!(i m sure i m not using select.select() correctly )

    I was looking at solution of Narada , https://gist.github.com/899383 . It is looking sync to me . I mean where is outer loop which keep care of which counter to run which time (which is not blocked).

    I know these are silly things to ask and not worthy of your time ,but i have tried myself to understand and no success. Hoping your explanation will help me to understand later parts in better way.

    1. Hi Amit, in Part 3 you don’t need the outer loop because the Twisted reactor provides that loop for you — the loop starts when you call reactor.run() and that function doesn’t return until the loop finally stops. So there is no need for your own loop (and no need for a select() call since there are no sockets to listen on).

      Does that make sense?

      dave

  9. Hi Dave! Thanks for tutorial, I really like the way you explain.

    One question I have is, how does Twisted benefits from smaller tasks (callbacks)? Would it make any difference for one single request, if I do say 100 smaller callbacks instead of 10 bigger? In the end, its the same amount of work. And those callbacks do they work actually blocking the reactor for a small period of time, right?

    1. Yes, you are right that it is about the same amount of work (the 100 requests
      do have the extra function call overhead). And while a callback (any callback)
      is running, the reactor cannot service any other requests. If your server only
      ever handles one request at a time, then it wouldn’t make any difference. Of
      course, if your server only handles one request at a time, then there’s no
      real reason to use the asynchronous model either.

  10. I love how you explicitly point out things like “when our callbacks are running, the Twisted loop is not running”! It really helps :)

  11. Hi Dave,
    Thanks for the tutorial. I’m new to python and async. I’m getting some errors in my attempt at exercise 1. My thoughts were to pass in a separate object that would store a dictionary of the different counters and their respective counts and would check for when the counters all were down to zero. However, I think that my class is running outside the reactor as any debugging print statements happen before the reactor start message. I think that this is the root of my issues. My ‘complete’ method is tossing errors that I don’t understand. Any insight you could offer would be a great assistance! Thanks! http://dpaste.com/hold/788866/

    1. Hey Seth, welcome to Python! The reason why you have some code running before
      the reactor starts is because the calls to .count() in callWhenRunning are
      happening before you call callWhenRunning. The argument to callWhenRunning
      should be a function reference (the function you want to call when the reactor
      is running), plus whatever arguments you want to pass to that function when
      it is actually called. So you need something like:

      reactor.callWhenRunning(Countdown().count, groupCounter)

      Does that make sense? Since you are new to Python, I have implemented a version
      of your idea in a bit more of a ‘Pythonic way’ here,
      if you interested, but feel free to figure it out on your own first if you like.

      1. Dave,
        Excellent! That makes complete sense. By looking at the documentation, apparently, passing group_counter (lines 50-52 in your re-write of my code) is available in def count (line 28) by being the second parameter in callWhenRunning. Thanks a ton!

  12. Hi Dave,
    I’m having trouble with exercise #1. You did refer to it in another reply, but it doesn’t show up. Can you repeat it or provide a new link?

    Your tutorial is fantastic — very well written and even fun to read.

    Thanks,
    dave

  13. I found the (a) solution:

    #!/usr/bin/env python

    class Countdown(object):
    number_running = 0

    def __init__(self, max):
    self.max_count = max
    self.counter = self.max_count
    Countdown.number_running += 1

    def count_down(self):
    if self.counter == 0:
    Countdown.number_running -= 1
    if Countdown.number_running == 0:
    reactor.stop()
    else:
    print self.counter, ‘…’
    self.counter -= 1
    reactor.callLater(1, self.count_down)

    from twisted.internet import reactor

    counter1 = Countdown(5)
    counter2 = Countdown(10)
    counter3 = Countdown(15)
    reactor.callWhenRunning(counter1.count_down)
    reactor.callWhenRunning(counter2.count_down)
    reactor.callWhenRunning(counter3.count_down)

    print ‘Start!’
    reactor.run()
    print ‘Stop!’

    1. Hi Dave,

      I used your solution as a springboard to my own but I think your solution only works if you have the same delay for each counter. The exercise asked us to have a variable delay and if you follow your code it doesn’t work. What I did was to move the if block dealing with the reactor stop below the original “if else” block to deal with this.

      No doubt my code can be massively improved, but here’s my solution that allows for both variable delay, and number of counts.

      class Countdown(object):

      number_of_counters = 0

      def __init__(self, name=”Countdown”, delay=1, counter=5):
      self.delay = delay
      self.counter = counter
      self.name = name
      Countdown.number_of_counters +=1

      def count(self):
      if self.counter == 0:
      print ‘A task (%s) has completed’ % (self.name,)
      Countdown.number_of_counters -= 1
      else:
      print self.name, self.counter, ‘…’
      self.counter -= 1
      reactor.callLater(self.delay, self.count)
      if Countdown.number_of_counters == 0:
      print ‘Stop the reactor’
      reactor.stop()

      from twisted.internet import reactor

      reactor.callWhenRunning(Countdown(“Counter 1″,3,3).count)
      reactor.callWhenRunning(Countdown(“Counter 2″,1,3).count)
      reactor.callWhenRunning(Countdown().count

      print ‘Start!’
      reactor.run()
      print ‘Stop!’

      And the output is as follows:

      Start!
      Counter 1 3 …
      Counter 2 3 …
      Countdown 5 …
      Counter 2 2 …
      Countdown 4 …
      Counter 2 1 …
      Countdown 3 …
      Counter 1 2 …
      A task (Counter 2) has completed
      Countdown 2 …
      Countdown 1 …
      A task (Countdown) has completed
      Counter 1 1 …
      A task (Counter 1) has completed
      Stop the reactor
      Stop!

  14. random question here on python, in the following code why is it reactor.callWhenRunning(stack) and NOT reactor.callWhenRunning(stack())… what is the difference in meaning between the two?

    def stack():
    print ‘The python stack:’
    traceback.print_stack()

    from twisted.internet import reactor
    reactor.callWhenRunning(stack)

    1. Hey Daniel, it’s the difference between a function call and a function reference.
      Consider the following from the interactive interpreter:


      >>> import time
      >>> print time.time
      <built-in function time>
      >>> print time.time()
      1349323851.11

      The first print shows the time.time function and the second the result
      of calling that same function. Now the callWhenRunning method expects
      a function reference and it will arrange for the reactor to call that function (using
      the reference) when the reactor starts. If we say:

      reactor.callWhenRunning(function_name)

      Then we are passing a function reference the reactor can use to call the function later
      by using the () operator. But if we say:

      reactor.callWhenRunning(function_name())

      Then Python will first call function_name and then it will call callWhenRunning
      with the result of the first function call as the first argument.

      Whew! Does that make sense?

  15. Many thanks for this lovely series! I’m new to Twisted and was unfamiliar with asynchronous programming terminology, so this has been a great help.

    I’ve been imagining a “reactor” as the thing keeping the program moving forward, perhaps by spitting out callbacks something like how a nuclear reactor spits out neutrons. While I still like the image, I suspect your account makes more sense…

    Here are my solutions for the exercises:

    #1: http://pastebin.com/uGjv6Fnh
    #2: http://pastebin.com/hqpGcJnm

    I actually came up with the two different approaches as solutions for #1, and then altered one of them to use LoopingCall.

    Is my sense that #2 (with or without LoopingCall) seems more “Twisted flavored” correct? And is the instinct (also in #2) to remove the call to reactor.stop from other callbacks, a good one (even if unnecessary here)?

    1. You are welcome! I quite like the imagery of a Twisted reactor firing out callbacks like a nuclear reactor firing off neutrons. And since the Twisted reactor really is driving the whole show, I think it definitely works well as a metaphor.

      Using LoopingCall would probably be what most Twisted developers would use because, well, it’s already there :)

      Your instinct to move the reactor.stop call from the Countdown object is definitely a good one. Few objects will
      need to call reactor.stop and it doesn’t make much sense in the Countdown object. I like the second solution better
      because the Countdown objects are actually performing the counting themselves, whereas in the first solution they
      are really just data structures.

      In the second solution the counters have to know about the monitors and vice-versa. Is there a way you could
      make the Countdown objects agnostic towards them?

  16. In writing the two solutions, I was thinking mostly of the difference between the first and second solutions as being how coordination was organized, i.e. between having the callbacks go to the aggregate pool vs going to the individual countdown objects. While I did notice that by the time I was done the Countdowns in #1 were stripped down almost to being structs, I chalked that up to this being a toy example.

    Though of course facilities like LoopingCall are the reason for using Twisted in the first place, the point of my aside was that even without LoopingCall, the organization of #2 would still be more suggestive of Twisted thinking — which you appear to agree with. :)

    Indeed, your suggestion of what to do further here seems to continue the process of separating the components: not only does the monitor not get to control the countdowns, they don’t even need to send it status messages.

    So far, the only strategy I’ve come up with that actually leaves the Countdowns blissfully oblivious of what’s going on around them is to have the monitor object poll them to see if they’ve finished. Like this:

    #3 http://pastebin.com/2WZ24H0k

    (The pasted version has (at least) two bugs, that don’t have any effect here: monitor.__init__ doesn’t set the rate attribute to the function argument, and monitor.poll fails to call self.looper.stop)

    Is that what you were getting at? It works, but I don’t have a very intuitive sense of whether it’s a good way to do this.

    Thanks for taking the time to look at my code.

    1. That’s definitely what I was getting at, but getting closer to the “Twisted”
      way of doing that would be to have the Countdown objects accept a generic
      callback parameter which is invoked when the count is done, i.e., don’t assume
      the thing on the other end is a countdown monitor with a ‘finalize’ method,
      just give whatever it might be a mechanism to find out when the count is
      finished.

      You could have countdown objects accept a ‘callback’ parameter that will be
      called with no arguments when the count is over. Then the monitor can pass
      something like lambda : self.finalize(ctdn) as the callback. Make sense?

      Eventually you will see that the fully “Twisted” way of doing this would be
      for the Countdown objects to provide a Deferred object, but that’s looking
      ahead.

  17. It seems there was one point I misunderstood in writing #3. My design goal there was that the Countdown objects were inward turned — an object like the monitor could examine what they were doing, but beyond a start signal, they accepted no outside information after construction, and called no other objects. I suppose I can say that this was at least asynchronous, but I’m not sure it could be called an asynchronous network…

    So here we go, a version in which the Countdown objects are very bad airline passengers that accept callbacks in brown paper wrappers from random passersby:

    #4: http://pastebin.com/WHJnqQ7f

    Also, now the monitor will track any object with a setCallback method. This feels a bit unspecific since there are events other than completion of a process that should trigger callbacks. Maybe it should be something like callWhenFinished?

    One rather arbitrary design goal I have used in all the versions is to store the startTime in only one place. This means I’m always trying to solve the problem of where and how each endTime should be generated. I included a commented out line that suggests how a timestamp could be passed back when the callback is triggered. Given that these callbacks triggered by a process completing, it doesn’t seem entirely out of place to send a timestamp. Though, now I think of it, I suppose the endTime argument could be made optional with a default. Are these the kinds of issues solved by Deferred objects?

    I find it useful to work through things on a low level like this. Thanks again for your help and patience here.

    1. Yes, that’s the sort of thing I had in mind. And callWhenFinished seems like a better
      name to me to. Deferreds solve the ‘managing callbacks’ problem so that each and every
      class like Countdown doesn’t have to manage its own set of callbacks. With the full
      complement of Deferreds and their helper classes, you would be able to dispense with
      the monitor class entirely.

      Implementing this stuff yourself first is a great way to understand why things are
      the way they are in a framework like Twisted.

  18. In one of the above comments I saw this fragment of code :

    from twisted.internet import reactor

    counter1 = Countdown(5)
    counter2 = Countdown(10)
    counter3 = Countdown(15)
    reactor.callWhenRunning(counter1.count_down)
    reactor.callWhenRunning(counter2.count_down)
    reactor.callWhenRunning(counter3.count_down)

    I’d like to ask that are all the 3 callWhenRunning() function calls asynchronous? Or does the second call wait for the first one to complete. Pardon me if this is a basic question. I seem to have some difficulty in understanding the callbacks. :(

    1. Hi, the callWhenRunning method is a bit special, because it is either
      synchronous or asynchronous depending on whether the reactor is actually
      running. If the reactor is running at the time you make the call, it just
      runs the function right away, it’s like you just called the function yourself.
      But if the reactor is not running, then the reactor schedules the function to
      be run once the reactor actually starts up.

      In the code fragment you saw, callWhenRunning is being called before the reactor
      is running (before the call to reactor.run()) so it is asynchronous. Does that
      make sense?

  19. If that’s asynchronous then in that case, shouldn’t the code give a slightly different output each time? Or at least there’s a chance of obtaining a different output? Please correct me if I am wrong.

    1. I’m not sure if Twisted guarantees if the scheduled calls to callWhenRunning
      will be called in order. Of course, if those functions themselves launch
      asynchronous operations, then in general you cannot expect a particular
      order of asynchronous operations. However, because of the contrived nature
      of this example, you may never see different output. All three calls are executing
      the same code and that code is simply calling reactor.callLater, rather
      than performing network communication or communicating with a separate
      processes.

  20. Hi Dave,
    Thanks for your great series.
    I have followed your discussion with Plover above and finally worked the solution to exercise#2 as following:
    http://pastebin.com/LWNeU3wv
    My basic idea is to set a manager to handle both the job of monitoring and managing of LoopingCall object. In this way, the client code can focus on its main task and only need to call the finalize method to tell the manager the job is done.
    Secondly, I want to make the manager a common manager of all kinds of tasks; thus the client must implant a interface with two method, Name() and Job(), to let the manager aware of its property without bound to it.
    I do not know whether this is a good idea or not. Or could you please give me some indication to further directions to craft this task?

    1. Having a task manager seems like a nice abstraction here. I think you could avoid the finalize callback by having a method on the job for asking if it is done.

      In terms of Python style, Name() and Job() look more like Java idioms. I would just access .name directly and have a get_job method. Also, _token doesn’t really tell you what that variable is for. How is a dictionary a token? What about _job_name2loop to remind you what the structure of the dictionary is.

      1. Thanks for your reply and the style advice is happily adopted.

        I am not sure how to avoid the finalize callback.

        I thought of keeping a property in the countdown class to indicate whether it is done. But I do not know how to make the manager know immediately that the job is done and I do not like the idea to fire a timer to detect the property in the job periodically. I suppose there must be some methods in twisted that can allow two task communicate but I have no idea what they are. May be I thought too complex about this issue but I could not find a way somehow.

        Could you please give me some indication about this issue?

        1. I would make the callback a method on the manager rather than a method on the job.
          The manager callback would:

          1. Call the job task function.
          2. Ask the job if it is done and stop.

          The other nice thing about that is the manager can catch exceptions from the job and cleanup as well.

          1. Like this?
            http://pastebin.com/NtvT8YKV

            I am not sure it is good to make the concrete task aware of a manager exist. In my previous version the task can be used without knowing the manager. The user do not want to use a manager can use the task as well. They can just leave the default finalize parameter as it be. I still cannot grasp the essence behind this style difference.

          2. That’s not quite what I meant. I agree that jobs should not know anything about their manager. Let’s make it a little more concrete. Say job objects have two methods:

            1. execute_job
            2. is_done

            And the manager has an internal method _execute_job(self, job).
            The manager would make a looping call like this: LoopingCall(self._execute_job, job).
            And that method could be something like:

            def _execute_job(self, job):
              job.execute_job()
              if job.is_done():
                ...
            

            Make sense?

Leave a Reply