An Introduction To Asynchronous Programming and Twisted

Part 22: The End

This concludes the introduction started here. You can find an index to the entire series here.

All Done

Whew! Thank you for sticking with me. When I started this series I didn’t realize it was going to be this long, or take this much time to make. But I have enjoyed creating it and hope you have enjoyed reading it.

Now that I have finished, I will look into the possibility of generating a PDF format. No promises, though.

I would like to conclude with a few suggestions on how to continue your Twisted education.

Further Reading

First, I would recommend reading the Twisted online documentation. Although it is much-maligned, I think it’s better than it is often given credit for.

If you want to use Twisted for web programming, then Jean-Paul Calderone has a well-regarded series called “Twisted Web in 60 Seconds“. I suspect it will take a little longer than that to read, though.

There is also a Twisted Book, which I can’t say much about as I haven’t read it.

But more important than any of those, I think, is to read the Twisted source code. Since that code is written by people who know Twisted very well, it is an excellent source of examples for how do to things the “Twisted Way”.

Suggested Exercises

  1. Port a synchronous program you wrote to Twisted.
  2. Write a new Twisted program from scratch.
  3. Pick a bug from the Twisted bug database and fix it. Submit a patch to the Twisted developers. Don’t forget to read about the process for making your contribution.

The End, Really

Happy Hacking!

Figure 47: The End
Figure 47: The End

29 thoughts on “An Introduction To Asynchronous Programming and Twisted”

  1. Dave,

    This was an incredibly well-written and well-thought-out tutorial. This morning I decided that Twisted was the right tool for my job and I went about learning it and I came upon this series. 8 hours later, I’ve gone through each and every article and worked through all the examples; now I fully understand the different parts within Twisted, how they fit together, and the thought processes that went into designing them. It is truly rare that any library has documentation this clear and helpful, and you should be proud of your contribution.

    I also feel lucky that out of all the times that I could have stumbled upon this, I happened to do so within 2 weeks of you finishing this epic series. Guess things just work out nicely sometimes :)

    Thank you,
    Jonathan

  2. Dave,

    After thinking about Twisted for a while, it seems to me like it’s very similar to other event-engine frameworks, especially those dealing with GUI’s. For example, a common pattern in Javascript is to subscribe to an onMouseOver() event, where you specify code that gets control when an event fires. I’m very familiar with this model, and if this were all Twisted did I would be happy, but not particularly impressed.

    The real magic of Twisted, as opposed to an ordinary event framework, is that deferreds are very lightweight and easy to create/listen to. Especially when combined with the @inlineCallbacks decorator, this is the best effort I’ve seen to simplify asynchronous programming. I’ve always hated the proliferation of callbacks that you get with event engines, and having lots of anonymous methods was always a pain (also annoyingly impossible in python).

    Anyway, thanks again for introducing me to the magic of Twisted.

    –Jonathan

  3. This is the best tutorial ever! You are like the teacher whom you remember for all your life because he taught you more than you were supposed to know. Taking Erlang, Haskell and poetry (I didn’t know of Cummings before) to Twisted and Async programming in general counts to 5 topics instead of 1. Brilliant!
    Thank you very much for your hard work!

  4. Great Twisted tutorial! Thank you so much for writing it. And thank you for explaining deferreds and deferred lists, those were pretty confusing. The connection you drew through Erlang and Haskell between asynchronous programming and lazy evaluation was new to me, I know some Haskell but that had not occurred to me.

    That crack you made about the reactor loop picture back in chapter 8 just about killed me.

  5. Thanks for this very well written and informative tutorial, Dave. After reading through the Twisted docs originally, my head was spinning and I didn’t really grasp the concepts behind writing Twisted applications. Your tutorial helped immensely. You explained so many important details that the Twisted docs didn’t even bother covering that seem to be crucial in understanding how to use the framework properly. Well done.

    1. Thank you! I don’t have any plans at the moment, but the urge
      to write it seized me at random, so who knows :) Did you have
      any specific topic in mind?

      1. Well, I’m currently struggling to get started with a correct directory layout for my little project. You make some random remarks during the tutorial about separating things out and I have trouble seeing how that would actually work.

        For example, I’m writing a protocol (or trying to, at least) that allows a client to request a range of datapoints from a server. What would be a good practise to actually keep code separated? Would I write just one Protocol class and make it work for both the client and the server? And would I differentiate between the two via the Factory? Or would it be better to write separate Protocols for client and server?

        Also, maybe not really in the Twisted scope per se, but I’d be interested in working with authenticated connections, via x501 or something else? Would that be easy to do with Twisted?

        Lastly, would the async mindset work for distributing some kind of work load (think genetical algorithmes or MapReduce or something) among several machines? Or would that just complicate things?

        Lots of questions, but I’m a real newbie with Twisted and async programming (well, not entirely, since I actually read your entire tutorial!). Any more insights and explained code examples would be welcome, really. The Twisted code itself is pretty advanced, really, not really good examples to actually try to learn from, imho.

        Thanks for the tutorial, that’s the only reason why I dived into this :)

        1. I think it makes sense to have just one Protocol class most of the time,
          since conceptually there is just one protocol that both ends are speaking.
          And the server would use a Factory whereas clients can generally get away
          without having one.

          The key things to keep separate are the business logic that your service is
          providing via the protocol and the protocol itself. So write a ‘DatapointService’
          with a pure python API that python code running in the same process could use
          normally, without making a network connection at all.

          Then expose that service via one or more protocols, with code that lives outside
          the datapoint service class.

          For example, you could implement a REST endpoint using Twisted web that returns
          that datapoints as JSON. Consider the ‘Web Server’ example on the front page of
          the twistedmatrix.com site (second tab). Replace the render_GET implementation
          with code that calls your datapoint service (with a normal python function call),
          sets the content type to ‘application/json’, and returns the datapoints in a json
          string.

          Make sense?

          Doing workload distribution is definitely a use-case for async, though if you just
          want a solution, you might consider using something like Hadoop.

      1. I was new to Twisted; your 22 part blog is a valuable reference. Thanks!
        Liked your rendition of the john hurt tune. Is that a steel-stringed acoustic guitar you’re playing?

Leave a Reply