Twisted Introduction

This multi-part series introduces Asynchronous Programming and the Twisted networking framework.

  1. In Which We Begin at the Beginning
  2. Slow Poetry and the Apocalypse
  3. Our Eye-beams Begin to Twist
  4. Twisted Poetry
  5. Twistier Poetry
  6. And Then We Took It Higher
  7. An Interlude, Deferred
  8. Deferred Poetry
  9. A Second Interlude, Deferred
  10. Poetry Transformed
  11. Your Poetry is Served
  12. A Poetry Transformation Server
  13. Deferred All The Way Down
  14. When a Deferred Isn’t
  15. Tested Poetry
  16. Twisted Daemonologie
  17. Just Another Way to Spell “Callback”
  18. Deferreds En Masse
  19. I Thought I Wanted It But I Changed My Mind
  20. Wheels within Wheels: Twisted and Erlang
  21. Lazy is as Lazy Doesn’t: Twisted and Haskell
  22. The End

This introduction has some translations in other languages:

119 thoughts on “Twisted Introduction”

  1. hi dave,

    I’m Jayson Pryde, and I’m new to twisted.
    I’ve been learning Twisted via your awesome tutorial, and I am already in part 13.
    I tried to implement my own client-server system, but I am encountering some hangs in the client.
    It’s probably because I messed up with callbacks in the deferred.
    I’ve asked this via this link in stackoverflow:

    http://stackoverflow.com/questions/42418021/asynchronous-client-in-twisted-not-sending-receiving-request-using-netstringr

    It would be greatly appreciated if you can find time and take a look, and help me point out where I messed up.
    So what happened in the client is I am already able to send a request, but it seems that the callback to process the request is not fired up.

    Hope you can find time. thanks a lot in advance!

    Reply

          1. Thanks a lot dave. I already merged it.
            Before I received your reply, this was somehow the same thing I did to make it working.
            I remove the other factory/class to make it work. But still, my question is, how come your examples work (even if they are already old) when I run it in my machine? And also, I’m still quite confused with the deferred returning a deferred.

            Thanks a lot again Dave! 🙂

          2. I’m not sure what you are asking — my examples work because the Twisted project has been very good about maintaining backwards compatibility. The code you posted didn’t work because of bugs I tried to explain in the pull request. I recommend reading the source code for Deferred itself and reworking the examples in the relevant chapters of the tutorial. Learning a new kind of programming takes time, you have to go slowly in the beginning.

  2. Hi dave,

    Thank you for your nice introduction to asynchronous programming, and I’m reading chapter 4 now. I think the philosophy of asynch programming is what I need for my project so I wish to ask you few questions.

    Let me explain myself first, currently I’m working on an automated web-site testing project. I decide use selenium + bs4 + unittest approach. The website I’m testing heavily relay on ajax and iframe, so I wish to develop someway better than traditional unittest case, which is separate interaction(I/O) and content assertion(RAM)

    My original idea is have a main thread do all navigation and interaction, and use bs4 to parse the DOM, and then create children to assert those parsed DOM. However, after read your blog, I realize I don’t really need threads to do it, I think asynchronous way suits my purpose too.

    So my question is, is my understanding of asynchronous programming suits my purpose correct? is this a possible solution for automated web testing? or is there anyone else already did/doing something similar? (since I really can’t find anything similar)

    Thank You
    Jack

    Reply

    1. Hey Jack, thanks for the kind words! If I’m understanding you, I think Twisted could be used in lieu of threads. I wonder, though, if you need threads or async at all? Is it not possible to do the interaction and then parse the result in the same thread? Both threaded and async code are more complex than vanilla single-threading and for functional testing the performance of a single, non-async thread may be acceptable.

      Reply

  3. Hi Dave, thank you for your answer!

    Since I’m using selenium to simulate user visiting website using browser, it actually involves heavy I/O operation: fetching pages. In fact, since the data in that website is huge and queries are not that optimized, some page may take more than 30s to load.

    This is why I’m looking for a method to separate the I/O operation from DOM assertion. You are right about threads, I think I don’t really need thread programming. But if my code can run DOM assertion while I/O fetching next page would be nice

    Furthermore, it’s also about code organization. Since the website currently is under developing, although main frame remain same, the content on single page might change frequently, that’s why I wish to keep them separate, for better organization and change

    Thank You
    Jack

    Reply

Leave a Reply