Twisted Introduction

the index link on this page:

http://krondo69349291.wpcomstaging.com/blog/?p=1825

links back to your main page, and not to the index.

great tutorial btw :)

Hi, Dave. Do you mind if i'l translate this twisted intro to russian and publish it?

Hello Dave,
I find your intro on Twisted to be the best I have found to date, clear and covers important details that support a real understanding of Twisted. I would like to ask if in the future you have any plans for other Twisted tutorials. Just for an example tutorial; as the intro above couldn't cover some deep areas of Twisted, I was thinking a multiple part (over time) tutorial that each covers say different abstractions. I would love to support you and the work you do in making Twisted a learning joy for me, a first time. Please let me know if any money I can send that will help.

Ed.

Thanks Dave, just thinking such work a labor of love ...wow.
I did look over the link you gave, and like it. Dave you take care, and I look forward to the parts you plan to add here, as I'm learning Twisted to build a larger tool set in python and your tutorial has been a great help in that direction.

Ed.

Hello, I understand this is not an interactive tutorial, if I may ask a question please. I find in part 5 something that I wish to understand; the word "buildFactory" has had me scratch my head because I can't seem to find it mentioned anywhere I have looked as portocol interfaces and others, along with the tutorial pages 4 and 6. Also, have not found a reference to it on web searches I have done too. Where do I find it or do I just lack some background in Twisted I need.

Ed.
Thank you.

Dave,

thank you for the help.

Ed.

Great intro!

Would be nice, if you could release the whole serie as a pdf.

Jo.

Thanks Dave for your excellent work!
I'm working on translating this series into Japanese. I've finished around part5.
Is it okay to keep publishing them in my space?
http://skitazaki.appspot.com/translation/twisted-intro-ja/index.html

Hi!

Thanks for hard work, i read all articles and my head-ache is gone :D
Even twisted book,that i lent from library of uni, was too hard to understand at the beginning, but no more...

Btw, if anyone want pdf format - you can download it from my google docs: https://docs.google.com/fileview?id=1kKQM0G6CvqU3BZKG9pyc2h37oZpdUjC7vG9JWA0Bk9ba42e5ZpKyZpp8T2dc&hl=en&authkey=CJThyr8L

And at the moment i'm working on estonian translation.

Hi timgluz,
can you tell how to download your pdf...it is asking for username/passwd when I click on the above link.

thanx

Dear Dave
I'm sure your writeup is a great source of knowledge.

But caused by the lame of my brain, I fail to understand how to adopt it to my problem.

I need to write application that :
1. Have to services in it:
1.a. a TCP server (just like a basic echo server)
1.b. an XMPP-Client/bot
2. Every msg come from XMPP-server, will wrote to StdOut (it'll be extended inthe future)
3. Every msg received by TCP-server part, will be forwarded by the xmpp-client to XMPP-server.

I post my code at pastebin:
1. echobot.py : http://pastebin.com/Mn4bRvCb
2. echobot.tac : http://pastebin.com/BgjMKiMC
3. the occured error : http://pastebin.com/6Rdge4L5

Kindly please give me any enlightment.

Sincerely
-bino-

I think your tutorials are really awesome, and I love how you're consistently putting so much work into this while remaining responsive and approachable.

This was *long* overdue, but the official documentation page now links back to you:

http://twistedmatrix.com/trac/wiki/Documentation

I couldn't figure out if marketing your tutorials as good-for-newbies or not was doing you a disservice or not; I'm just mainly using them as a pointer for newbies who are willing to learn. If you disagree with the wording, please feel free to e-mail me about it.

Finally, a very good introduction + tutorials + how-to in Twisted!

Keep them coming, maybe even publish it as a book later? ;)

Thanks much! :)

This guide looks good (I haven't actually read much of it yet :-)).

Given the length, it would be better to have a PDF for an e-reader, or printing it out, but the PDF mentioned earlier in the comments only go up to part 10.

Any plans to produce an updated PDF? Are the parts that are in the PDF up-to-date, or has there been edits, so that one would be better off just reading the whole guide online?

This is amazing - thanks! Now I just need to find the time to read it all. :)

Dear Dave,

I have converted the twisted tutorial into .mobi format so that I can read it on my kindle. Do you mind if I publish the ebook version of this tutorial?

Regards,
Sangeet Kumar

Hi Dave,

I have converted your tutorial into .mobi format to read in my kindle, so do you mind if I publish the ebook version of this tutorial.

Regards,
Sangeet

I've been skimming through this, but one thing I haven't seen is an example of a two-way protocol eg. a poetry server that takes a poem name as input and returns the requested poem. Do you cover this at all?

Thanks for that :) Am I also right in thinking that the factories are "single use only", eg. once you use a PoetryClientFactory to get one poem, you can't use it again?

great articles! far better than in my opinion. you should write a book with the name , lol.

great articles! far better than 'twisted essentials' in my opinion. you should write a book with the name 'asyn-model and twisted', lol.

You might want to highlight the code in chapters using this: http://www.neathighlighter.com/

Thanks for your work on "Twisted Introduction".
Is there up to date PDF version available?

(I was directed here from #python IRC channel.)

Joliprint looses formatting of code completely making it unreadable. I tried readability addon but it makes code look bad too.

I raised this issue using Feedback option on joliprint.com In the meantime I could create PDFs by cutting out unneeded elements from DOM using Firebug and then print to PDF using PDF Creator. I could send you one article so you could see if it looks ok.

I couldn't find your email anywhere on krondo.com...

Thanks a lot for this tutorial. It is incredibly well done and includes wonderful nerdy jokes. Great stuff.

Really like your introduce!

Hello Dave,

i am writing my Thesis right now about performance on the Internet and wonder which kind of licese is your tutorial using. Or in other words, am I allowed to use three images from your work. Mainly those about sync and async Modell.
Thanks for your work!

Hi, how would one handle a situation where the server closes the port? I don't get any callbacks to the connectionlost function in a factory.

Hi Dave.
First, thank you for the great job you have done. I believe, every reader of this excellent tutorial deeply appreciates the effort you put into it. *applause*

If you don't mind, I have a couple of questions regarding Twisted in particucular and programming in general. Hope you could answer them :)

Suppose you have a protocol that is almost completely symmetric, i.e. it doesn't matter who the server is and who the client is. However, a few cases when this actually matters should be handled. For this purpose one could make two distinct but very similar implementations, subclassing both ClientFactory and ServerFactory, then doing connectTCP on a ClientFactory subclass instance and listenTCP on the ServerFactory subclass instance. But this sounds quite ridiculous, doesn't it? So, alternatively, one could subclass the generic Factory class and use it for both serving and being a client. However, here arises a problem I've bumped into recently: how could one tell whether he is accepting a connection or connecting himself from within the connectionMade callback on Factory subclass?

And there's something more generic. Suppose you have a large project that has to handle both the network interaction and (ugly, huh?) a GUI. How would you structure such a project? I've come up with a solution where you have a base object which is in charge for everything and stores references to other objects which are responsible for the specific parts of the program. For instance, you could have a base class called Application and it could store references to classes like NetworkController, GUI, DiskController and so on. (Here I use the words "class" and "object" interchangeably, hope you're not going to find me and make me write "Class is not an object" 9001 times for that.) These "child" objects, in turn, store references to the Application itself, so whenever a class from the network-responsible part has to reach the GUI to tell it that the bandwidth is exhausted and immediate action has to be taken, it simply calls something like self.app.gui.TellTheUserWeAreScrewed(). However, I believe there exists a better approach but I can't just figure out what it is :( Hope you can help me.

Hi Dave, great tutorial.
I now have a running twisted daemon working fine and dandy which (listens) receives information over the internet from a number of gsm devices.

The problem is that this device's ip/port change every so often and when this hapens it leaves the open connection socket listening for more data(not in time wait or finished state) and creates another socket. eventually the machine runs out of sockets and everything starts to fail.

So far I have to restart the daemon every now and then to workarround this issue. but that's just the lamest solution. and it's not even a solution because even if a get a cron job to do it, when I scale the amount of devices it will become unpractical.

I tried to set a timeout but since there is not any specific response that the server is waiting for then timeout is never met.

So, is there a way close sockets aotumatically when they have say 180 seconds idle, or a way to tell the sockets to close after each transmission, or any other solution that I can implement, or something that I should change in my implementation. or anything you can think of that will still work when I have at least 1000 devices transmitting every 90 seconds.

I am running in a centos 5 server VPS. python 2.4

it does make sense, let me see if I get this straight,
Every time I open a new socket i statr a countdown timer (say 120 seconds) and every time I receive data I update the timer (back to 120 seconds) so that if 120 seconde shall pass with no activity whatsoever I close the socket using the loseConnection method.
sound pretty nifty,
what I don't get is how do I do that.
what should i do in the protocol init and in the data received , do you have an example?
Thank you very much for your help.

I tried to use a method called connection.timeout but it never worked for me.

ok, now if I have 128 sockets in my machine this solution will only allow me 128 clients to be connected in a two minute frame since the devices send data every 90 seconds (roughly). and when i do loose the connection the device has to login again prior to sending data. so it still makes me a bit unconfortable, isn't thre a way that many clients can share a socket. or that I could take care of saying 1000 devices even if I only have 128 sockets. or that it can drop a socket whithout droping the connection or without leeting the device know his connectio has been dropped. so that it does't feel the need to relogin all the time

BTW the part of the reactor.callLater that i don't get is, how do I update the call later, I thought that i could only make call something later and that's it. not that i could keep adjusting the timer, it a wicked idea that opens up many watchdog like possibilities.
I will check the documentation on the call later api.
forgive my typos, i need to get a new keyboard soon.

Hey Dave I am using the twisted.internet API
I did check on the call later, and the only issue would be the one with the sockets, since i am on a virtual private server (shared server) thats all i get and I tried to change the ammount of socketsit by modifiying /proc/sys/net/core/somaxconn but even as root I am unable to change it. Likely it only chageable by the administrators.

Now on the other hand the devices I am listening to areable to speack UDP also,
Do you in you expert opinion think that it would be a good idea to switch to udp?
would that allow me to have unlimited clients?
what is the tradeoff?

Is there a way for me to drop the socket without letting the device know and the making a new one 90 seconds later when it transmits again, only to drop it right after the package has been received.

I get 100 Byte packages every 90 seconds from remote devices informating me the current state of a number of variables via gprs/internet.

they do indeed, but its apin that every time they are first reconnecting, then relogin which is not needed, and then sending a package, then i do loseConnection to free the socket, but i do get the same behavior all over again, and i was expecting to just get the data package.
thats because the connection is been closes cleanly, yet if there were a way that I could drop it (not claenly close it) in such a manner that the device wouldn't know, then when it transmits again the reactor.listen on the port would listen to the incoming package open a socket, process it, and close it again, so that the socket is used only in the time it takes the package to be delivered.
I did this and i got recursive attempts to login from the device which were satisfactorily responded from the server.
that is the device would succesfully login, then the connection lost close the connection and then the device attempts to login again instead of just transmitting the data.

The devices did what i want them to when I had my first issue which was that the gprs would rotate the ip/port of the device and the device would open another socket when it sended the next data package(not trying to relogin), leaving the other socket in an established state, eventually making the server run out of sockets even when I had only one or two devices.

Hi, ricardo. If you still reading this, I'm doing the same thing with gps/gprs trackers which has mostly the same behaviour. And I found that the 'timeout' solution is quite appropriate in our situation (I use TCP connections). The only difference is that I don't know exactly refresh time interval for each device, so I use some alogirithm to predict timeout value.

BTW if you have the list of open connectoins where each connection mapped to the gps device (I do), then for every new login request from the device you could consider all old connections already in list is lost, so you could manually 'lose' them.

Another thing for ricardo - if you're limited in number of simultaneously opened TCP connections, then you could try to close connection after receiving first data packet (which is after login packet). In that case, each next portion of data will be consisting off two packets: login and data

Hi Dave,

I have just started your introduction and i am enjoying it very much.

I was also looking for something to do to give back what the python community has been giving me for the past two years. I have noticed that there is no one working in a spanish translation of the tutorial. How about i do it?

Please, feel free to contact me, i'll be waiting for your permission!

Marcial

can I translate this tutorial into Korean and publish in my blog? How can I reach you via like email?

It would have been nice if your tutorial titles did a better job describing the topics they're covering

Going straight to the top of my “to read” list, thanks for all your effort!

Twisted musculature should not be used without consent. Slow moving poetry should learn to keep the pace with novelists who are more advanced in style.

Dude, thank you for this tutorial!

I spent like a week on the twisted documentation without getting my head around it.
This really brought light into the cave.

Hey Dave!

I just started to read the blog but the link to Part 2 of the article is broken, its redirecting to the Index page. Please make the change as Part 2 has many references elsewhere. I am anxious to read the full article. :)

Utkarsh

hi dave,

I'm Jayson Pryde, and I'm new to twisted.
I've been learning Twisted via your awesome tutorial, and I am already in part 13.
I tried to implement my own client-server system, but I am encountering some hangs in the client.
It's probably because I messed up with callbacks in the deferred.
I've asked this via this link in stackoverflow:

http://stackoverflow.com/questions/42418021/asynchronous-client-in-twisted-not-sending-receiving-request-using-netstringr

It would be greatly appreciated if you can find time and take a look, and help me point out where I messed up.
So what happened in the client is I am already able to send a request, but it seems that the callback to process the request is not fired up.

Hope you can find time. thanks a lot in advance!

Hi dave,

Thank you for your nice introduction to asynchronous programming, and I'm reading chapter 4 now. I think the philosophy of asynch programming is what I need for my project so I wish to ask you few questions.

Let me explain myself first, currently I'm working on an automated web-site testing project. I decide use selenium + bs4 + unittest approach. The website I'm testing heavily relay on ajax and iframe, so I wish to develop someway better than traditional unittest case, which is separate interaction(I/O) and content assertion(RAM)

My original idea is have a main thread do all navigation and interaction, and use bs4 to parse the DOM, and then create children to assert those parsed DOM. However, after read your blog, I realize I don't really need threads to do it, I think asynchronous way suits my purpose too.

So my question is, is my understanding of asynchronous programming suits my purpose correct? is this a possible solution for automated web testing? or is there anyone else already did/doing something similar? (since I really can't find anything similar)

Thank You
Jack

Hi Dave, thank you for your answer!

Since I'm using selenium to simulate user visiting website using browser, it actually involves heavy I/O operation: fetching pages. In fact, since the data in that website is huge and queries are not that optimized, some page may take more than 30s to load.

This is why I'm looking for a method to separate the I/O operation from DOM assertion. You are right about threads, I think I don't really need thread programming. But if my code can run DOM assertion while I/O fetching next page would be nice

Furthermore, it's also about code organization. Since the website currently is under developing, although main frame remain same, the content on single page might change frequently, that's why I wish to keep them separate, for better organization and change

Thank You
Jack

Hey sir i am new in learning of twisted and i learn many thing but i am confuse at one point of my script i am making script where user just need to give the number of URL of any website then script will be extract data from the website like Title, Description and Links which are on the website and after getting links it will also extract these three details from the extracted link and so on untill the link not completed (Similar to Sitemap Crawler). this is an idea.

I write this code using Twisted Python but it complete only one cycle after that reactor stop can you please help and that how i can run the Twsited addcallback function into the loop according to certain condition did not complete thank you

Please look at the code. Thank you

2:)Extracting Code From Website

def parseHtml(html):
print("\t\t\t----------------------")
print("\t\t\t Requesting Website ")
print("\t\t\t----------------------")
# Create this Defered Variable for returning the Function Values to another function
defered = defer.Deferred()

# Now Calling another function which will be extract 3 thing from the Response(Website Code)
# reactor.callLater (Received 3 parameters 1: Delay time, Return Type, 3:parameter mean Data
reactor.callLater(3, defered.callback, html)
# defered will be return the Value to another function
return defered
pass

3:)Extracting Title,Description,Links(URL) from Source Code

def ExtractingData(response, url, crawling, Webname, tocrawl):
crawled = set([])
WebKeyword = []
WebTitle = []
WebDescription = []
keywordregex = re.compile('<meta\sname=["\']keywords["\']\scontent="\'["\']\s/>')
linkregex = re.compile('<a\shref=\'|"[\'"].?>')
print("\t\t\t--------------------------------")
print("\t\t\t Extracting Data from Website ")
print("\t\t\t--------------------------------")
msg = response
startPos = msg.find('')
print("Start position:{}".format(startPos))
if startPos != -1:
endPos = msg.find('', startPos + 7)
if endPos != -1:
title = msg[startPos + 7:endPos]
print("Title:{}".format(title))
WebTitle.append(title)
else:
WebTitle.append("N/A")
pass
pass
# Getting Description from the Website
Soup = BeautifulSoup(msg, 'html.parser')
Desc = Soup.findAll(attrs={"name": "description"})
if len(Desc) <= 0:
print("N/A")
WebDescription.append("N/A")
else:
print("Description:{}".format(Desc[0]['content'].encode('utf-8')))
WebDescription.append(Desc[0]['content'].encode('utf-8'))
pass
keywordlist = keywordregex.findall(msg)
if len(keywordlist) > 0:
keywordlist = keywordlist[0]
keywordlist = keywordlist.split(", ")
print ("Keyword:{}".format(keywordlist))
WebKeyword.append(keywordlist)
else:
WebKeyword.append("N/A")
pass
links = linkregex.findall(msg)
# print("Links:{}".format(links))
crawled.add(crawling)
for link in (links.pop(0) for _ in xrange(len(links))):
if link.startswith('/'):
link = 'http://' + url[1] + link
elif link.startswith('#'):
link = 'http://' + url[1] + url[2] + link
elif not link.startswith('http'):
link = 'http://' + url[1] + '/' + link
pass
if link not in crawled:
if Webname[0] in link:
print("Link:{}".format(link))
tocrawl.add(link)
pass
pass
pass
print("Crawled URL:{}".format(len(tocrawl)))
defers=defer.Deferred()
defers.callback(None)
return
pass

1:)Sitemap Crawler

def SitemapCrawler(Link):
tocrawl = {Link}
Webname = str(Link).replace("http://", "").split(".")
Iterator = False
while Iterator is not True:
try:
crawling = tocrawl.pop()
except:
Iterator = True
pass
url = urlparse.urlparse(crawling)
#downloadPage(crawling,"NewFile.txt")
print("Calling:{}".format(crawling))
d = getPage(crawling)
# Extracting Code or Website Response from Pages
d.addCallback(parseHtml)
# Calling Finish/Stop Function at the END of Processes
d.addCallback(ExtractingData, url=url, crawling=crawling, Webname=Webname, tocrawl=tocrawl)
reactor.run()
pass
# StoringIntoDatabase(WebTitle, WebKeyword, crawledList, WebDescription)
pass

4:)Finish Process

def finishingProcess():
print("\t\t\t Stopping the Process........")
# 3 is the delay time which will be stop all work without this we cannot stop the work
reactor.callLater(5, reactor.stop)
pass

Hi Dave,

My English is not very well, so I hope you can understand what I want to tell you.

I come from Taiwan, and I'm learning Python. I read your "Twisted Introduction", it's really helpful for me. I know someone already had translated into Simplified Chinese, but some of the content is not very ...correct.

So I have re-translated it to Traditional Chinese, also modified your sample code make they run in Python 3.I want to put re-translated articles and modified code on my blog and GitHub. But I want to get your permission first.

Your articles are very helpful to me, so I want to share them with others people who want to learn Twisted. I hope I can get your permission.

Thank you.
Shan-Ho Chan

Thank you for doing such a service for mankind man. 4 birds with one stone python/sockets/twisted/async_coding. Reaching out from Turkey.

Very good.
Verificado em Putas Rio Claro SP, Rio Claro - SP

Hi Dave,
Everyone seems to point to this as the definitive Twisted starting point, and it was very helpful up through part three, but after that I can't get any of the twisted clients to work. I'm sure something was changed in Twisted, but I'm not sure what. Do you have any ideas what might be broken, or do you know of any other good tutorials using a more recent version of Twisted?

PS: I'm using Python 3.7.5 with Twisted 19.10.0

Dave, you did awesome job, thanks you very much for sharing!

wish you all the best,
Roman