Twisted Introduction

Hey sir i am new in learning of twisted and i learn many thing but i am confuse at one point of my script i am making script where user just need to give the number of URL of any website then script will be extract data from the website like Title, Description and Links which are on the website and after getting links it will also extract these three details from the extracted link and so on untill the link not completed (Similar to Sitemap Crawler). this is an idea.

I write this code using Twisted Python but it complete only one cycle after that reactor stop can you please help and that how i can run the Twsited addcallback function into the loop according to certain condition did not complete thank you

Please look at the code. Thank you

2:)Extracting Code From Website

def parseHtml(html):
print(“\t\t\t———————-“)
print(“\t\t\t Requesting Website “)
print(“\t\t\t———————-“)
# Create this Defered Variable for returning the Function Values to another function
defered = defer.Deferred()

# Now Calling another function which will be extract 3 thing from the Response(Website Code)
# reactor.callLater (Received 3 parameters 1: Delay time, Return Type, 3:parameter mean Data
reactor.callLater(3, defered.callback, html)
# defered will be return the Value to another function
return defered
pass


3:)Extracting Title,Description,Links(URL) from Source Code

def ExtractingData(response, url, crawling, Webname, tocrawl):
crawled = set([])
WebKeyword = []
WebTitle = []
WebDescription = []
keywordregex = re.compile(‘<meta\sname=[“\’]keywords[“\’]\scontent=“\’[“\’]\s/>’)
linkregex = re.compile(‘<a\shref=\’|”[\'”].?>’)
print(“\t\t\t——————————–“)
print(“\t\t\t Extracting Data from Website “)
print(“\t\t\t——————————–“)
msg = response
startPos = msg.find(”)
print(“Start position:{}”.format(startPos))
if startPos != -1:
endPos = msg.find(”, startPos + 7)
if endPos != -1:
title = msg[startPos + 7:endPos]
print(“Title:{}”.format(title))
WebTitle.append(title)
else:
WebTitle.append(“N/A”)
pass
pass
# Getting Description from the Website
Soup = BeautifulSoup(msg, ‘html.parser’)
Desc = Soup.findAll(attrs={“name”: “description”})
if len(Desc) <= 0:
print(“N/A”)
WebDescription.append(“N/A”)
else:
print(“Description:{}”.format(Desc[0][‘content’].encode(‘utf-8’)))
WebDescription.append(Desc[0][‘content’].encode(‘utf-8’))
pass
keywordlist = keywordregex.findall(msg)
if len(keywordlist) > 0:
keywordlist = keywordlist[0]
keywordlist = keywordlist.split(“, “)
print (“Keyword:{}”.format(keywordlist))
WebKeyword.append(keywordlist)
else:
WebKeyword.append(“N/A”)
pass
links = linkregex.findall(msg)
# print(“Links:{}”.format(links))
crawled.add(crawling)
for link in (links.pop(0) for _ in xrange(len(links))):
if link.startswith(‘/’):
link = ‘http://’ + url[1] + link
elif link.startswith(‘#’):
link = ‘http://’ + url[1] + url[2] + link
elif not link.startswith(‘http’):
link = ‘http://’ + url[1] + ‘/’ + link
pass
if link not in crawled:
if Webname[0] in link:
print(“Link:{}”.format(link))
tocrawl.add(link)
pass
pass
pass
print(“Crawled URL:{}”.format(len(tocrawl)))
defers=defer.Deferred()
defers.callback(None)
return
pass

1:)Sitemap Crawler

def SitemapCrawler(Link):
tocrawl = {Link}
Webname = str(Link).replace(“http://”, “”).split(“.”)
Iterator = False
while Iterator is not True:
try:
crawling = tocrawl.pop()
except:
Iterator = True
pass
url = urlparse.urlparse(crawling)
#downloadPage(crawling,”NewFile.txt”)
print(“Calling:{}”.format(crawling))
d = getPage(crawling)
# Extracting Code or Website Response from Pages
d.addCallback(parseHtml)
# Calling Finish/Stop Function at the END of Processes
d.addCallback(ExtractingData, url=url, crawling=crawling, Webname=Webname, tocrawl=tocrawl)
reactor.run()
pass
# StoringIntoDatabase(WebTitle, WebKeyword, crawledList, WebDescription)
pass

4:)Finish Process

def finishingProcess():
print(“\t\t\t Stopping the Process……..”)
# 3 is the delay time which will be stop all work without this we cannot stop the work
reactor.callLater(5, reactor.stop)
pass

Thank you for doing such a service for mankind man. 4 birds with one stone python/sockets/twisted/async_coding. Reaching out from Turkey.