Dave
unread,Jan 20, 2012, 12:42:43 AM1/20/12Sign in to reply to author
Sign in to forward
You do not have permission to delete messages in this group
Either email addresses are anonymous for this group or you need the view member email addresses permission to view the original message
to gevent: coroutine-based Python network library
I am running 20 threads downloading various webpages through https
proxies and occasionally receive the following error:
Traceback (most recent call last):
File "/home/dave/projects/gks/env-gks/local/lib/python2.7/site-
packages/gevent/greenlet.py", line 390, in run
result = self._run(*self.args, **self.kwargs)
File "/home/dave/projects/gks/src/gksaccount.py", line 45, in scrape
raw_csv = self._csv_download(post_data, headers)
File "/home/dave/projects/gks/src/account.py", line 245, in
_csv_download
proxy=self.proxy)
File "/home/dave/projects/gks/env-gks/local/lib/python2.7/site-
packages/webscraping/download.py", line 128, in get
html = self.fetch(url, headers=headers, data=data, proxy=proxy,
user_agent=user_agent, opener=opener)
File "/home/dave/projects/gks/env-gks/local/lib/python2.7/site-
packages/webscraping/download.py", line 202, in fetch
response = opener.open(urllib2.Request(url, data, headers))
File "/usr/lib/python2.7/urllib2.py", line 391, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 409, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 369, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1193, in https_open
return self.do_open(httplib.HTTPSConnection, req)
File "/usr/lib/python2.7/urllib2.py", line 1160, in do_open
raise URLError(err)
URLError: <urlopen error [Errno 8] _ssl.c:499: EOF occurred in
violation of protocol>
<Greenlet at 0xb5689ac: <bound method GKSAccount.scrape of
<gksaccount.GKSAccount object at 0x932024c>>> failed with URLError
I did some Google searching a read about the 1024 file descriptor
limit and another library "leaking" sockets. It might be a red
herring, but I'm throwing it out there.
Thanks for the great library and help!
Dave