gevent + urllib2 blocking
59 views
Skip to first unread message
月忧茗
Feb 22, 2013, 10:37:58 AM2/22/13
to gev...@googlegroups.com
Hi,
One problem, thanks for help.
import gevent.monkey
gevent.monkey.match_all()
from lxml import etree
# I using xpath parse the html
def _get(p):
url = BUILD_URL(p)
html = urllib2.urlopen(url)
# RUN AT HERE AND BLOCKING
# ver1
tree = etree.parse(html, parse)
# ver2
# tree = etree.fromstring(html.read(), parse)
....
workers = [gevent.spawn(_get, i) for i in xrange(1, 100)]
gevent.joinall(workers)
When using ver1, etree.parse(html, parse)
I noticed there are only two greenlets runs at "RUN AT HERE AND BLOCKING"
others are not running!
But, when I using ver2, this code will running normally as expected.
Why this ?
One problem, thanks for help.
import gevent.monkey
gevent.monkey.match_all()
from lxml import etree
# I using xpath parse the html
def _get(p):
url = BUILD_URL(p)
html = urllib2.urlopen(url)
# RUN AT HERE AND BLOCKING
# ver1
tree = etree.parse(html, parse)
# ver2
# tree = etree.fromstring(html.read(), parse)
....
workers = [gevent.spawn(_get, i) for i in xrange(1, 100)]
gevent.joinall(workers)
When using ver1, etree.parse(html, parse)
I noticed there are only two greenlets runs at "RUN AT HERE AND BLOCKING"
others are not running!
But, when I using ver2, this code will running normally as expected.
Why this ?
Reply all
Reply to author
Forward
0 new messages