trouble passing lxml.etree into mpm worker method
slestak989
I am trying to parse an etree once, and use it for several xpath
queries. it works well until I try to to use multiprocessing. I have
purposefully set num_workers to 1 to get mpm working, then I will
increase it to 2 or 4.
I create consumers, use them, get results, terminate them, then spin
up new consumers in the same def.
The code is at http://paste.pocoo.org/show/256317/
Class ssXpathTask() is the task item being pushed into the run queue.
It is trying, just the tree is not usable when it gets into the
ss_xpaths() method.
With no mpm, this works fine, albeit slowly.
Steve Romanow
Creating 4 consumers
Parsing Vbook file, 21673 unique cin's found.
18 have no images.
sample = 0162299
Creating 4 consumers
19 in the queue
Process Consumer-5:
Traceback (most recent call last):
File "C:\Python26\lib\multiprocessing\process.py", line 232, in
_bootstrap
self.run()
File
"C:\Users\sromanow\Documents\projects\xpo\modules\validation\validate.py",
line 28, in run
answer = next_task()
File
"C:\Users\sromanow\Documents\projects\xpo\modules\validation\validate.py",
line 45, in __call__
return ss_xpaths(self.tree,self.cin)
File
"C:\Users\sromanow\Documents\projects\xpo\modules\validation\validate.py",
line 58, in ss_xpaths
result = tree.xpath(item_descr_expr, cin = cin)
File "lxml.etree.pyx", line 1829, in lxml.etree._ElementTree.xpath
(src/lxml/lxml.etree.c:41641)
File "lxml.etree.pyx", line 1550, in
lxml.etree._ElementTree._assertHasRoot (src/lxml/lxml.etree.c:39095)
AssertionError: ElementTree not initialized, missing root
Steve Romanow
cannot pass an Etree by reference across processes.
I ended up modifying my xpath to work on the parent node of my search
results and the speed increase made it where mulitprocessing was not
necessary.