Unit testing multiprocessing code on Windows
Matt Chaput
multiprocessing on Windows?
The problem is that with both "python setup.py tests" and "nosetests",
when they get to testing any code that starts Processes they spawn
multiple copies of the testing suite (i.e. the new processes start
running tests as if they were started with "python setup.py
tests"/"nosetests"). The test runner in PyDev works properly.
Maybe multiprocessing is starting new Windows processes by copying the
command line of the current process? But if the command line is
"nosetests", it's a one way ticket to an infinite explosion of processes.
Any thoughts?
Thanks,
Matt
Matt Chaput
multiprocessing on Windows?
The problem is that with both "python setup.py tests" and "nosetests",
when they get to a multiprocessing test they spawn multiple copies of
the testing suite. The test runner in PyDev works properly.
Maybe multiprocessing is starting new Windows processes by copying the
command line of the current process, but if the command line is
phi...@semanchuk.com
> Does anyone know the "right" way to write a unit test for code that
> uses multiprocessing on Windows?
>
> The problem is that with both "python setup.py tests" and
> "nosetests", when they get to testing any code that starts Processes
> they spawn multiple copies of the testing suite (i.e. the new
> processes start running tests as if they were started with "python
> setup.py tests"/"nosetests"). The test runner in PyDev works properly.
>
> Maybe multiprocessing is starting new Windows processes by copying
> the command line of the current process? But if the command line is
> "nosetests", it's a one way ticket to an infinite explosion of
> processes.
Hi Matt,
I assume you're aware of this documentation, especially the item
entitled "Safe importing of main module"?
http://docs.python.org/release/2.6.6/library/multiprocessing.html#windows
HTH
P
Terry Reedy
> Does anyone know the "right" way to write a unit test for code that uses
> multiprocessing on Windows?
I would start with Lib/test/test_multiprocessing.
--
Terry Jan Reedy
Matt Chaput
Good idea, but on the one hand it doesn't seem to be doing anything
special, and on the other hand it seems to do it's own things like not
having its test cases inherit from unittest.TestCase. I also don't know
if the Python devs start it with distutils or nosetests, which are the
ones I'm having a problem with. For example, starting my test suite
inside PyDev doesn't show the bug.
My test code isn't doing anything unusual... this is pretty much all I
do to trigger the bug. (None of the imported code has anything to do
with processes.)
from __future__ import with_statement
import unittest
import random
from whoosh import fields, query
from whoosh.support.testing import TempIndex
try:
import multiprocessing
except ImportError:
multiprocessing = None
if multiprocessing:
class MPFCTask(multiprocessing.Process):
def __init__(self, storage, indexname):
multiprocessing.Process.__init__(self)
self.storage = storage
self.indexname = indexname
def run(self):
ix = self.storage.open_index(self.indexname)
with ix.searcher() as s:
r = s.search(query.Every(), sortedby="key", limit=None)
result = "".join([h["key"] for h in r])
assert result ==
"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz", result
class TestSorting(unittest.TestCase):
def test_mp_fieldcache(self):
if not multiprocessing:
return
schema = fields.Schema(key=fields.KEYWORD(stored=True))
with TempIndex(schema, "mpfieldcache") as ix:
domain =
list(u"abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ")
random.shuffle(domain)
w = ix.writer()
for char in domain:
w.add_document(key=char)
w.commit()
tasks = [MPFCTask(ix.storage, ix.indexname) for _ in xrange(4)]
for task in tasks:
task.start()
for task in tasks:
task.join()
if __name__ == '__main__':
unittest.main()
Matt Chaput
> Hi Matt,
> I assume you're aware of this documentation, especially the item
> entitled "Safe importing of main module"?
>
> http://docs.python.org/release/2.6.6/library/multiprocessing.html#windows
Yes, but the thing is my code isn't __main__, my unittest classes are
being loaded by setup.py test or nosetests. And while I'm assured
multiprocessing doesn't duplicate the original command line, what I get
sure looks like it, because if I use "python setup.py test" that command
seems to be re-run for every Process that starts, but if I use
"nosetests" then *that* seems to be re-run for every Process.
Matt
David
>
> The problem is that with both "python setup.py tests" and "nosetests",
> Maybe multiprocessing is starting new Windows processes by copying the
> command line of the current process? But if the command line is
> "nosetests", it's a one way ticket to an infinite explosion of processes.
You can adapt this code to inhibit test execution if another process is
testing yet.
http://code.activestate.com/recipes/474070-creating-a-single-instance-application/