parallel test running throwing pickleError

[girish ramnani]

when running the django test in parallel using the runner.py .
_pickle.PicklingError: Can't pickle <class 'unittest.loader.ModuleImportFailure'>: attribute lookup ModuleImportFailure on unittest.loader failed

error is thrown, and when the runner is run using a single process. the test suite executes.

I am using a new virtualenv with django installed in editable mode.

[Simon Charette]

Hi Girish,

I think you didn't to install the `tblib` package[1].

Cheers,
Simon

[1] https://docs.djangoproject.com/en/1.9/ref/django-admin/#cmdoption-test--parallel

[girish ramnani]

Sorry i forgot to mention that, i have installed the tblib. this is my pip list output

Django (1.10.dev20160216200705, /home/girish/Documents/Github/django)
fancycompleter (0.4)
ordereddict (1.1)
pdbpp (0.8.3)
pip (8.0.3)
py (1.4.31)
Pygments (2.1.2)
pytest (2.8.7)
setuptools (18.2)
tblib (1.2.0)
wheel (0.24.0)
wmctrl (0.3)

[Harry Moreno]

Any progress made on this front? I'm seeing this too in a Django 1.9 app using python 2.7

[Tim Graham]

A sample project to reproduce would be helpful.

[Jonas Trappenberg]

You can debug by adding code like this to django/test/runner.py just after it determined the args (around line 315 in django 1.9):

import pickle
for arg in args:
    print(arg)
    pickle.dumps(arg)

I found that pickle raises the same exceptions as cPickle, but, since it's all Python, lets you set breakpoints around the exception to find out what's going on.

The ModuleImportFailure for example comes from python's system-wide unittest/loader.py from a function called _make_failed_import_test where it's dynamically created and hence can't be pickled. Setting a breakpoint inside that method around the error message should give you the required pointers to fix the import and thus the pickle exception.

For anyone else coming here and debugging pickle exceptions: another failure I ran into was from calling unittest.TestCase.addCleanup() with an instancemethod, e.g. something like self.addCleanup(self.remove_things). Making self.remove_things a global function fixed this too.

Hope this helps!

[komu wairagu]

This can also occur if 

- you are running your tests in parallel (eg; python manage.py test --parallel=3`)

-  and there's an import error.

In such a case; your test runner is unable to report the import error and instead reports a _pickle.PicklingError.

This happens because of this bug(https://bugs.python.org/issue31297) in python. The bug appears to have been fixed in python2 but was NOT backported to python2.7

To detect the actual import error; run your tests in sequence(--parallel=1)