Using a PEX with to run tests in parallel with pytest-xdist

[Mathis Antony]

Hi,

I've been trying to use a PEX to run tests in multiple process with with pytest-xdist but I can't seem to get it to work.

I think the reason it doesn't work is that pytest creates processes that do not inherit sys.path from the parent python process and therefore can't find any of the packages packaged with pex.

The (empty) example below shows the error I'm running into.

lulu@lulus ➜  minimal git:(master) ✗  mkvirtualenv clean
Using base prefix '/usr'
New python executable in /home/lulu/envs/clean/bin/python3
Also creating executable in /home/lulu/envs/clean/bin/python
Installing setuptools, pip, wheel...done.
virtualenvwrapper.user_scripts creating /home/lulu/envs/clean/bin/predeactivate
virtualenvwrapper.user_scripts creating /home/lulu/envs/clean/bin/postdeactivate
virtualenvwrapper.user_scripts creating /home/lulu/envs/clean/bin/preactivate
virtualenvwrapper.user_scripts creating /home/lulu/envs/clean/bin/postactivate
virtualenvwrapper.user_scripts creating /home/lulu/envs/clean/bin/get_env_details

(clean) lulu@lulus ➜  minimal git:(master) ✗  pip install pex
Collecting pex
  Using cached pex-1.1.10-py2.py3-none-any.whl
Collecting setuptools<20.11,>=2.2 (from pex)
  Using cached setuptools-20.10.1-py2.py3-none-any.whl
Installing collected packages: setuptools, pex
  Found existing installation: setuptools 21.2.2
    Uninstalling setuptools-21.2.2:
      Successfully uninstalled setuptools-21.2.2
Successfully installed pex-1.1.10 setuptools-20.10.1

(clean) lulu@lulus ➜  minimal git:(master) ✗  pex pytest pytest-xdist -o pex.pex

(clean) lulu@lulus ➜  minimal git:(master) ✗  PEX_MODULE=pytest ./pex.pex -n 2
================================================= test session starts =================================================
platform linux -- Python 3.5.1, pytest-2.9.1, py-1.4.31, pluggy-0.3.1
rootdir: /home/lulu/snippets/pytest_docker/minimal, inifile:
plugins: xdist-1.14
gw0 I / gw1 ITraceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 3, in <module>
  File "/home/lulu/.pex/install/execnet-1.4.1-py2.py3-none-any.whl.c34b5b3a3e260c938d0a55a12f00732609d1f3a4/execnet-1.4.1-py2.py3-none-any.whl/execnet/__init__.py", line 9, in <module>
    import apipkg
ImportError: No module named 'apipkg'
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/lulu/.pex/install/pytest-2.9.1-py2.py3-none-any.whl.de80d0e0f3ea1ada1e4d4c8598120dc7ff110f32/pytest-2.9.1-py2.py3-none-any.whl/_pytest/main.py", line 92, in wrap_session
INTERNALERROR>     config.hook.pytest_sessionstart(session=session)
INTERNALERROR>   File "/home/lulu/.pex/install/pytest-2.9.1-py2.py3-none-any.whl.de80d0e0f3ea1ada1e4d4c8598120dc7ff110f32/pytest-2.9.1-py2.py3-none-any.whl/_pytest/vendored_packages/pluggy.py", line 724, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/home/lulu/.pex/install/pytest-2.9.1-py2.py3-none-any.whl.de80d0e0f3ea1ada1e4d4c8598120dc7ff110f32/pytest-2.9.1-py2.py3-none-any.whl/_pytest/vendored_packages/pluggy.py", line 338, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/lulu/.pex/install/pytest-2.9.1-py2.py3-none-any.whl.de80d0e0f3ea1ada1e4d4c8598120dc7ff110f32/pytest-2.9.1-py2.py3-none-any.whl/_pytest/vendored_packages/pluggy.py", line 333, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/home/lulu/.pex/install/pytest-2.9.1-py2.py3-none-any.whl.de80d0e0f3ea1ada1e4d4c8598120dc7ff110f32/pytest-2.9.1-py2.py3-none-any.whl/_pytest/vendored_packages/pluggy.py", line 596, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/lulu/.pex/install/pytest_xdist-1.14-py2.py3-none-any.whl.7cac688e4a81cde40d97f1b945f5875338cf77ba/pytest_xdist-1.14-py2.py3-none-any.whl/xdist/dsession.py", line 509, in pytest_sessionstart
INTERNALERROR>     nodes = self.nodemanager.setup_nodes(putevent=self.queue.put)
INTERNALERROR>   File "/home/lulu/.pex/install/pytest_xdist-1.14-py2.py3-none-any.whl.7cac688e4a81cde40d97f1b945f5875338cf77ba/pytest_xdist-1.14-py2.py3-none-any.whl/xdist/slavemanage.py", line 48, in setup_nodes
INTERNALERROR>     nodes.append(self.setup_node(spec, putevent))
INTERNALERROR>   File "/home/lulu/.pex/install/pytest_xdist-1.14-py2.py3-none-any.whl.7cac688e4a81cde40d97f1b945f5875338cf77ba/pytest_xdist-1.14-py2.py3-none-any.whl/xdist/slavemanage.py", line 52, in setup_node
INTERNALERROR>     gw = self.group.makegateway(spec)
INTERNALERROR>   File "/home/lulu/.pex/install/execnet-1.4.1-py2.py3-none-any.whl.c34b5b3a3e260c938d0a55a12f00732609d1f3a4/execnet-1.4.1-py2.py3-none-any.whl/execnet/multi.py", line 128, in makegateway
INTERNALERROR>     gw = gateway_bootstrap.bootstrap(io, spec)
INTERNALERROR>   File "/home/lulu/.pex/install/execnet-1.4.1-py2.py3-none-any.whl.c34b5b3a3e260c938d0a55a12f00732609d1f3a4/execnet-1.4.1-py2.py3-none-any.whl/execnet/gateway_bootstrap.py", line 92, in bootstrap
INTERNALERROR>     bootstrap_import(io, spec)
INTERNALERROR>   File "/home/lulu/.pex/install/execnet-1.4.1-py2.py3-none-any.whl.c34b5b3a3e260c938d0a55a12f00732609d1f3a4/execnet-1.4.1-py2.py3-none-any.whl/execnet/gateway_bootstrap.py", line 27, in bootstrap_import
INTERNALERROR>     s = io.read(1)
INTERNALERROR>   File "/home/lulu/.pex/install/execnet-1.4.1-py2.py3-none-any.whl.c34b5b3a3e260c938d0a55a12f00732609d1f3a4/execnet-1.4.1-py2.py3-none-any.whl/execnet/gateway_base.py", line 389, in read
INTERNALERROR>     "expected %d bytes, got %d" % (numbytes, len(buf)))
INTERNALERROR> EOFError: expected 1 bytes, got 0

(clean) lulu@lulus ➜  minimal git:(master) ✗  PEX_MODULE=pytest ./pex.pex -n 0
================================================= test session starts =================================================
platform linux -- Python 3.5.1, pytest-2.9.1, py-1.4.31, pluggy-0.3.1
rootdir: /home/lulu/snippets/pytest_docker/minimal, inifile:
plugins: xdist-1.14
collected 0 items


============================================ no tests ran in 0.00 seconds =============================================

Maybe the "closest" I got to making this run is with the following command:

lulu@lulus ➜  minimal git:(master) ✗  PEX_MODULE=pytest ./pex.pex --dist=load --tx "2*popen//python=python pex.pex"
================================================= test session starts =================================================
platform linux -- Python 3.5.1, pytest-2.9.1, py-1.4.31, pluggy-0.3.1
rootdir: /home/lulu/snippets/pytest_docker/minimal/2*popen, inifile:
plugins: xdist-1.14
gw0 I / gw1 ICould not open -u in the environment [pex.pex]: [Errno 2] No such file or directory: '-u'
gw0 C / gw1 IINTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/lulu/.pex/install/pytest-2.9.1-py2.py3-none-any.whl.de80d0e0f3ea1ada1e4d4c8598120dc7ff110f32/pytest-2.9.1-py2.py3-none-any.whl/_pytest/main.py", line 92, in wrap_session
INTERNALERROR>     config.hook.pytest_sessionstart(session=session)
INTERNALERROR>   File "/home/lulu/.pex/install/pytest-2.9.1-py2.py3-none-any.whl.de80d0e0f3ea1ada1e4d4c8598120dc7ff110f32/pytest-2.9.1-py2.py3-none-any.whl/_pytest/vendored_packages/pluggy.py", line 724, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/home/lulu/.pex/install/pytest-2.9.1-py2.py3-none-any.whl.de80d0e0f3ea1ada1e4d4c8598120dc7ff110f32/pytest-2.9.1-py2.py3-none-any.whl/_pytest/vendored_packages/pluggy.py", line 338, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/lulu/.pex/install/pytest-2.9.1-py2.py3-none-any.whl.de80d0e0f3ea1ada1e4d4c8598120dc7ff110f32/pytest-2.9.1-py2.py3-none-any.whl/_pytest/vendored_packages/pluggy.py", line 333, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/home/lulu/.pex/install/pytest-2.9.1-py2.py3-none-any.whl.de80d0e0f3ea1ada1e4d4c8598120dc7ff110f32/pytest-2.9.1-py2.py3-none-any.whl/_pytest/vendored_packages/pluggy.py", line 596, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/lulu/.pex/install/pytest_xdist-1.14-py2.py3-none-any.whl.7cac688e4a81cde40d97f1b945f5875338cf77ba/pytest_xdist-1.14-py2.py3-none-any.whl/xdist/dsession.py", line 509, in pytest_sessionstart
INTERNALERROR>     nodes = self.nodemanager.setup_nodes(putevent=self.queue.put)
INTERNALERROR>   File "/home/lulu/.pex/install/pytest_xdist-1.14-py2.py3-none-any.whl.7cac688e4a81cde40d97f1b945f5875338cf77ba/pytest_xdist-1.14-py2.py3-none-any.whl/xdist/slavemanage.py", line 48, in setup_nodes
INTERNALERROR>     nodes.append(self.setup_node(spec, putevent))
INTERNALERROR>   File "/home/lulu/.pex/install/pytest_xdist-1.14-py2.py3-none-any.whl.7cac688e4a81cde40d97f1b945f5875338cf77ba/pytest_xdist-1.14-py2.py3-none-any.whl/xdist/slavemanage.py", line 57, in setup_node
INTERNALERROR>     node.setup()
INTERNALERROR>   File "/home/lulu/.pex/install/pytest_xdist-1.14-py2.py3-none-any.whl.7cac688e4a81cde40d97f1b945f5875338cf77ba/pytest_xdist-1.14-py2.py3-none-any.whl/xdist/slavemanage.py", line 235, in setup
INTERNALERROR>     self.channel = self.gateway.remote_exec(xdist.remote)
INTERNALERROR>   File "/home/lulu/.pex/install/execnet-1.4.1-py2.py3-none-any.whl.c34b5b3a3e260c938d0a55a12f00732609d1f3a4/execnet-1.4.1-py2.py3-none-any.whl/execnet/gateway.py", line 126, in remote_exec
INTERNALERROR>     channel = self.newchannel()
INTERNALERROR>   File "/home/lulu/.pex/install/execnet-1.4.1-py2.py3-none-any.whl.c34b5b3a3e260c938d0a55a12f00732609d1f3a4/execnet-1.4.1-py2.py3-none-any.whl/execnet/gateway_base.py", line 1002, in newchannel
INTERNALERROR>     return self._channelfactory.new()
INTERNALERROR>   File "/home/lulu/.pex/install/execnet-1.4.1-py2.py3-none-any.whl.c34b5b3a3e260c938d0a55a12f00732609d1f3a4/execnet-1.4.1-py2.py3-none-any.whl/execnet/gateway_base.py", line 779, in new
INTERNALERROR>     raise IOError("connexion already closed: %s" % (self.gateway,))
INTERNALERROR> OSError: connexion already closed: <Gateway id='gw0' not-receiving, thread model, 0 active channels>

Which I think fails because somewhere -u is passed to the worker process, as in,

lulu@lulus ➜  minimal git:(master) ✗  python pex.pex -u
Could not open -u in the environment [pex.pex]: [Errno 2] No such file or directory: '-u'

Thanks for reading,

Mathis

[Kris Wilson]

you need a way to bootstrap the pex environment and pass that on in the context of the new worker processes.

to demonstrate what I mean, here's a super crude hack that makes this work via pytest's `conftest.py`:

[illuminati ~]$ pex pytest pytest-xdist -o pex.pex -c py.test

[illuminati ~]$ cat >conftest.py

import os, sys

def activate_pex(pex_path):

  sys.path.insert(0, os.path.abspath(os.path.join(pex_path, '.bootstrap')))

  from _pex import pex_bootstrapper

  pex_bootstrapper.bootstrap_pex_env(pex_path)

def pytest_configure(config):

  activate_pex('/Users/kwilson/pex.pex')

  os.environ['PYTHONPATH'] = ':'.join(sys.path)

[illuminati ~]$ ./pex.pex -n 2

========================================================= test session starts ==========================================================

platform darwin -- Python 2.7.10, pytest-2.9.1, py-1.4.31, pluggy-0.3.1

rootdir: /Users/kwilson, inifile: pytest.ini

plugins: xdist-1.14

gw0 ok / gw1 ok

[Mathis Antony]

Thanks for the quick help Kris. Was able to get this to work with your conftest.py file and an environment variable.

import os

import sys

def activate_pex(pex_path):

    sys.path.insert(0, os.path.abspath(os.path.join(pex_path, '.bootstrap')))

    from _pex import pex_bootstrapper

    pex_bootstrapper.bootstrap_pex_env(pex_path)

def pytest_configure(config):

    pex_path = os.environ.get('PEXPATH')

    if pex_path:

        activate_pex(pex_path)

        os.environ['PYTHONPATH'] = ':'.join(sys.path)