cannot build pex that depends on scikit-image

Jesse Hersch

unread,

Jun 13, 2018, 1:10:26 PM6/13/18

to Pants Developers

Hi folks,

I am unable to build a pex that uses the scikit-image package: https://pypi.org/project/scikit-image

What I'm running:

$ ./pants --version

1.7.0rc1

$ python --version

Python 3.6.3

$ cat /etc/os-release

NAME="Ubuntu"

VERSION="17.04 (Zesty Zapus)"

...

This is the entry for scikit-image and cython in 3rdparty:

python_requirement_library(

name='scikit-image',

requirements=[python_requirement(name='scikit-image', requirement='scikit-image==0.14.0')]

)

python_requirement_library(

name='cython',

requirements=[python_requirement(name='cython', requirement='cython==0.28.2')]

)

Here's my BUILD file:

python_library(

name='landsat-lib-test',

sources=['landsat.py'],

dependencies=[

'3rdparty:scikit-image',

]

)

And here's landsat.py:

import skimage

print(skimage)

When I try to create the binary:

$ ./pants binary libraries/landsat-lib-test:

10:05:16 00:00 [main]

(To run a reporting server: ./pants server)

10:05:16 00:00 [setup]

10:05:16 00:00 [parse]

Executing tasks in goals: jvm-platform-validate -> bootstrap -> imports -> unpack-jars -> deferred-sources -> gen -> pyprep -> resolve -> resources -> compile -> binary

10:05:16 00:00 [jvm-platform-validate]

10:05:16 00:00 [bootstrap]

10:05:16 00:00 [substitute-aliased-targets]

10:05:16 00:00 [jar-dependency-management]

10:05:16 00:00 [bootstrap-jvm-tools]

10:05:16 00:00 [provide-tools-jar]

10:05:16 00:00 [imports]

10:05:16 00:00 [ivy-imports]

10:05:16 00:00 [unpack-jars]

10:05:16 00:00 [deferred-sources]

10:05:16 00:00 [gen]

10:05:16 00:00 [antlr-java]

10:05:16 00:00 [antlr-py]

10:05:16 00:00 [jaxb]

10:05:16 00:00 [protoc]

10:05:16 00:00 [ragel]

10:05:16 00:00 [thrift-java]

10:05:16 00:00 [thrift-py]

10:05:16 00:00 [wire]

10:05:16 00:00 [pyprep]

10:05:16 00:00 [interpreter]

10:05:16 00:00 [build-local-dists]

10:05:16 00:00 [requirements]

10:05:16 00:00 [cache]

No cached artifacts for 1 target.

Invalidated 1 target.**** Failed to install scikit-image-0.14.0 (caused by: NonZeroExit("received exit code 1 during execution of `[u'/usr/local/bin/python3.6', '-', 'bdist_wheel', '--dist-dir=/tmp/tmptNyaMm']` while trying to execute `[u'/usr/local/bin/python3.6', '-', 'bdist_wheel', '--dist-dir=/tmp/tmptNyaMm']`",)

):

stdout:

Cython >= 0.23 not found; falling back to pre-built geometry.c

Cython >= 0.23 not found; falling back to pre-built transform.c

Cython >= 0.23 not found; falling back to pre-built interpolation.c

Cython >= 0.23 not found; falling back to pre-built _draw.c

Cython >= 0.23 not found; falling back to pre-built corner_cy.c

Cython >= 0.23 not found; falling back to pre-built censure_cy.c

Cython >= 0.23 not found; falling back to pre-built orb_cy.c

Cython >= 0.23 not found; falling back to pre-built brief_cy.c

Cython >= 0.23 not found; falling back to pre-built _texture.c

Cython >= 0.23 not found; falling back to pre-built _hessian_det_appx.c

Cython >= 0.23 not found; falling back to pre-built _hoghistogram.c

stderr:

Partial import of skimage during the build process.

Traceback (most recent call last):

File "/tmp/tmpt4RwSB/scikit-image-0.14.0/skimage/_build.py", line 30, in cython

from Cython import __version__

ModuleNotFoundError: No module named 'Cython'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "<stdin>", line 7, in <module>

File "setup.py", line 149, in <module>

**extra

File "/usr/local/lib/python3.6/site-packages/numpy/distutils/core.py", line 135, in setup

config = configuration()

File "setup.py", line 70, in configuration

config.add_subpackage('skimage')

File "/usr/local/lib/python3.6/site-packages/numpy/distutils/misc_util.py", line 1029, in add_subpackage

caller_level = 2)

File "/usr/local/lib/python3.6/site-packages/numpy/distutils/misc_util.py", line 998, in get_subpackage

caller_level = caller_level + 1)

File "/usr/local/lib/python3.6/site-packages/numpy/distutils/misc_util.py", line 935, in _get_configuration_from_setup_py

config = setup_module.configuration(*args)

File "skimage/setup.py", line 14, in configuration

config.add_subpackage('feature')

File "/usr/local/lib/python3.6/site-packages/numpy/distutils/misc_util.py", line 1029, in add_subpackage

caller_level = 2)

File "/usr/local/lib/python3.6/site-packages/numpy/distutils/misc_util.py", line 998, in get_subpackage

caller_level = caller_level + 1)

File "/usr/local/lib/python3.6/site-packages/numpy/distutils/misc_util.py", line 935, in _get_configuration_from_setup_py

config = setup_module.configuration(*args)

File "skimage/feature/setup.py", line 22, in configuration

cython(['_haar.pyx'], working_path=base_path)

File "/tmp/tmpt4RwSB/scikit-image-0.14.0/skimage/_build.py", line 42, in cython

% CYTHON_VERSION)

RuntimeError: Cython >= 0.23 is required to build scikit-image from git checkout

Waiting for background workers to finish.

10:05:18 00:02 [complete]

FAILURE

Exception caught: (<class 'pex.resolver.Untranslateable'>)

Exception message: Package SourcePackage(u'file:///home/jhersch/src/gemini/.pants.d/python-setup/resolved_requirements/CPython-3.6.3/scikit-image-0.14.0.tar.gz') is not translateable by ChainedTranslator(WheelTranslator, EggTranslator, SourceTranslator)

I do have cython installed in the environment where I run pants:

$ pip freeze | grep Cython

Cython==0.28.2

It doesn't help if I add cython to the BUILD file like this, I get the same error in that case:

python_library(

name='landsat-lib-test',

sources=['landsat.py'],

dependencies=[

'3rdparty:cython',

'3rdparty:scikit-image',

]

)

How do I tell pants where cython is located? Is this use case supported?

Thanks,

Jesse

Jesse Hersch

unread,

Jun 13, 2018, 1:21:23 PM6/13/18

to Pants Developers

It always helps to post a question in order to answer it yourself I suppose. Problem was that the python instance at /usr/local/bin/python3.6 did not have cython. Installed it there and got past this particular issue.

John Sirois

unread,

Jun 13, 2018, 2:31:48 PM6/13/18

to Jesse Hersch, pants-devel

Ok, thanks for figuring that out. It sounds like scikit-image is missing a dependency. You can reproducibly work around this by adding a dependency on cython to your landsat-lib-test library.

Kris Wilson

unread,

Jun 13, 2018, 2:51:47 PM6/13/18

to John Sirois, Jesse Hersch, pants-devel

does this improve for you at all if you use pants 1.8.0dev3 (with cython explicitly uninstalled)?

I'm pretty sure that at least on the surface this is a manylinux resolution thing. because pants 1.7.0 can't resolve the existing manylinux bdists on pypi for scikit-image, pex falls back to attempting to build that from the sdist via a `python3.6 setup.py bdist_wheel` against those sources. this build process requires Cython (but does not appear to be declared as such in scikit-image's setup.py install_requires fwict) - which is why manually installing that makes it work.

but: with pants 1.8.0 and manylinux support, this would just select the existing manylinux bdist in lieu of this fallback build step which would obviate the build issue there (and be faster and more reliable to boot).

Jesse Hersch

unread,

Jun 13, 2018, 7:28:13 PM6/13/18

to Pants Developers

Didn't realize 1.8.0 has manylinux support, that's great. However I get an error when trying it:

$ ./pants --version

INFO] Resolving new plugins...:

pantsbuild.pants.contrib.node==1.8.0.dev3

Exception caught: (<class 'pex.resolver.Unsatisfiable'>)

File "/home/jhersch/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0.dev3/bin/pants", line 11, in <module>

sys.exit(main())

File "/home/jhersch/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0.dev3/local/lib/python2.7/site-packages/pants/bin/pants_loader.py", line 69, in main

PantsLoader.run()

File "/home/jhersch/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0.dev3/local/lib/python2.7/site-packages/pants/bin/pants_loader.py", line 65, in run

cls.load_and_execute(entrypoint)

File "/home/jhersch/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0.dev3/local/lib/python2.7/site-packages/pants/bin/pants_loader.py", line 58, in load_and_execute

entrypoint_main()

File "/home/jhersch/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0.dev3/local/lib/python2.7/site-packages/pants/bin/pants_exe.py", line 39, in main

PantsRunner(exiter, start_time=start_time).run()

File "/home/jhersch/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0.dev3/local/lib/python2.7/site-packages/pants/bin/pants_runner.py", line 53, in run

return runner.run()

File "/home/jhersch/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0.dev3/local/lib/python2.7/site-packages/pants/bin/local_pants_runner.py", line 46, in run

self._run()

File "/home/jhersch/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0.dev3/local/lib/python2.7/site-packages/pants/bin/local_pants_runner.py", line 54, in _run

build_config = BuildConfigInitializer.get(options_bootstrapper)

File "/home/jhersch/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0.dev3/local/lib/python2.7/site-packages/pants/init/options_initializer.py", line 39, in get

cls._cached_build_config = cls(options_bootstrapper, working_set).setup()

File "/home/jhersch/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0.dev3/local/lib/python2.7/site-packages/pants/init/options_initializer.py", line 49, in __init__

self._working_set = working_set or PluginResolver(self._options_bootstrapper).resolve()

File "/home/jhersch/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0.dev3/local/lib/python2.7/site-packages/pants/init/plugin_resolver.py", line 74, in resolve

for plugin_location in self._resolve_plugin_locations():

File "/home/jhersch/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0.dev3/local/lib/python2.7/site-packages/pants/init/plugin_resolver.py", line 100, in _resolve_exact_plugin_locations

for plugin in self._resolve_plugins():

File "/home/jhersch/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0.dev3/local/lib/python2.7/site-packages/pants/init/plugin_resolver.py", line 115, in _resolve_plugins

allow_prereleases=PANTS_SEMVER.is_prerelease)

File "/home/jhersch/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0.dev3/local/lib/python2.7/site-packages/pex/resolver.py", line 453, in resolve

return resolver.resolve(resolvables_from_iterable(requirements, builder))

File "/home/jhersch/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0.dev3/local/lib/python2.7/site-packages/pex/resolver.py", line 235, in resolve

resolvable_set.merge(resolvable, packages, parent)

File "/home/jhersch/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0.dev3/local/lib/python2.7/site-packages/pex/resolver.py", line 131, in merge

self._check()

File "/home/jhersch/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0.dev3/local/lib/python2.7/site-packages/pex/resolver.py", line 123, in _check

self._synthesize_parents(name)

Exception message: Could not satisfy all requirements for pantsbuild.pants==1.8.0.dev3:

pantsbuild.pants==1.8.0.dev3(from: pantsbuild.pants.contrib.node==1.8.0.dev3)

Jesse Hersch

unread,

Jun 14, 2018, 11:19:34 AM6/14/18

to Pants Developers

Nevermind I had a conflicting set of dependency requirements in 3rdparty. My fault.

John Sirois

unread,

Jun 14, 2018, 12:39:07 PM6/14/18

to Jesse Hersch, pants-devel

Sorry to fork this thread, but the error below is reproducible and highlights a bug: https://github.com/pantsbuild/pants/issues/5958

Reply all

Reply to author

Forward