AttributeError when building a python binary

[Jesse Hersch]

Hi folks!

Just starting out with pants and having some difficulty.  This is on linux, pants version 1.2.0.

I am trying to build a python project, say package A. The package A uses an internal python library, package B.

If A does not specify B in the BUILD file, it builds ok:

------------------------------------
./pants clean-all & ./pants binary services/satellite_service/service:satellite-service
<snip>
16:42:59 00:01   [binary]
16:42:59 00:01     [binary-jvm-prep-command]
16:42:59 00:01       [jvm_prep_command]
16:42:59 00:01     [binary-prep-command]
16:42:59 00:01     [python-binary-create]
                   pants_backend_python_tasks_python_binary_create_PythonBinaryCreate will read from local artifact cache at /home/jhersch/src/gemini/.cache/pants_backend_python_tasks_python_binary_create_PythonBinaryCreate
16:42:59 00:01       [cache].
                     pants_backend_python_tasks_python_binary_create_PythonBinaryCreate will write to local artifact cache at /home/jhersch/src/gemini/.cache/pants_backend_python_tasks_python_binary_create_PythonBinaryCreate
                     Caching artifacts for 1 target.DEBUG] Skipping insert of existing artifact: CacheKey(id=u'services.satellite_service.service.satellite-service', hash=u'8411bd39cf61ade20817fe04d2ddcd86c6804311-TaskIdentityFingerprintStrategy.480cf9e938bd_53c4bb9c6553')

                   Using cached artifacts for 1 target.
                   created .pants.d/binary/python-binary-create/0550b754d96b/services.satellite_service.service.satellite-service/current/satellite-service.pex
                   created pex copy dist/satellite-service.pex
16:42:59 00:01     [jvm]
16:42:59 00:01     [dup]
               Waiting for background workers to finish.
16:42:59 00:01   [complete]
               SUCCESS
------------------------------------

Now, the problem comes if I add a dependency on Package B, then I get this AttributeError. Seems like it's a bug in python that is being called by pants actually?  It's looking for
SourceFileLoader where it doesn't exist. Similar bug: http://stackoverflow.com/questions/33365471/importlib-bootstrap-has-no-attribute-sourceloader

------------------------------------
./pants clean-all & ./pants binary services/satellite_service/service:satellite-service
<snip>
16:45:04 00:01   [binary]
16:45:04 00:01     [binary-jvm-prep-command]
16:45:04 00:01       [jvm_prep_command]
16:45:04 00:01     [binary-prep-command]
16:45:04 00:01     [python-binary-create]
                   pants_backend_python_tasks_python_binary_create_PythonBinaryCreate will read from local artifact cache at /home/jhersch/src/gemini/.cache/pants_backend_python_tasks_python_binary_create_PythonBinaryCreate
16:45:04 00:01       [cache]
                   No cached artifacts for 1 target.
                   Invalidated 1 target.
                       installed /home/jhersch/src/gemini/.pants.d/python-setup/interpreters/CPython-3.5.2.tmp.972d81bc439442c99ddbbaf970ad67da/setuptools-5.4.1-py3.5.egg
                       installed /home/jhersch/src/gemini/.pants.d/python-setup/interpreters/CPython-3.5.2.tmp.972d81bc439442c99ddbbaf970ad67da/wheel-0.29.0-py3.5.egg
                       installed /home/jhersch/src/gemini/.pants.d/python-setup/interpreters/CPython-3.5.2/setuptools-5.4.1-py3.5.egg
                       installed /home/jhersch/src/gemini/.pants.d/python-setup/interpreters/CPython-3.5.2/wheel-0.29.0-py3.5.egg**** Failed to install setuptools-5.4.1 (caused by: NonZeroExit("received exit code 1 during execution of `['/usr/local/bin/python3.6', '-', 'bdist_egg', '--dist-dir=/tmp/tmpZkyyUw']`",)
):
stdout:

stderr:
Traceback (most recent call last):
  File "<stdin>", line 5, in <module>
  File "setup.py", line 19, in <module>
    exec(init_file.read(), command_ns)
  File "<string>", line 11, in <module>
  File "/tmp/tmp_eoWZZ/setuptools-5.4.1/setuptools/__init__.py", line 12, in <module>
    from setuptools.extension import Extension
  File "/tmp/tmp_eoWZZ/setuptools-5.4.1/setuptools/extension.py", line 7, in <module>
    from setuptools.dist import _get_unpatched
  File "/tmp/tmp_eoWZZ/setuptools-5.4.1/setuptools/dist.py", line 18, in <module>
    import pkg_resources
  File "/tmp/tmp_eoWZZ/setuptools-5.4.1/pkg_resources.py", line 1520, in <module>
    register_loader_type(importlib_bootstrap.SourceFileLoader, DefaultProvider)
AttributeError: module 'importlib._bootstrap' has no attribute 'SourceFileLoader'


16:45:14 00:11   [complete]
               FAILURE
Exception caught: (<class 'pex.resolver.Untranslateable'>)

Exception message: Package SourcePackage(u'https://pypi.python.org/packages/cb/55/4143b8a178e17c39c246d95dd0620e8c26c699ffc5267eb5d444b6b2429a/setuptools-5.4.1.tar.gz#md5=3540a44b90017cbb851840934156848e') is not translateable by ChainedTranslator(EggTranslator, SourceTranslator)




Here's my BUILD file for the project that fails to build, the line that causes the fail is the line with libraries/json_logger. If I remove that line it builds ok, but doesn't include the internal package of course which is a requirement for me.

------------------------------------
python_library(
  name='impl',
  sources=globs('*.py'),
  dependencies=[
    '3rdparty:alembic',
    '3rdparty:click',
    '3rdparty:Flask',
    '3rdparty:Flask-Migrate',
    '3rdparty:Flask-Script',
    '3rdparty:Flask-SQLAlchemy',
    '3rdparty:Flask-Testing',
    '3rdparty:itsdangerous',
    '3rdparty:Jinja2',
    '3rdparty:jplephem',
    '3rdparty:Mako',
    '3rdparty:MarkupSafe',
    '3rdparty:numpy',
    '3rdparty:pbr',
    '3rdparty:psycopg2',
    '3rdparty:python-editor',
    '3rdparty:sgp4',
    '3rdparty:skyfield',
    '3rdparty:SQLAlchemy',
    '3rdparty:stevedore',
    '3rdparty:toml',
    '3rdparty:Werkzeug',
    'libraries/json_logger'
  ]
)

python_binary(
  name='satellite-service',
  source='app.py',
  dependencies=[':impl']
)
------------------------------------

Thanks for looking!

Jesse

[Eric Zundel Ayers]

Jesse, could you try executing pants with a python2 binary and see if that helps?

[Jesse Hersch]

Of course I can try, but the code I want to build is python3.6.  Is it a problem for pants to build python 3 projects?

[Jesse Hersch]

ok so what does "executing pants with a python2 binary" mean exactly?  

What I tried was create a python2.7 virtualenv and then run pants in that. got the same error.

[Kris Wilson]

based on the output, it looks like pants is failing to run `python3.6 setup.py bdist_egg` against the setuptools-5.4.1 sdist during interpreter bootstrap. you could try fetching setuptools-5.4.1.tar.gz and running the same command to attempt an external repro. ultimately, this is likely just a matter of pants not yet being tested with python 3.6 due to it just being released a week or two ago.

we did recently bump setuptools to the latest version, so you might try this with a version of pants that includes the setuptools version bump (may need to be a dev release).

[Jesse Hersch]

Just to wrap this up, it appears to be bug in pants 1.2 when building python 3.6 projects. 

I put this in pants.ini instead and it works now:

pants_version: 1.3.0.dev6