New pants install doesn't compile working PEXes.

[Katie Lucas]

Hello.

We're currently using PEX with TOX to build python apps on 2.6. However we're also looking at migrating our systems to 2.7 early next year so I've been trying to get PEX working with 2.7 -- switching to Pants is defn an option (which at least has the advantage that I'm used to BUILD files) so I have a new Pants install to play with.

However; both PEX under Pants and on its own fails to build working PEXs.



I'm compiling a "hello world", no libraries, just the one file.

I have a basic BUILD which specifies the one target.


./pants binary /home/katie/pants:hello

<lots of output, ending with SUCCESS>

dist/hello.pex
/usr/bin/python2.7: bad pyc data

With debugging on;


<much loading>
Python 2.7.5 (default, Apr 10 2015, 08:09:05)
[GCC 4.8.3 20140911 (Red Hat 4.8.3-7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
# zipimport: found 79 names in dist/hello.pex
# /usr/lib64/python2.7/runpy.pyc matches /usr/lib64/python2.7/runpy.py
import runpy # precompiled fom /usr/lib64/python2.7/runpy.pyc
dlopen("/usr/lib64/python2.7/lib-dynload/syslog.so", 2);
import syslog # dynamically loaded from /usr/lib64/python2.7/lib-dynload/syslog.so
import imp # builtin
# /usr/lib64/python2.7/pkgutil.pyc matches /usr/lib64/python2.7/pkgutil.py
import pkgutil # precompiled from /usr/lib64/python2.7/pkgutil.pyc
dlopen("/usr/lib64/python2.7/lib-dynload/zlibmodule.so", 2);
import zlib # dynamically loaded from /usr/lib64/python2.7/lib-dynload/zlibmodule.so
# zipimport: zlib available
/usr/bin/python2.7: bad pyc data
<shutdown output>

Inside the PEX, the pyc files are zero length -- the "bad pyc data" message is coming from the zlib module which does a length check on the file before trying to read the magic/version number out of and bails at that point when it finds a zero.

<many files...>
10212 12-11-2015 11:33 .bootstrap/_pex/variables.py
0 12-11-2015 11:33 .bootstrap/_pex/variables.pyc
246 12-11-2015 11:33 .bootstrap/_pex/version.py
0 12-11-2015 11:33 .bootstrap/_pex/version.pyc
101340 12-11-2015 11:33 .bootstrap/pkg_resources.py
0 12-11-2015 11:33 .bootstrap/pkg_resources.pyc
870 12-11-2015 11:33 PEX-INFO
737 12-11-2015 11:33 __main__.py
0 12-11-2015 11:33 __main__.pyc
93 12-11-2015 11:33 hello.py
0 12-11-2015 11:33 hello.pyc
--------- -------
316293 79 files


This happens with the standalone PEX distribution on this machine as well (although the code compiles entirely happily on py2.6 virtual machines).


I could probably hack this (unpack the zip, compile the pyc files, put the zip back together) but that kind of feels.... wrong...

Anyone seen anything like this before?

[Kris Wilson]

I can't say that I've ever seen that before. Any chance you can provide a copy of your test repo and/or reproduction steps? What's your pex/pants version?

There's a possibility this might be related to your python distribution as well - is it a fresh, custom build?

FWIW 2.7.5 is fairly old and known to contain bugs - ideally you could get onto something >2.7.8 - or better yet, straight to 2.7.11 to leverage the new performance improvements w/ computed gotos.

-Kris

[Matthew Olsen]

It could be related to the python version but this still seems like a case where PEX should be providing some kind of error when it builds the .pex file.  I would consider this a pex bug.

[Katie Lucas]

On Friday, December 11, 2015 at 6:38:28 PM UTC, Kris Wilson wrote:

I can't say that I've ever seen that before. Any chance you can provide a copy of your test repo and/or reproduction steps? What's your pex/pants version?

There's a possibility this might be related to your python distribution as well - is it a fresh, custom build?

FWIW 2.7.5 is fairly old and known to contain bugs - ideally you could get onto something >2.7.8 - or better yet, straight to 2.7.11 to leverage the new performance improvements w/ computed gotos.

In an ideal world, yes -- but the reason for picking 2.7.5 was that it's the standard install for Centos 7.

(One the reasons this has only just started looking like a possible move is that currently our install is oriented around 2.6!)

I've had a go installing 2.7.11[1] -- and interestingly PEX itself works fine; pexifies itself and that can pexify our apps. So we have a build route albeit not ideal. CTO isn't averse to the idea of deploying 2.7.11 (at least on the build machines) but we'd all have more faith in things if we knew why empty PYCs were being generated..

Pants, however, still builds a binary missing the PYC files. I'll be honest -- that might just be it picking up the wrong Python version which is where I can pick this up on Monday -- or even picking up the wrong pex version. Does it use the system installed version or does it do the work itself?

It does feel like a pexy issue -- I could fathom it not including the compiled files or including bad versioned ones, but to end up with empty ones feels like something's really not right and something it should complain about.

[1] Which was a little sunny adventure all on its own requiring both the MySQL and SQLite dev kits for various reasons.

[Kris Wilson]

It's tough to speculate on the actual root cause without more data and/or a repro. If you file a github issue with your pants version (or sha), BUILD file + test target contents I can take a look at reproducing this against 2.7.5 and dig into what's happening.

> Does it use the system installed version or does it do the work itself?

Pants uses a version constraint system for interpreter selection (e.g. 'CPython>=2.7,<3' - which I believe is the default). Given this constraint, pants will analyze your PATH looking for all python interpreters and then select the lowest version that satisfies the constraint - which in your case may still be 2.7.5 if thats on the $PATH alongside 2.7.11.

You should be able to do something along the lines of: `./pants -i "CPython==2.7.11" ...` (or a likewise setting in pants.ini) to force pants onto a specific interpreter version:

$ ./pants help-advanced |grep -A2 interpreter

-i=<requirement> (-i=<requirement>) ..., --interpreter=<requirement> (--interpreter=<requirement>) ... (default: [])

    Constrain what Python interpreters to use.  Uses Requirement format from

    pkg_resources, e.g. 'CPython>=2.6,<3' or 'PyPy'. By default, no constraints

    are used.  Multiple constraints may be added.  They will be ORed together.

Additionally, if 2.7.5 is in fact not on your path anymore a `./pants clean-all` should purge the interpreter cache.

-Kris

[Katie Lucas]

Specifying the Python version does indeed fix the issue -- the output
in dist is a valid pex which executes.

I should be able to upload a replication -- it's literally a hello
world python and a BUILD.