Problems deploying a PEX built on OS X to ubuntu target

[Mike Cripps (Football Radar)]

Hi Pants folks

We're attempting to build a .pex so we can execute it in our production environment. We are currently building the pex on an OS X machine, and deploying to Ubuntu Linux.

When we copy (scp) the pex to the remote machine and attempt to run it, we get the following error:

Traceback (most recent call last):
File "/home/ubuntu/cron.pex/.bootstrap/_pex/pex.py", line 263, in execute
working_set = self._env.activate()
File "/home/ubuntu/cron.pex/.bootstrap/_pex/environment.py", line 123, in activate
self._working_set = self._activate()
File "/home/ubuntu/cron.pex/.bootstrap/_pex/environment.py", line 141, in _activate
resolved = working_set.resolve(all_reqs, env=self)
File "/home/ubuntu/cron.pex/.bootstrap/pkg_resources.py", line 639, in resolve
raise DistributionNotFound(req)
DistributionNotFound: thrift

However '//3rdparty/python:thrift' is clearly marked as a dependency of our python_library. We thought that perhaps this is because we didn't specify the 'platforms' in the python_binary and so we did that as follows:

platforms = ['current', 'linux-x86_64'],

But now when we try to build the pex we get this error:

Exception message: Package SourcePackage(u'file:///Users/.../.pants.d/python/eggs/SQLAlchemy-0.9.8.tar.gz') is not translateable.

We have cleared the entire virtualenv (multiple times!) and I admit, at this point I'm a little lost and confused - it feels like we're almost there, but this error message doesn't return many hits on Google. Has anyone else successfully built a .pex on OS X and deployed it to linux? Our task is made harder by the fact that when I try to build the binary on my linux desktop I get errors about scipy and numpy missing something (again, they are marked as dependencies of the library):

ImportError: No module named numpy.distutils.core

Which suggests it's not looking in the virtualenv for those imports correctly. Maybe the fact it works on the OS X machine at all is a lucky fluke.

Thanks,

Mike

[John Sirois]

On Feb 24, 2015 9:33 AM, "Mike Cripps (Football Radar)" <mike....@footballradar.com> wrote:
>
> Hi Pants folks
>
> We're attempting to build a .pex so we can execute it in our production environment. We are currently building the pex on an OS X machine, and deploying to Ubuntu Linux.
>
> When we copy (scp) the pex to the remote machine and attempt to run it, we get the following error:
>
> Traceback (most recent call last):
> File "/home/ubuntu/cron.pex/.bootstrap/_pex/pex.py", line 263, in execute
> working_set = self._env.activate()
> File "/home/ubuntu/cron.pex/.bootstrap/_pex/environment.py", line 123, in activate
> self._working_set = self._activate()
> File "/home/ubuntu/cron.pex/.bootstrap/_pex/environment.py", line 141, in _activate
> resolved = working_set.resolve(all_reqs, env=self)
> File "/home/ubuntu/cron.pex/.bootstrap/pkg_resources.py", line 639, in resolve
> raise DistributionNotFound(req)
> DistributionNotFound: thrift
>
> However '//3rdparty/python:thrift' is clearly marked as a dependency of our python_library. We thought that perhaps this is because we didn't specify the 'platforms' in the python_binary and so we did that as follows:
>
> platforms = ['current', 'linux-x86_64'],

From my phone so this is terse,  but you're on the right track here.  The remaining issue is pants has no magic for creating linux platform specific bdists on osx - it can't.  Unfortunately the error message is a bit obtuse.

You need to have pre-built linux bdists available in some repo and add this repo to your pants.ini.  Pantsbuild has this set up for its own cross platform pex builds (it uses psutil for example).  See this enabling section in pants.ini: https://github.com/pantsbuild/pants/blob/master/pants.ini#L174

The repos key needs to have an urgent pointing to a flat directory of prebuilt bdists and the indices line is important so that pants falls back to searching pypi for everything else.

[John Sirois]

And it's easy to check if you've got this setup right before deploying - just list the pex (it's a zip file) and confirm there are 2 bdists for each platform specific dep /in .deps/).  You should see an osx _and_ a linux egg for thrift.

[Andy Reitz]

Hi Mike,

John is exactly right. We build PEX files on OS X that we deploy on Linux all the time at Twitter, and it works great -- so long as the right pre-conditions exist. When pants says "DistributionNotFound: thrift", what it really means is that it couldn't find a bdist version of thrift -- something like "Thrift-<version>-py2.7-linux-x86_64.egg".

This can be confusing because I believe that pants will make a bdist for the platform that it's running on, which explains why things work fine if you build on Mac and run on Mac. But as John said, pants doesn't contain a cross-compiler, so for other environments, it must rely on downloading an appropriate file.

You'll want to tweak this section of your pants.ini:

https://github.com/pantsbuild/pants/blob/master/pants.ini#L175

As John said, add a URL that  contains your bdists ahead of any URLs that contain sdists. In addition, you'll need to compile up your platform-specific bdist. I think this entails downloading the source to a Linux machine, and running something like "python2.7 setup.py bdist_egg".

-Andy.

[Mike Cripps (Football Radar)]

Thanks for the quick responses!

I'll have a go at building the dependencies on my linux box and uploading to some server.

Is there a difference between eggs and wheels? I see some of the dependencies are .whl and some are .egg - so I'm guessing there's nothing "important" different between them.

Thanks again

Mike

[Mike Cripps (Football Radar)]

Just a quick update,

After building the eggs and putting them in my very own cheeseshop (boy, that sounds like a weird sentence out of context) I was able to build a .pex that has both OS X and linux .eggs in it and it now runs in production perfectly happily.

Thanks for your help!

Mike

[John Sirois]

That's great to hear.  I had re-read your original problem and noticed scipy/numpy and cringed, but remained silent.  In the past at Twitter just building platform specific eggs for these proved difficult IIRC and doing so such that the eggs worked in a pex was, again IIRC, a showstopper for reasons I cannot remember.  So again - yay!

[adam hajari]

I think the pants.ini file linked to here has changed significantly so here's a link to the version of that file from around the time of this post: https://github.com/pantsbuild/pants/blob/b92e334418f8504f8d972299423f766d7876de15/pants.ini#L174

Documentation on this feature has also been added since this post and can be found here: https://pantsbuild.github.io/howto_develop.html#building-pants-pex-for-production

Also, some of the issues with numpy and scipy may be related to this issue: https://github.com/pantsbuild/pex/issues/281

Since pants can't use the prebuilt linux wheels from PyPI, it might fail when attempting to build those libraries if the system level dependencies aren't available.

[Eric Zundel Ayers]

Here is my attempt to answer how to build a multi-platform pex (macos/linux) on stack overflow:

http://stackoverflow.com/questions/34979100/pants-includes-os-x-specific-python-wheels

[Simeon Franklin]

For this problem I've really appreciated the PEP513/manylinux project. 

I use the manylinux docker image from my OSX laptop. The image has multiple Pythons (2 & 3, various ABI's) setup for generating Python wheels which are compatible with a broad spectrum of Linux distros. It does this by statically linking everything not on a lowest common denominator whitelist and it does mean frequently I can download wheels (including numpy/scipy) which will run on whatever Linux I've got.

To use with pants & pex I've had to rename them since PEX doesn't support the manylinux OS tag but its made generation much easier...