bazel, python and pypi

[P. Oscar Boykin]

TL;DR:

does anyone out there have any pypi bazel rules to share?

========

Python of course is a very popular language. So I would expect some folks are using bazel with python in production.

I watched this talk about dropbox's use of bazel:

https://www.youtube.com/watch?v=JXGjzzBKKDo

but I don't find any code from them. They mention having some code generator that generates rules for pypi packages. They seem happy with them. Such a generator would be incredibly useful, I think for many folks. Does anyone know how to reach them? I tried with twitter, no luck: https://twitter.com/posco/status/852563760170450944

Using the built-in python support leaves you with no pypi support and using the system python version (which different developers may not agree about on their machines).

The non-hermetic python bug was filed some time back, but seems still far off.

https://github.com/bazelbuild/bazel/issues/890

tagged for 1.0

And the pypi bug:

https://github.com/bazelbuild/bazel/issues/699

is tagged for 0.7

There is a prototype implementation linked:

https://github.com/gengo/rules_pypi

but it seems like it is not being developed and comments imply it won't work for many important cases.

Does anyone out there have any good bazel pypi interop rules they are willing to open-source so we can collaborate?

[ms...@dropbox.com]

Unfortunately, packaging Python is more complicated than just PyPI. I've never seen that talk, but I work on the system in question.

To build Python hermetically, you need to control Python and all the transient native libraries it might load. Each PyPI libraries expands that scope of potential dependencies. Additionally, you must choose whether all of you production environment will support exactly one version of an external library (this is the "vendoring" discussion). These choices extend native code as well - how many versions of zlib do we need? How do you track critical security fixes, etc. For stability, it's not really tenable to support multiple versions. You also need to deal with multiple OS releases and various incarnations of the runtime (glibc etc) that may not be fully compatible. We solve this with a "portable" runtime that makes every Linux OS capable of running binaries produced by Bazel.

Once you have constrained the problem a bit, you can write a program with some simple heuristics to generate accurate BUILD files for python. PyPI and native code generally must be handled by hand as a one-time cost. The tools assist you, but you must still resolve some issues by hand - (ambiguous imports).

The by-product of these constraints is that I don't think our rules for Python/Go/Typescript/etc are portable or externally useful in their current form. A lot of decisions made were a influenced by the timeline and internal environment. I'm not sure there is a clear path to open sourcing something like that in a way that's actually helpful for the community since the scope is so large.