Python: It doesn't really work; what is the path forward?

[evan....@bluecore.com]

TL;DR: Bazel does not work for Python that depends on third-party code via "standard" PyPI packages. What is the plan to fix this? Are people actively working on it? How can we collaborate to make progress on this?

Details and current non-working solutions:

I've been experimenting with using Bazel at Bluecore (~35 engineers with a lot of Python). Here are the current approaches, and why none of them really work:

Built-in rules:
* No support for PyPI packages, requiring manual effort to import each one.
* Namespace packages cause problems (see below).
* Skylark rules can't see the imports attribute, so custom rules can't interact with it correctly https://github.com/bazelbuild/bazel/issues/2617
* The imports attribute is weird: it has cross-rule effects in a way that seems surprising to me (this isn't fatal, but I think it does decrease "usability" of the rules)

rules_python: This seems to be an "official" attempt to make Python work? It makes using PyPI packages much easier, but still has problems:

* Not reproducible, since it depends on whatever "pip" resolves the dependencies to. This is convenient, but the results can change between builds. It seems to me that an approach like Maven bazel-deps would be better, where we can check in the results of dependency resolution so future builds gets the exact same results.

* Namespace packages such as Google's APIs don't work (fatal for us): https://github.com/bazelbuild/rules_python/issues/14

PEX via bazel_rules_pex [2]: PEX is a tool that packages python like PAR/subpar, originally from Twitter and the Pants build tool [3], and also used by Buck [4]. These rules actually work! I wrote a tool to generate the appropriate WORKSPACE and BUILD declarations, and so far I've been able to get Bluecore's code to work.

* PEX has bugs related to resolving packages correctly
* PEX is slow: Both in terms of building a PEX, and executing it. For packages that depend on large libraries (e.g. scipy, numpy, scikit-learn), this makes the edit/test cycle very slow.

My own Python zip rules: As an experiment, I've written rules that package everything into one zip, mostly to see if it can be substantially faster than PEX and to understand the issues better. I think this is a workable solution, but I would rather not have to maintain my own set of build rules.

The Path Forward

In my opinion, the "ideal" solution is to "bless" rules_python as the official rules and fix its issues. The problem is this will require someone who has understanding of Bazel internals, as well as deep knowledge about Python's somewhat arcane imports system. Probably this means the Bazel team and Google would need to take this up.

An alternative is that the "external" Bazel users could collaborate on a set of Skylark rules, like the Scala and Rust rules. From what I understand of Bazel and Python, I think to workaround the import issues, this would probably need to abandon the native rules entirely, and take the "produce a single zip" approach to avoid the imports/namespace package weirdness.

I'd love to help with this, if anyone is interested.

Evan Jones

[1] https://github.com/bazelbuild/rules_python
[2] https://github.com/benley/bazel_rules_pex
[3] https://github.com/pantsbuild/pex
[4] https://buckbuild.com/rule/python_binary.html

[Dave Stanke]

Hi Evan,

Great timing! Following on from the recent Bazel user conference, several people in Google's Bazel team are now making it a priority to plot a path forward for Python, and implement the necessary changes. Your interest and feedback are very valuable here, and so is your offer to contribute! :D

It's true that we're not in a great place right now. The good news is that this (IMHO) a totally tractable problem, and we have resources who can work on it. Now we need to align on a path forward—your proposal is a great start so let's iterate on it until we have agreement.

I suggest we assemble an interest group that represents people from across the community, including Googlers. Who's in? 

Thanks,

-Dave (Bazel product manager)

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/f3703ac4-135d-422b-81c3-0986469c3c7e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Hyrum Wright]

I'd be willing to participate in a python-bazel interest group.

I spent a few months last fall trying to use Bazel as our build system in a majority-python shop.  I got things mostly working, but was unable to make progress on a few fronts:

• Test coverage support for Python (https://github.com/bazelbuild/rules_python/issues/43)
  
• Python 3 support (e.g., https://github.com/bazelbuild/rules_python/issues/33, https://github.com/google/containerregistry/issues/42)
• Support for pip packages with native components (https://github.com/bazelbuild/rules_python/issues/48)
• Packaging everything in to a docker image

I've tried hacking on the relevant rules, but since these goals are not part of the core Bazel team's efforts, I haven't gotten much support.  I've kind of resigned myself to filing issues, sending an occasional PR and hoping that Python support would become important to the core team at some point.

If it looks like I can contribute to something with long-term traction to succeed, I'm happy to do so.

-Hyrum

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/f3703ac4-135d-422b-81c3-0986469c3c7e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAAsxKoyGg1TmpTdTDZU-L01gxm5W%3Dkwi2KrnuH-4nsi2WqU0gQ%40mail.gmail.com.

[cmo...@uber.com]

George Gensure recommended that I also join the python-bazel interest group.  I'm investigating Python support in our org. and there are still some hurdles to overcome, it seems.  I'd like to make sure the roadmap has the right requirements to be flexible and generic.

Thanks,

Chad Moore (Uber ATG) 

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/f3703ac4-135d-422b-81c3-0986469c3c7e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

[Evan Jones]

This is exciting, I'm glad there is interest! I would obviously like to participate in any "interest group" that forms. Any suggestions on what we should actually *do*? My suggestion might be to try and arrange an initial conference call for people who are using Bazel with Python (or want to be), where we can briefly discuss what approaches we are taking, what issues we have run into, and discuss potential solutions? We could then report back to bazel-discuss with notes and figure out what improvements can be made.

As a minor contribution, I'm trying to extract my hacky rules and tools that "work for Bluecore", since at the very least they've let me work around a few of the issues I've mentioned, so they might be useful to others or can serve as an example of an alternative approach (which might not be ideal).

Finally, I must admit: I am slightly skeptical about groups like this, since sometimes they can result in lots of talk but little action. However, I would even just be interested in hearing from other Bazel Python users to better understand what works and what doesn't.

Thanks!

Evan

--
You received this message because you are subscribed to a topic in the Google Groups "bazel-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bazel-discuss/NMO6KPyPKh4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bazel-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/9e928339-991b-4c49-a9e3-329ea949e9d5%40googlegroups.com.

[stew...@google.com]

Count me in for the interest group!

[Dave Stanke]

Awesome! Great to see so much energy. And totally agreed that we want to be sure that any group we form is actually useful to its participants!

So as you suggest, let's kick this off with an initial conference call to get aligned on what we want to get out of this, and lay some ground rules. I've posted a Doodle with some times that happen to work for me:

https://doodle.com/poll/5yig865d4xq7ew6i

Interested folks: please indicate your availability. Thanks!

Dave

To unsubscribe from this group and all its topics, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/9e928339-991b-4c49-a9e3-329ea949e9d5%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CACRPLjgwfx2PZP7xqEbf2F38sdaKuZ5oxD9iUkgmVXX1XqUAcA%40mail.gmail.com.

[Dave Stanke]

Update: thanks to everyone for indicating availability. We've selected Tuesday, Jan 30 at 1PM Eastern. 

Here's the link to the Hangout: https://hangouts.google.com/hangouts/_/google.com/bazel-python

I plan to send some notes in advance, but, briefly, the goal is to reveal everyone's goals for the group and goals for Bazel+Python; identify key differences between goals; and coordinate efforts moving forward. Come prepared to tell your Bazel/Python story.

Thanks,

Dave

Dave

To unsubscribe from this group and all its topics, send an email to bazel-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/9e928339-991b-4c49-a9e3-329ea949e9d5%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

[lcid...@gmail.com]

On Wednesday, January 24, 2018 at 10:15:24 AM UTC+1, Dave Stanke wrote:
> Update: thanks to everyone for indicating availability. We've selected Tuesday, Jan 30 at 1PM Eastern. 
>
>
> Here's the link to the Hangout: https://hangouts.google.com/hangouts/_/google.com/bazel-python

Since a world does exist outside of the Americas: 1PM Eastern is GMT-5 and 18:00 in UTC.

[Oleg Tsarev]

On Monday, January 15, 2018 at 8:18:18 PM UTC+1, Hyrum Wright wrote:

I'd be willing to participate in a python-bazel interest group.

I spent a few months last fall trying to use Bazel as our build system in a majority-python shop.  I got things mostly working, but was unable to make progress on a few fronts:

• Test coverage support for Python (https://github.com/bazelbuild/rules_python/issues/43)
  
• Python 3 support (e.g., https://github.com/bazelbuild/rules_python/issues/33, https://github.com/google/containerregistry/issues/42)
• Support for pip packages with native components (https://github.com/bazelbuild/rules_python/issues/48)
• Packaging everything in to a docker image

python + docker + cross-platform

https://github.com/bazelbuild/rules_python/pull/61 

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/f3703ac4-135d-422b-81c3-0986469c3c7e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

[mbench...@newstore.com]

I hope it's not too late to join the party but I am also interested into what's the plan are, I can also try to help as much as I can since we are facing similar issues in my organization, thanks Evan for starting the discussion.

Mouad

[Alexey Dudko]

If it is still possible I would like to join this call. Looks like the road map has covered the most issues with python support in bazel. Would be nice to hear how you plan to implement everything and if I can provide some help.

Alexey Dudko

[Evan Jones]

The link Dave Stanke posted should be an open Google Hangout for anyone to join, today at 1:00 PM/13:00 US/Eastern / 18:00 UTC:

https://hangouts.google.com/hangouts/_/google.com/bazel-python

Evan

On Tue, Jan 30, 2018 at 6:02 AM, Alexey Dudko <dud...@gmail.com> wrote:

If it is still possible I would like to join this call. Looks like the road map has covered the most issues with python support in bazel. Would be nice to hear how you plan to implement everything and if I can provide some help.

Alexey Dudko

--

You received this message because you are subscribed to a topic in the Google Groups "bazel-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bazel-discuss/NMO6KPyPKh4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bazel-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/840bf39d-c39a-40f8-a171-d14b81a28dcb%40googlegroups.com.

[Dave Stanke]

Yup, it's an open link. Please join!

On Tue, Jan 30, 2018 at 9:20 AM Evan Jones <evan....@bluecore.com> wrote:

The link Dave Stanke posted should be an open Google Hangout for anyone to join, today at 1:00 PM/13:00 US/Eastern / 18:00 UTC:

https://hangouts.google.com/hangouts/_/google.com/bazel-python

Evan

On Tue, Jan 30, 2018 at 6:02 AM, Alexey Dudko <dud...@gmail.com> wrote:

If it is still possible I would like to join this call. Looks like the road map has covered the most issues with python support in bazel. Would be nice to hear how you plan to implement everything and if I can provide some help.

Alexey Dudko

--
You received this message because you are subscribed to a topic in the Google Groups "bazel-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bazel-discuss/NMO6KPyPKh4/unsubscribe.

To unsubscribe from this group and all its topics, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/840bf39d-c39a-40f8-a171-d14b81a28dcb%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CACRPLjj%3DDkBt%2B46e0imp7WZBmP8rf8vSmZC%2BM07-JwADgKuZsA%40mail.gmail.com.

[Philipp Wollermann]

Can you share notes from the call please? :)

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAAsxKoybiyW%2BeWSkfJZfYMQdCh%2Bj9QJ09u1bF582LFOj8L9rNg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Google Germany GmbH
Erika-Mann-Straße 33
80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

[Dave Stanke]

Sure, summary and links are here: https://groups.google.com/forum/#!topic/bazel-discuss/k8MN4BmyjTY