Proposal to expose "py_interpreter" attribute

[sbag...@grailbio.com]

Hello,

While there may be larger plans underfoot to have proper ecosystem and toolchain support for python (namely Operation Purple Boa), the current situation is quite messy when it comes to supporting both python2 and python3 code in the same repo.

Some github issues that talk about this in more detail for a variety of users:

https://github.com/bazelbuild/rules_python/issues/62

https://github.com/bazelbuild/bazel/issues/3517

https://github.com/bazelbuild/bazel/issues/1580

While we wait for the proper support to come, I propose that we expose the "py_interpreter" attribute in python rules as public. Or make a new attribute just for the executable python targets.

The only purpose of this attribute is to substitute for the PYTHON_BINARY variable in the executable wrapper stub, so the change is fairly well contained.

This will alleviate the pain points for many users for whom it is currently really difficult to support both python2 and python3 uses in their monorepos.

If there is reasonable consensus on this, I can submit a pull request.

Thanks.

Sid

[Austin Schuh]

We set the py_runtime to point to a script with the following in it:

if head -n 1 "$1" | grep -q python3; then

  exec "$BASE_PATH"/usr/bin/python3 "$@"

else

  exec "$BASE_PATH"/usr/bin/python2 "$@"

fi

That triggers off the shebang to pick the right python.  Definitely not pretty, but it bought us some more time until this gets solved for real.

Austin

--
You received this message because you are subscribed to the Google Groups "Bazel/Python Special Interest Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-sig-pyth...@googlegroups.com.
To post to this group, send email to bazel-si...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-sig-python/d093f668-6f8a-47d9-82b9-cde2ef5130bb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Josh Burkart]

+1 for some limited-scope, interim workarounds like this!

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-sig-python/CABsbf%3DEuhw5G7tN5HF8AtT_RP4K3XHtbMez%2B9stOSWdnH-_FxQ%40mail.gmail.com.

[markus....@ecosia.org]

How exactly is that working Austin? I just tried doing the same in the following way:

py_runtime(

name = "py",

files = [],

interpreter = "py_select.sh",

)

and py_select.sh:

#! /usr/bin/env bash

if head -n 1 "$1" | grep -q python3; then

exec "$BASE_PATH"/usr/bin/python3 "$@"

else

exec "$BASE_PATH"/usr/bin/python2 "$@"

fi

then I specify that runtime in python_top and I constantly get my binaries executed with python2, because it is trying to read the first line of the python files. Or are you just adding the name of the python version on top of your files in a comment?

[Marco Farrugia]

Their python likely uses shebangs for this, but just the name would do.

#! /usr/bin/env python3

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-sig-python/6055ecc5-5e92-4a88-ac65-c1a804454371%40googlegroups.com.

[Brian Silverman]

I work on the same code as Austin, and yes, we do put shebangs at the top of our Python binary mains. Some of them use /usr/bin/env like Marco's example, while others are just #!/usr/bin/python2 or #!/usr/bin/python3. Doing one of those has the added advantage of helping tools (vim, Gerrit, etc) syntax highlight them.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-sig-python/CAEMOC3muGYDf9%3DAyObsJ%3DBYTRVEKepfc_miXpooUWDT%3DM9C%2BMA%40mail.gmail.com.

[Jon Brandvein]

To respond to the original suggestion: I've been looking into py2 vs 3 issues as a first step toward a saner Python-Bazel ecosystem. Exposing py_interpreter sounds like a reasonable option. But since py_interpreter/py_runtime may serve as a good stepping stone to something like py_toolchain as described in the doc, and since there are open questions about how exactly to integrate such a rule with Bazel's general support for platforms and toolchains, I'd rather write a design doc before taking action here.

In the meantime, we have these kinds of indirection hacks (see also the one using selects).