A post-BazelCon recap

[Jon Brandvein]

Greetings Pythoneering Bazelers,

We’ve had a little time for the dust to settle from BazelCon 2018, and I wanted to recap our plans for better Python support in the short and long term. I’ll talk first about community and then about some of the upcoming technical work. (For those who weren’t able to attend the Python SIG, slides are here.)

My main takeaway from the conference was that there is a lot of potential bandwidth in the community for contributing Starlark rules and bugfixes. It seems there are two main blockers on the Bazel side. First, we have not provided the community with a centralized place to merge these contributions. Second, it is difficult to write Starlark rules that build on existing Python support. Both of these factors lead to forked rulesets and redundant efforts.

To the first point, we want to make bazelbuild/rules_python the canonical repository for Python rules and designs. This has always been the plan unofficially, but it’s important to say so explicitly. From an end-user perspective, people should be able to trust that this repo supports their core use cases, is well-maintained, and is of production quality. From an ecosystem perspective, new experimental rules should aim to be compatible with this repo.

The choice to make rules_python the focal point, as opposed to rules_pyz or another repo, is just a matter of namespacing and not an endorsement of this ruleset’s particular existing design. We’ll deprecate, modify, and merge features as needed to reach an end-state that makes sense.

As a small step in this direction, going forward, all design proposals that affect Python support will be indexed there. In particular, proposals that affect the native Python rules will be indexed both in bazelbuild/rules_python and in bazelbuild/proposals, and will be subject to the Bazel design process.

The second point, that it is difficult to write rules that interoperate with native Python rules, can be addressed by adding a well-defined “sandwich” API to the native Python rules, like we are doing for other native rulesets. Eventually the native rules will simply be migrated to Starlark and live in rules_python, though we don’t anticipate that happening in the immediate future.

The core technical improvements that we plan to make in the native rules over the next few quarters include the following (think of this as an abridged roadmap):

• Revamp the model for supporting Python 2 and 3 in the same build (Q4; see tracking bug and a design proposal I’ll link in a separate email)

• Support multiple Python interpreters in the same build, with per-target and toolchain-based interpreter selection (Q4-Q1)

• Improve/redesign pip integration, so that there’s an officially blessed ruleset that allows for multiplatform builds of projects having PyPI dependencies (Q4-Q1..?)

• Sandwich API (native rules can depend on Starlark-written ones and vice versa) (Q1-Q2)

• Redesign/document runfiles layout (Q1-Q2)

These are rough estimates, of course. Importantly, the following items have *lower* priority than the above goals:

• Building CPython extensions within Bazel (as opposed to depending on them via pre-built wheels)

• Rules for building or deploying PyPI packages

• Migrating all native rules to Starlark (in rules_python)

• Vendoring CPython itself into the build (but note that toolchain rules will make it easier to depend on the system interpreter across multiplatform builds)

• Various tooling integrations (linting, typing, etc.)

Once again, thanks for all the great input at the conference, and your continued involvement with Bazel and Python.

[markus....@ecosia.org]

 I was not at BazelCon but python support is very important for us so thanks for posting this here and sharing the slides. I think that sounds like a pretty good roadmap and also for us the most critical thing right now is to be able to build py2/py3 in the same build without having to specify any flags.

About the open question if people want fine-grained choice of python minor versions - it is not that important for me personally but I know it is quite important for some developers on our team, the argument being, if we do not get python hermetically then at least there should be a way to specify the exact python version that should be used to build/run this project. Not sure how much more complexity that would add though.

[Jon Brandvein]

> if we do not get python hermetically then at least there should be a way to specify the exact python version that should be used to build/run this project

Absolutely, and that's planned under the second bullet (multiple interpreters in the same build). The difference between that and Py 2/3 support is that the latter has dedicated features in the rules (srcs_version) for checking compatibility with the given interpreter.

My point about minor versions is not so much whether we support having multiple interpreters -- we will -- but to what extent it's important to be able to annotate py_library targets with constraints about minor versions, the way srcs_version does for 2/3. At the very least you'll be able to manually say this target should use this interpreter.

--
You received this message because you are subscribed to the Google Groups "Bazel/Python Special Interest Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-sig-pyth...@googlegroups.com.
To post to this group, send email to bazel-si...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-sig-python/3b794d90-134a-49a1-a2b8-65c52b35f3f0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jordan Adler]

This looks like a great first step Jon, thanks for taking this work on.

A few thoughts on your open questions:

• Does anyone care about 2to3 support?
  
• I wouldn't recommend investing in integrating 2to3 as part of the rules_python work.  2to3 isn't sufficiently automatic and I would advise against 2to3 for migrations in favor of future or six which have more robust designs.
• Should we be using Py 3 as the default?
• Yes.
• Do people want support for fine-grained choice of Python minor versions?
  
• Yes.  What would the benefit be of not permitting the maximum possible granularity when specifying a dependency in Bazel?  I ought to be able to specify the maximum level of granularity possible, up to and including providing a full source tree for the (patched) python interpreter.

Further I'd suggest you reach out to the Python core developer team, particularly the release manager, to get a better understanding of how to interpret version numbers going forward.  My understanding is that the next few releases will be 3.8, 3.9, and 4.0, each having roughly the same volume of changes.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-sig-python/CAEoyrevCGMw2tMzGtO7FxWQU4oyQ_MCcDNdafvJigvj3i61y_g%40mail.gmail.com.

[Jon Brandvein]

Thanks for the points Jordan! Some of the open questions from my slides were answered with the help of the SIG participants. In particular, current thinking is that:

- 2to3 support will not be added to native rules and the existing stub will be removed (bazel #1393).

- Python 3 will be made the default at some point (bazel #6647).

- Choosing a specific interpreter will definitely be possible. Choosing a checked-in interpreter should also be possible, though that might be better supported later on.

I looked into the Python major version naming situation a bit. From some cursory searching it sounds like there's no consensus -- some people say "Python 4.0" to mean "the next breaking change release", and others use it to mean Python 3.10. It's also possible that by the time there's a Python 4, Python 2 will be a distant memory. I'll hold off on worrying about this until we have more direction from the Python core developers.

[Jon Brandvein]

Just an update on rules_python: I'm declaring a sort of triage bankruptcy -- all issues are being labeled with "untriaged", and I'll go through assigning priorities as I can.

However, I expect to not be able to work on the vast majority of these issues in the near future, since I'm handling the tasks from my first email above. The stuff I want to get to in rules_python is:

- fixing CI failures, and major breakages with current or near current versions of Bazel

- merging PRs for bugfixes where the solution appears non-controversial

[Jon Brandvein]

FYI CI is fixed.

Note that in Bazel 0.20 (to be released very soon) you'll need to update to avoid breaking by default. Alternatively, pass the flags --incompatible_remove_native_http_archive=false and --incompatible_remove_native_git_repository=false. Please see #105 for details.