bootstrapping java_tools?

[beas...@vmware.com]

Bazel has a circular source dependency on itself in the form of the java_tools distribution, such that even when bootstrapping Bazel there is a dependency on precompiled C++ artifacts from java_tools via remote_java_tools_prebuilt, and the java_tools artifacts themselves must be built using Bazel. I would like to be able to bootstrap the java_tools from source as well, but that doesn't seem to be an option currently.

So, what is the expectation here? (I imagine that whatever reasons there are to support bootstrapping Bazel would also apply to the java_tools.) I assume this is required by Linux distros (ex: Debian) to support native packages, right? Is this something the Bazel Authors would support given the cycles/volunteers?

[Tobias Werth]

+Yun Peng +Ivo Ristovski List 

On Fri, May 6, 2022 at 9:58 AM beas...@vmware.com <beas...@vmware.com> wrote:

Bazel has a circular source dependency on itself in the form of the java_tools distribution, such that even when bootstrapping Bazel there is a dependency on precompiled C++ artifacts from java_tools via remote_java_tools_prebuilt, and the java_tools artifacts themselves must be built using Bazel. I would like to be able to bootstrap the java_tools from source as well, but that doesn't seem to be an option currently.

So, what is the expectation here? (I imagine that whatever reasons there are to support bootstrapping Bazel would also apply to the java_tools.) I assume this is required by Linux distros (ex: Debian) to support native packages, right? Is this something the Bazel Authors would support given the cycles/volunteers?

--
You received this message because you are subscribed to the Google Groups "bazel-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-dev/6c844026-4c00-4bd1-9e9f-52c94fc19a51n%40googlegroups.com.

--

Google Germany GmbH

Erika-Mann-Straße 33

80636 München

Geschäftsführer: Paul Manicle, Liana Sebastian

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich. Falls Sie diese fälschlicherweise erhalten haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail an die falsche Person gesendet wurde. 

     

This e-mail is confidential. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person.

[Yun Peng]

> even when bootstrapping Bazel there is a dependency on precompiled C++ artifacts from java_tools via remote_java_tools_prebuilt

IIRC, when bootstrapping Bazel, we are using a different java toolchain: https://github.com/bazelbuild/bazel/blob/master/scripts/bootstrap/bootstrap.sh#L39, therefore the java_tools dependencies shouldn't be needed. At least this was true when we successfully packaged Bazel for Debian. There was a plan to package java_tools for Debian with Bazel as the build system, but that plan hasn't moved forward.

However, I checked the java_tools dependencies do end up in the bootstrap archive, I'm testing in https://github.com/bazelbuild/bazel/pull/15415 to remove them, but it failed. 

Ivo, do you know how and when this dependency was introduced for bootstrapping Bazel?

[beas...@vmware.com]

> IIRC, when bootstrapping Bazel, we are using a different java toolchain [...] therefore the java_tools dependencies shouldn't be needed

Yup, saw those. :) So, IIUC it looks like there are two problems:

1. As you discovered, the bootstrapping toolchain still depends on remote_java_tools.
   • I think we can correct that by overriding `header_compiler`, `head_compiler_direct`, and `jacocorunner` w/ `None` in BUILD.bootstrap. Doing so eliminates the remote_java_tools references in the toolchain definition.
2. However, the bootstrapped Bazel binary itself then still injects a dependency upon a prebuilt distribution of java_tools via the embedded jdk.WORKSPACE. I am now trying to figure out how to build this myself without depending on `java_tools-v11.6.zip` and `java_tools_linux-v11.6.zip`. I was expecting to be able to use the bootstrapped Bazel binary to build //src:java_tools.zip and //src:java_tools_prebuilt.zip at the java_tools-v11.6 commit (40c9b756d7f8bac20801e2a54e3d83b2112c80c6).
   • Would it make sense for the bootstrapped Bazel to (perhaps optionally) bundle its locally built toolchain artifacts and a toolchain definition such that a user could specify `--extra_toolchains=@bazel_tools//tools/jdk:bootstrap_toolchain_definition` in order to build java_tools? If this is the right path forward, I'd be happy to contribute a PR.

Thanks for following up and sharing the pointers.

[Yun Peng]

On Tue, May 10, 2022 at 3:39 AM beas...@vmware.com <beas...@vmware.com> wrote:

> IIRC, when bootstrapping Bazel, we are using a different java toolchain [...] therefore the java_tools dependencies shouldn't be needed

Yup, saw those. :) So, IIUC it looks like there are two problems:

1. As you discovered, the bootstrapping toolchain still depends on remote_java_tools.
   • I think we can correct that by overriding `header_compiler`, `head_compiler_direct`, and `jacocorunner` w/ `None` in BUILD.bootstrap. Doing so eliminates the remote_java_tools references in the toolchain definition.

That sounds great, I think we should fix this.

1. However, the bootstrapped Bazel binary itself then still injects a dependency upon a prebuilt distribution of java_tools via the embedded jdk.WORKSPACE. I am now trying to figure out how to build this myself without depending on `java_tools-v11.6.zip` and `java_tools_linux-v11.6.zip`. I was expecting to be able to use the bootstrapped Bazel binary to build //src:java_tools.zip and //src:java_tools_prebuilt.zip at the java_tools-v11.6 commit (40c9b756d7f8bac20801e2a54e3d83b2112c80c6).
   • Would it make sense for the bootstrapped Bazel to (perhaps optionally) bundle its locally built toolchain artifacts and a toolchain definition such that a user could specify `--extra_toolchains=@bazel_tools//tools/jdk:bootstrap_toolchain_definition` in order to build java_tools? If this is the right path forward, I'd be happy to contribute a PR.

The java_tools is also built from the Bazel repo, where the bootstrapping toolchain exists. Maybe we can just use the bootstrap toolchain to build the java_tools without bundling it in Bazel?

 

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-dev/f0b996a4-756c-4ff7-ad1a-6f0c785e4d3dn%40googlegroups.com.

[Ryan Beasley]

The java_tools is also built from the Bazel repo, where the bootstrapping toolchain exists. Maybe we can just use the bootstrap toolchain to build the java_tools without bundling it in Bazel?

I'll defer to the Bazel Authors, but one thought:

1. If any other tools in the Bazel repository are decoupled from Bazel release cycle, they would also need their own bootstrap support such as a separate foo_tools-1.0.0-dist.zip​ w/ a separate derived​ tree, custom build script(s), etc., right? (Would remote_coverage_tools/coverage_output_generator fall into this category?) With this in mind, I still wonder if, instead of replicating the "bootstrap" release deliverables to each sub-project, there ought to be a self-contained "bootstrap" flavor of Bazel which is capable of building them from source.
1. Bootstrap Bazel w/ included bootstrap toolchain:
   ```
   unzip bazel-5.1.1-dist.zip
   env BAZEL_INCLUDE_BOOTSTRAP_TOOLCHAIN=1 ./compile.sh  # builds //src:bazel_nojdk_bootstrap instead
   ```
2.  Bootstrap java_tools using bootstrapped Bazel + included toolchain:
   ```
   git clone github.com/bazelbuild/bazel
   cd bazel
   git checkout 190533739ce4457cd5a3b31573fc2ae66f92d53e  # java_tools 11.7.1
   $BOOTSTRAP_ROOT/output/bazel build --additional_toolchains=@bazel_tools//tools/jdk:bootstrap_toolchain_definition -c opt //src:java_tools_prebuilt.zip
   ```

All that said, how should I proceed? Should I file a new "unable to bootstrap java_tools from source" issue under bazelbuild/java_tools, or should I file a "include bootstrap Java toolchain with Bazel" issue under bazelbuild/bazel? I can find cycles to work on this and contribute PRs.

Thanks!

---

From: Yun Peng <pcl...@google.com>
Sent: Tuesday, May 10, 2022 3:58 AM
To: Ryan Beasley <beas...@vmware.com>
Cc: bazel-dev <baze...@googlegroups.com>
Subject: Re: [bazel-dev] bootstrapping java_tools?

 

⚠ External Email

---

⚠ External Email: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender.

[Yun Peng]

 > All that said, how should I proceed? Should I file a new "unable to bootstrap java_tools from source" issue under bazelbuild/java_tools, or should I file a "include bootstrap Java toolchain with Bazel" issue under bazelbuild/bazel? I can find cycles to work on this and contribute PRs.

I'm fine with both. +Ivo Ristovski List should probably make the final decision on this?