Fwd: [Bazel-team] Bazel Removal from Debian

[Yun Peng]

If anyone is interested in keeping Bazel as an official Debian package, please help with this one!

---------- Forwarded message ---------
From: Olek Wojnar <ol...@debian.org>
Date: Sun, Feb 19, 2023 at 4:33 PM
Subject: [Bazel-team] Bazel Removal from Debian
To: Debian Bazel Discussion List <debian...@lists.debian.org>, Debian Java Discussion List <debia...@lists.debian.org>
Cc: Debian Bazel Packaging Team <bazel...@lists.launchpad.net>

Fellow Bazel contributors (and Java Team members),

I've discussed this with a few of you but I want this to go to the widest possible audience right now. We still have no idea why Bazel Java tests are failing [1] and it seems to indicate an actual problem with Bazel, as opposed to just a flaky test. (It would be nice if I'm wrong about that) To save people a click, the failing error is:
Unrecognized VM option 'UseParallelOldGC'
(Which has been patched out of the code [2] so I have no idea where it's coming from)

So, if we cannot fix this, I'll have to remove Bazel from Debian testing before the bookworm release. Meaning Bazel will NOT be in the next stable Debian release.

1 March 2023 is the last day to make an upload that will definitely make it in before the Hard Freeze.

So, I'm asking all interested people to please please take a look and see if you can figure out how to fix either the tests or the suspected underlying issue.

Pull Requests from your own branch in the bazel-bootstrap repository [3] are preferred but I'll take patches or even "you should look at line xxx in this one source file."

Please reply to this thread with any ideas or suggestions. Hopefully with all our eyes on this we can make this really frustrating bug a bit more shallow.

Thanks in advance!!

-Olek

[1] https://salsa.debian.org/bazel-team/bazel-bootstrap/-/jobs/3862692
[2] https://salsa.debian.org/bazel-team/bazel-bootstrap/-/blob/master/debian/patches/fix_JDK-16_test_failures.patch
[3] https://salsa.debian.org/bazel-team/bazel-bootstrap

--
Mailing list: https://launchpad.net/~bazel-team
Post to     : bazel...@lists.launchpad.net
Unsubscribe : https://launchpad.net/~bazel-team
More help   : https://help.launchpad.net/ListHelp

[Tobias Werth]

This is not about Bazel itself but the Java tools / worker to compile Bazel.

+Lars Clausen +Liam Miller-Cushon +Ivo Ristovski List 

Where are the flags for the javac actions defined nowadays?

Olek, which version of Bazel are you packaging?

--
You received this message because you are subscribed to the Google Groups "bazel-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-dev/CAOZBPs4jBm6dPSf-Rkof2QKiR-MXfmD-AKuDf0iKFBOajb%2B7iQ%40mail.gmail.com.

--

[Yun Peng]

I think it's in the java_tools defined in the WORKSPACE file and jdk.WORKSPACE (they need to be in sync as the comment indicates):

• https://salsa.debian.org/bazel-team/bazel-bootstrap/-/blob/master/WORKSPACE#L971-981
  
• https://salsa.debian.org/bazel-team/bazel-bootstrap/-/blob/master/src/main/java/com/google/devtools/build/lib/bazel/rules/java/jdk.WORKSPACE#L188-197

pcloudy@pcloudy-macbookpro3:~/workspace/tmp/java_tools_old
$ wget https://mirror.bazel.build/bazel_java_tools/releases/javac11/v10.6/java_tools_javac11_linux-v10.6.zip
pcloudy@pcloudy-macbookpro3:~/workspace/tmp/java_tools_old
$ unzip java_tools_javac11_linux-v10.6.zip
Archive:  java_tools_javac11_linux-v10.6.zip
  inflating: BUILD
   creating: java_tools/
 ...
pcloudy@pcloudy-macbookpro3:~/workspace/tmp/java_tools_old
$ ls
BUILD                              README.md                          java_toolchain_default.bzl         java_tools                         java_tools_javac11_linux-v10.6.zip
pcloudy@pcloudy-macbookpro3:~/workspace/tmp/java_tools_old
$ grep -r "UseParallelOldGC"
./java_toolchain_default.bzl:        "-XX:+UseParallelOldGC",

./java_toolchain_default.bzl:        "-XX:+UseParallelOldGC",

A possible solution is to update java_tools to a newer version that no longer uses "-XX:+UseParallelOldGC". But I'm not sure which one, and whether it's still compatible with the Bazel version being packaged.

[Yun Peng]

> Regarding java_tools, we still don't have that packaged in Debian so I'm not sure where those flags would be coming from. Yun, that was one of the things we had the mock repos for, right? I looked through that patch and only saw a reference to Jacoco. Or is there some extra magic hidden somewhere that I'm missing?

We only made sure Bazel bootstrap without internet access, but when running tests, Bazel still can download extra dependencies defined in WORKSPACE suffix (jdk.WORKSPACE) I guess that's how the flag was pulled in.

On Tue, Feb 21, 2023 at 5:17 PM Olek Wojnar <ol...@debian.org> wrote:

So we unfortunately still have version 4.2.3 in Debian. I had hoped to upgrade to 5.x before the freeze but life did not cooperate with me. :(

(It's very close though and I'm planning to backport it to stable after release)

Regarding java_tools, we still don't have that packaged in Debian so I'm not sure where those flags would be coming from. Yun, that was one of the things we had the mock repos for, right? I looked through that patch and only saw a reference to Jacoco. Or is there some extra magic hidden somewhere that I'm missing?

-Olek

On February 21, 2023 4:02:26 PM UTC, Yun Peng <pcl...@google.com> wrote:
>I think it's in the java_tools defined in the WORKSPACE file and
>jdk.WORKSPACE (they need to be in sync as the comment indicates):
>

>   -
>   https://salsa.debian.org/bazel-team/bazel-bootstrap/-/blob/master/WORKSPACE#L971-981
>   -

>> +Lars Clausen <lar...@google.com> +Liam Miller-Cushon <cus...@google.com> +Ivo
>> Ristovski List <il...@google.com>

>>> <https://groups.google.com/d/msgid/bazel-dev/CAOZBPs4jBm6dPSf-Rkof2QKiR-MXfmD-AKuDf0iKFBOajb%2B7iQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>
>>
>> --
>>
>>

[Brian Silverman]

I would find a Debian package with only these useful:

* The shell completions

* Empty `/etc/bazel.bazelrc` registered as a conffile

* `/usr/bin/bazel` shell script that looks for `tools/bazel`.

Having `/usr/bin/bazel-real` as bazelisk (or teaching the shell script to look for `/usr/bin/bazelisk` or something) seems fine too, but I still want my `tools/bazel`-which-is-bazelisk to be used.

I never use the installed `/usr/bin/bazel-real` on purpose, I don't keep the package up to date so it's never relevant.

Setting up something with Debian's alternatives system to allow choosing bazelisk or something as the bazel-real binary could work too, for maximum flexibility.

On Fri, Feb 24, 2023 at 12:12 AM Andreas Ames <andreas.0...@gmail.com> wrote:

Could it be sensible to have a "bazelisk" package and a "bazel_is_bazelisk" package instead in Debian?

Cheers,

Andreas

Am Fr., 24. Feb. 2023 um 00:08 Uhr schrieb Filip Filmar <fil...@gmail.com>:

On Wed, Feb 22, 2023 at 1:56 PM Olek Wojnar <ol...@debian.org> wrote:

Any other suggestions? 

 

Frankly, why not just drop the package altogether?

Various projects supporting bazel lock the version with `.bazelversion`, and you can at most offer one. Because of this, anyone who needs bazel to work will have to download bazelisk anyways.

F

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAKaOXijX%2BSN9hM2r9pDzhNacyAdXBJVchpcpd8fKO540J1WGtA%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAHAfzDnuiJ7VYwXkptkdLnDVNMDLy55-NBA_osUErh%3DoG3%3DW5A%40mail.gmail.com.

[Tobias Werth]

On Fri, Feb 24, 2023 at 10:17 AM Brian Silverman <bsilve...@gmail.com> wrote:

I would find a Debian package with only these useful:

* The shell completions

* Empty `/etc/bazel.bazelrc` registered as a conffile

* `/usr/bin/bazel` shell script that looks for `tools/bazel`.

Having `/usr/bin/bazel-real` as bazelisk (or teaching the shell script to look for `/usr/bin/bazelisk` or something) seems fine too, but I still want my `tools/bazel`-which-is-bazelisk to be used.

Bazelisk should respect tools/bazel, see https://github.com/bazelbuild/bazelisk#other-features 

I never use the installed `/usr/bin/bazel-real` on purpose, I don't keep the package up to date so it's never relevant.

Setting up something with Debian's alternatives system to allow choosing bazelisk or something as the bazel-real binary could work too, for maximum flexibility.

On Fri, Feb 24, 2023 at 12:12 AM Andreas Ames <andreas.0...@gmail.com> wrote:

Could it be sensible to have a "bazelisk" package and a "bazel_is_bazelisk" package instead in Debian?

Cheers,

Andreas

Am Fr., 24. Feb. 2023 um 00:08 Uhr schrieb Filip Filmar <fil...@gmail.com>:

On Wed, Feb 22, 2023 at 1:56 PM Olek Wojnar <ol...@debian.org> wrote:

Any other suggestions? 

 

Frankly, why not just drop the package altogether?

Various projects supporting bazel lock the version with `.bazelversion`, and you can at most offer one. Because of this, anyone who needs bazel to work will have to download bazelisk anyways.

F

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAKaOXijX%2BSN9hM2r9pDzhNacyAdXBJVchpcpd8fKO540J1WGtA%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAHAfzDnuiJ7VYwXkptkdLnDVNMDLy55-NBA_osUErh%3DoG3%3DW5A%40mail.gmail.com.

--

Google Germany GmbH

Erika-Mann-Straße 33

80636 München

Geschäftsführer: Paul Manicle, Liana Sebastian

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich. Falls Sie diese fälschlicherweise erhalten haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail an die falsche Person gesendet wurde. 

     

This e-mail is confidential. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person.

[Olek Wojnar]

Sadly, I've answered my own question: there are no java_tools releases compatible with Bazel 4.x that use 'UseParallelGC'.

Any other suggestions? 

On Tue, Feb 21, 2023 at 11:58 AM Olek Wojnar <ol...@debian.org> wrote:

Uh, oh...I think I might have just realized the little detail that I felt I was missing......

On February 21, 2023 4:20:23 PM UTC, Yun Peng <pcl...@google.com> wrote:
>We only made sure Bazel bootstrap without internet access, but when running
>tests, Bazel still can download extra dependencies defined in WORKSPACE
>suffix (jdk.WORKSPACE) I guess that's how the flag was pulled in.

I think you're right! We're not *supposed* to use Internet resources for tests and, if we need to download any test data, we're supposed to mark that as a test restriction. I don't have that restriction marked and therefore we all assumed that nothing was being downloaded. Rereading some of the autopkgtest documentation more carefully, it sounds like it doesn't actively *prevent* Internet access, it just trusts that you aren't doing it. And I didn't realize that on Bazel execution it was downloading those resources. For normal usage, it wouldn't matter. But it does for the test!

Ok, so does anyone know which java_tools we could use that would prevent this problem?

And I definitely need to work on packaging java_tools for Debian since a native java_tools would have allowed us to just patch the problematic flags...

-Olek

[Olek Wojnar]

(Forwarding to distro lists from Gmail since they apparently don't like my @debian.org email address)

---------- Forwarded message ---------
From: Olek Wojnar <ol...@debian.org>

Date: Sun, Mar 5, 2023, 11:04
Subject: Re: [bazel-discuss] Re: [bazel-dev] Fwd: [Bazel-team] Bazel Removal from Debian
To: Filip Filmar <fil...@gmail.com>
Cc: Yun Peng <pcl...@google.com>, Tobias Werth <twe...@google.com>, Lars Clausen <lar...@google.com>, Liam Miller-Cushon <cus...@google.com>, Ivo Ristovski List <il...@google.com>, bazel-dev <baze...@googlegroups.com>, bazel-discuss <bazel-...@googlegroups.com>

Hi Filip,

First of all, sorry for the slow reply. All my recent spare time was used up by trying to fix that package.

On February 23, 2023 11:08:02 PM UTC, Filip Filmar <fil...@gmail.com> wrote:
>On Wed, Feb 22, 2023 at 1:56 PM Olek Wojnar <ol...@debian.org> wrote:
>
>> Any other suggestions?
>>
>
>Frankly, why not just drop the package altogether?

So I think the question you're meaning to ask is, "why is there a native Bazel package in Debian?"

There's a long story there but the simple answer is: "So that we can build Debian packages that use Bazel as their build system." We're not quite there yet but that's the goal.

For a developer working on an independent software project it makes MUCH more sense to use bazelisk or a package provided directly by the Bazel team.

But Debian requires a natively-built toolchain for Debian packages, both for reproducibility and for security. As more upstream projects transition to using Bazel, the need for Debian to natively build Bazel software packages becomes more keen.

>Various projects supporting bazel lock the version with `.bazelversion`,

That can be patched in Debian and isn't something we're very concerned about.

>and you can at most offer one.

Actually, we can ship an arbitrary number of stable Bazel versions in Debian. Originally we planned to, but discussions with the Bazel team and Tensorflow packagers lead me to believe that a higher version of Bazel should work without extensive difficulty for projects written to build with a lower version.

If you're aware of insurmountable difficulties in building a project with a newer version of Bazel, please let me know! That information would definitely be helpful for our roadmap!

-Olek

[Olek Wojnar]

Thanks again to Yun for that big insight! And thanks to everyone else for your comments and suggestions!

To summarize my earlier comments (and in case they didn't make it to the distro lists from my @debian.org email): I'm going to just accept that the Java tests don't work for the Debian 12 release. At least now I understand why. I'm still planning to backport Bazel 5 to Debian 12 shortly after release. My efforts will then focus on Bazel 6. Since packaging so much of the Bazel infrastructure for Debian was fairly time-intensive, we've fallen too far behind HEAD which makes it very difficult to make progress on that Distribution Friendly bug on GitHub. So, plan in a nutshell is:

1) Package Bazel 5 (bootstrap)

2) Package Bazel 6 (bootstrap)

3) Package remaining dependencies for non-bootstrap Bazel 6 (only 3 left, I think, and I explored a bit of the relatively challenging java_tools while trying to troubleshoot the Java tests)

4) Package Bazel 6 (non-bootstrap)

5) Upstream code changes which will allow Bazel 7 to build natively in Debian and other distributions with minimal patches

I'll update our team page soon [1] and we welcome anyone interested to join us! More people helping will help us reach those goals sooner!

Thanks again!

-Olek

[1] https://salsa.debian.org/bazel-team/meta