$(JAVA) and $(JAVABASE) vs. minimal JDK9 in Bazel 0.16

[Tobias Werth]

Hi,

the next Bazel release 0.16 will use JDK9.

JDK9 offers the possibility to create a minimal JDK by specifying the modules that you need. For Bazel and the JavaBuilder the list of necessary modules is:

java.base,java.compiler,java.logging,java.management,jdk.management,jdk.unsupported,jdk.compiler,jdk.net,jdk.zipfs

This brings down the size of the embedded JDK to ~19MB and reduces the binary size by ~60MB.

All good things come with a price:

With Bazel 0.15, you were able to access the full embedded JDK with $(JAVA) or $(JAVABASE) in genrules. If we go with the minimal JDK for the next release, you cannot use modules like java.xml or java.corba unless you install your own local JDK and use that.

WDYT?

Jakob and Tobi

--

Tobias Werth

Software Engineer

twe...@google.com

Google Germany GmbH

Erika-Mann-Straße 33

80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado 

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich. Falls sie diese fälschlicherweise erhalten haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail an die falsche Person gesendet wurde. 

     

This e-mail is confidential. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person.

[László Csomor]

That size reduction sounds great! Thanks for your efforts!

I reckon the embedded JDK's purpose is to allow running Bazel without requiring a separate JDK (or JRE), and as a bonus to compile any embedded tools should they be implemented in Java. (I don't know if there are any.)

What's the use case for $(JAVA) in a genrule? If it's to run arbitrary Java binaries, then wouldn't using java_binary (in conjunction with java_import if necessary) eliminate the need for $(JAVA)? And if so, you'd already be building Java rules and needed a full JDK, so $(JAVA) sounds like a misfeature and no longer providing a full JDK sounds reasonable. (Let's put aside for now that this would potentially break existing genrules that rely on $(JAVA).)

How does that logic sound?

--
László Csomor | Software Engineer | laszlo...@google.com

Google Germany GmbH | Erika-Mann-Str. 33 | 80636 München | Germany

Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CACJA90%2BV%2BgtzqfQBsod8kmKXx%3Dt1qxOYO%2BqzrDJvp_8_hKrGcQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[ittai zeidman]

Do you have any thoughts on how this will play together with remote execution?

You received this message because you are subscribed to the Google Groups "Bazel/JVM Special Interest Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-sig-jv...@googlegroups.com.
To post to this group, send email to bazel-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-sig-jvm/CAJ5fxHLeT4McVEBpKF%3DrusUyAt1ff-X-9BNOUCtZi969Y5tHWQ%40mail.gmail.com.

[Lukács T. Berki]

This is an excellent question!

Per se, this won't make remote execution harder or easier, but it points in the same direction as we need to go to make remote execution a first-class citizen: currently, we ship a full-featured JDK with Bazel and we use that by default as "the JDK" (as opposed to requiring people to provide a separate one). This is convenient because Bazel needs a JDK to run and because JavaBuilder is coupled to a particular JDK version, but it's also somewhat odd because we treat Java specially. because unlike C++ or Android, we include the tools required to build a language in Bazel itself.

If you want remote execution, you'll need to provide a JVM/JDK to run on the remote machines. If you don't, we'll try the embedded JVM and fail if the workers are different from the machine where Bazel runs. This not that different in principle from when you require a JVM/JDK for any other reason, except that so far, we've let people get away without that and use this "courtesy JDK". And this change will make said "courtesy JDK" a bit less useful.

Eventually, I suppose this will lead to Bazel knowing about two kinds of JDKs: one that's only for running Bazel itself and JavaBuilder and one that's a full JDK. I'm not entirely sure whether these two concepts should be "officially" separated.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAOfK4wW5j7oKn86o6W%2BLit%3DnERV3h0KoiRtHh%3D2YDVGfna6zdA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Lukács T. Berki | Software Engineer | lbe...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891

[Jakob Buchgraber]

Hi Ittai,

On Thu, Jun 28, 2018 at 2:28 PM ittai zeidman <itt...@gmail.com> wrote:

Do you have any thoughts on how this will play together with remote execution?

What specifically do you have in mind? It's my understanding that Bazel currently and in the future expects

JDKs to be at the same path locally and on the remote execution system. The local and remote execution

JDK need to be identical. So from that perspective nothing should change.

Best,

Jakob

[Jakob Buchgraber]

On Thu, Jun 28, 2018 at 11:46 AM László Csomor <laszlo...@google.com> wrote:

That size reduction sounds great! Thanks for your efforts!

I reckon the embedded JDK's purpose is to allow running Bazel without requiring a separate JDK (or JRE), and as a bonus to compile any embedded tools should they be implemented in Java. (I don't know if there are any.)

What's the use case for $(JAVA) in a genrule? If it's to run arbitrary Java binaries, then wouldn't using java_binary (in conjunction with java_import if necessary) eliminate the need for $(JAVA)? And if so, you'd already be building Java rules and needed a full JDK, so $(JAVA) sounds like a misfeature and no longer providing a full JDK sounds reasonable. (Let's put aside for now that this would potentially break existing genrules that rely on $(JAVA).)

How does that logic sound?

Sounds great. Tobi and I applied the same logic this morning, however we obviously don't want to inflict pain on our users and break

them and so we need to be careful with this change. I also checked the Bazel source code and there is one test that uses $(JAVA) and

one that does uses $(JAVABASE)/bin/javap in a genrule. So I think we should clean that up :-).

[Daniel Halperin]

On Thu, Jun 28, 2018 at 6:50 AM 'Jakob Buchgraber' via bazel-discuss <bazel-...@googlegroups.com> wrote:

I also checked the Bazel source code and there is one test that uses $(JAVA) and

one that does uses $(JAVABASE)/bin/javap in a genrule. So I think we should clean that up :-).

Perhaps, linking to the PR(s) where you make this change in the release notes will be helpful. If folks see how you modified your own repo for this change, they may well have a roadmap to the fix.

Thanks,

Dan

[Jakob Buchgraber]

I haven't made the change yet and don't plan to before the 0.16.0 release :-).

We created a minimal JDK that would work with the current state of the Bazel repo.

Best,

Jakob

[Liam Miller-Cushon]

On Thu, Jun 28, 2018 at 2:46 AM 'László Csomor' via Bazel/JVM Special Interest Group <bazel-...@googlegroups.com> wrote:

I reckon the embedded JDK's purpose is to allow running Bazel without requiring a separate JDK (or JRE), and as a bonus to compile any embedded tools should they be implemented in Java. (I don't know if there are any.)

The embedded JDK is currently also the default RHS host_javabase, i.e. it is used to run all embedded Java-based host tools (which includes the Java compiler), and any user-provided host tools.

 

What's the use case for $(JAVA) in a genrule? If it's to run arbitrary Java binaries, then wouldn't using java_binary (in conjunction with java_import if necessary) eliminate the need for $(JAVA)? And if so, you'd already be building Java rules and needed a full JDK, so $(JAVA) sounds like a misfeature and no longer providing a full JDK sounds reasonable. (Let's put aside for now that this would potentially break existing genrules that rely on $(JAVA).)

Supporting the use of $(JAVA) in genrules equivalent to supporting the use of host tools that use java_binary; they both need a Java runtime.

The rationale for making the embedded JDK the default was outlined here: https://groups.google.com/d/msg/bazel-dev/ycRQMZ3woj8/Sa1SuJXPBgAJ. The Java toolchain requires a recent RHS host_javabase, and it's easier to keep the embedded JDK up to date than to require everyone using Bazel for Java development have the latest available JDK release.

[ia...@stripe.com]

I filed one issue about a concrete bug i can easily have repo'd publicly about this -- https://github.com/bazelbuild/bazel/issues/5572

But internally trying to test the 0.16 release with these changes seeing that the rt.jar isn't in the runfiles of java binaries and thus classpaths. Is this expected or a bug? it means that java.lang.Object isn't in the reflection mirror

in something like

$ cd bazel-out/host/bin/external/io_bazel_rules_scala/src/java/io/bazel/rulesscala/scalac/scalac.runfiles/embedded_jdk

$ find . -name '*.jar'

./lib/jrt-fs.jar

is this possibly a configuration issue or some sort of bug?

[Liam Miller-Cushon]

On Wed, Jul 11, 2018 at 8:09 AM ianoc via Bazel/JVM Special Interest Group <bazel-...@googlegroups.com> wrote:

But internally trying to test the 0.16 release with these changes seeing that the rt.jar isn't in the runfiles of java binaries and thus classpaths. Is this expected or a bug? it means that java.lang.Object isn't in the reflection mirror

in something like

$ cd bazel-out/host/bin/external/io_bazel_rules_scala/src/java/io/bazel/rulesscala/scalac/scalac.runfiles/embedded_jdk

$ find . -name '*.jar'

./lib/jrt-fs.jar

is this possibly a configuration issue or some sort of bug?

JDK 9 removes the rt.jar artifact, as documented here: https://docs.oracle.com/javase/9/whatsnew/toc.htm#JSNEW-GUID-527735CF-44E1-4144-919B-E7D7CC9CDD4D

Note that particular binary is a host tool that runs during the build, so it uses the `build --host_javabase`, which now defaults to the embedded JDK 9. If you do a `bazel build :some_java_binary && bazel-bin/some_java_binary`, that will not be affected by this change.

Are you using a version of rules_scala that includes the fix for https://github.com/bazelbuild/rules_scala/issues/429?