How to effectively create a non-fat JAR file while following Bazel's "Best Practices" for Java?

[nicke...@gmail.com]

Hi all,

I'm pretty new to Bazel, and had a quick question regarding building Java libraries using it. I noticed in the "Java and Bazel" page of the Bazel documentation site, there are a handful of "Best Practices" that are listed, as seen here:

https://docs.bazel.build/versions/master/bazel-and-java.html#best-practices

For my Java library, I did as the documentation suggested and add BUILD files to every package in my library, all using a non-recursive glob and a local "java_library" rule. However, in regards to generating a final JAR file, I've been rather stumped. I added one final "java_library" rule to the base BUILD file in the directory root, which lists the local "java_library" rule I have in my top-level package as its only dependency. However, when building JAR files via Bazel the class files of dependencies aren't included unless you create a deployment JAR. For a Java library (not binary), this is less than ideal, as it either means your JAR will contain the class files for EVERY dependency (third-party and local), or no class files at all. So just to give a minimalistic example, assuming you have this directory structure:

src
main
java
BUILD
Library.java
BUILD

And the nested BUILD file contains:

java_library(
name = "java",
srcs = glob(["*.scala"]),
deps = ["@org_apache_commons_commons_lang3//jar"],
visibility = ["//visibility:public"]
)

While the parent-level BUILD file contains:

java_library(
name = "lib",
deps = ["//src/main/java:java"],
visibility = ["//visibility:public"]
)

The compiled Library.class will never make its way into the final JAR unless you compile lib_deploy.jar, and at the point all the class files for Apache Lang3 will be included in the JAR as well. What approach should be taken to include all of the local dependencies, but not the external ones, while still adhering to the best practices? Hopefully this all makes sense. Thanks.

[László Csomor]

Hi,

IIUC you'd like to build a deploy jar without class files from certain dependencies. I can't think of an idiomatic way to do that. I recommend you file a feature request.

In the meantime my best idea is to write a java_binary that depends on everything, and a genrule that depends on the java_binary's deploy jar and repackages the deploy jar without the third-party class files.

For the record java_library isn't fit for deployment. Its purpose is to organize code into logical units we call "libraries". (I'm surprised Bazel supports building lib_deploy.jar at all.)

I hope this helps!

Cheers,

Laszlo

--
László Csomor | Software Engineer | laszlo...@google.com

Google Germany GmbH | Erika-Mann-Str. 33 | 80636 München | Germany
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/78c80fb4-cd56-42df-981d-48cc9130eeb5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tyler Nickerson]

Hm, alright. I’ll look into making a feature request. I suppose with that sentiment though the real question becomes, given those “best practices”, how are Bazel devs expected to develop a Maven JARs using Bazel?  Maven JARs (or any repository JARs for that matter) do not include the class files of the JAR’s dependencies, but should contain all local class files. Additionally, using a _binary rule to build a JAR that isn’t even meant to be executable seems like it could be confusing, at least from a semantics standpoint. Wondering what the best approach is here. Surely devs have built non-executable deployment JARs using Bazel and the structural guidelines mentioned in the link I posted, right?

Thanks!

[ittai zeidman]

I think the context wasn’t clear.
What you’re looking for is exporting outputs built by bazel to Maven? I suggest you search through the issues of the bazel repo since it has been discussed several times and people even linked to their own examples

[david.o...@gmail.com]

On Monday, April 30, 2018 at 6:24:30 PM UTC+2, Tyler Nickerson wrote:

> Hm, alright. I’ll look into making a feature request. I suppose with that sentiment though the real question becomes, given those “best practices”, how are Bazel devs expected to develop a Maven JARs using Bazel?

You are asking the right question. lib_deploy.jar is correct
approach to achieve what you want. We do it all the time to
produce Maven artifacts.

What you are trying to do is to cut some edges from the
dependency graph to produce final artifact. You don't need
to file a feature request. It's already there: [1].

This is a major missing Bazel feature and is known in other
build tools as a "list of provided deps". Buck added this
feature (on our request) 5 years ago to java_library.

Unfortunately it's still not implemented in Bazel. So given
provided_deps attribute exposed some day in java_library in
Bazel, the fix of your (non working) example would just be:

java_library(
name = "java",
srcs = glob(["*.scala"]),

provided_deps = ["@org_apache_commons_commons_lang3//jar"],

visibility = ["//visibility:public"]
)

There is a workaround, though, but it's cumbersome, known as neverlink.
That would mean, that you would need to duplicate substantial
parts of your build tool chain. Moreover, maven_jar rule doesn't
support it. That why Gerrit Code Review project created its own clone
of maven_jar, and duplicated every external imported artifact
with and without neverlink=1.

1. Workaround is to use http_file and then java_import to define
java_library for external dependency with neverlink.

I recently did something similar to add
javax.annotation as neverlink library in gRPC
build tool chain, that is using maven_jar: [2].

2. Workaround is to define wrapper around the maven_jar,
something like:

java_library(
name = "commons-lang3-neverlink",
neverlink = 1,
visibility = ["//visibility:public"],
exports = ["@org_apache_commons_commons_lang3//jar"],
)

3. Workaround, you could just use maven_jar from
Gerrit's bazlets repository: [3]. That would automagically
add for you neverlink library that can be used like this:

"@org_apache_commons_commons_lang3//jar:neverlink"

* [1] https://github.com/bazelbuild/bazel/issues/1402
* [2] https://github.com/grpc/grpc-java/commit/23fcedfb6ff144da3f2d767c3aa67490d2dd7f5d
* [3] https://github.com/GerritCodeReview/bazlets/blob/master/tools/maven_jar.bzl

[ittai zeidman]

David,
Are you familiar with java_import_external?
If I’m not mistaken it creates a link/neverlink target per jar (if you ask it to).
+100 on provided

--
You received this message because you are subscribed to a topic in the Google Groups "bazel-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bazel-discuss/PvssQ709iGw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/bd591488-bd1a-48ce-9b1b-400a385ec4cc%40googlegroups.com.

[david.o...@gmail.com]

On Tuesday, May 1, 2018 at 7:14:01 PM UTC+2, ittai zeidman wrote:
> David,
> Are you familiar with java_import_external?
> If I’m not mistaken it creates a link/neverlink target per jar (if you ask it to).

Thanks for pointing this out, Ittai. Now, that Justine
upstreamed it and java_import_external is included in
Bazel core, it could be easily used:

load("@bazel_tools//tools/build_defs/repo:java.bzl", "java_import_external")
java_import_external(
name = "com_google_guava",
licenses = ["notice"],
neverlink = True,
jar_urls = [
"https://mirror.bazel.build/repo1.maven.org/maven2/com/google/guava/guava/21.0/guava-21.0.jar",
"https://repo1.maven.org/maven2/com/google/guava/guava/21.0/guava-21.0.jar",
],
jar_sha256 = "972139718abc8a4893fa78cba8cf7b2c903f35c97aaf44fa3031b0669948b480",
)

This would pass neverlink = True directly to java_import:

java_import(
name = "com_google_guava",
jars = ["guava-21.0.jar"],
neverlink = True,
)

Still, my experience is, that the same dependencies
are needed with flipped neverlink value. One example
is to not transitively include library foo in final
artifact, but still run JUnit tests against it.

That could end up specifying external dependency twice
and thus even downloading it twice in the worst case
with the swapped neverlink value: True and False: [1].

That why I think that being able to specify an edge in
the dependency graph as provided and cut of the entire
transitive closure from the final artifact, is a better
choice comparing to mark a node in the dependency graph
as provided.

[1] https://github.com/GerritCodeReview/gerrit/blob/master/lib/bouncycastle/BUILD#L1-L28

[ittai zeidman]

To clarify- I agree with you. We now need to jump through hoops.

What I wanted to show is that with Justine's solution the hoops are smaller.

The rule generates both linkable and neverlink targets from the same jar.

See generated_linkable_rule_name and generated_rule_name attributes

--
You received this message because you are subscribed to a topic in the Google Groups "bazel-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bazel-discuss/PvssQ709iGw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/3efcff8c-014b-422b-a1fc-f91cdb540982%40googlegroups.com.

[david.o...@gmail.com]

On Wednesday, May 2, 2018 at 6:37:14 AM UTC+2, ittai zeidman wrote:
> To clarify- I agree with you. We now need to jump through hoops.
> What I wanted to show is that with Justine's solution the hoops are smaller.
> The rule generates both linkable and neverlink targets from the same jar.
> See generated_linkable_rule_name and generated_rule_name attributes

I think we agree on this. My point was that neverlink=1
almost always need the opposite part: neverlink=0. In
java_import_external this is achieved through:
extra_build_file_content attribute to avoid downloading
the artifact twice.

In my previous example, to have guava library in both
permutations one would say this (like it is done in
nomulus project for appengine SDK: [1]).

java_import_external(
name = "com_google_guava",

jar_urls = [
"https://mirror.bazel.build/repo1.maven.org/maven2/com/google/guava/guava/21.0/guava-21.0.jar",
"https://repo1.maven.org/maven2/com/google/guava/guava/21.0/guava-21.0.jar",
],
licenses = ["notice"],
neverlink = True,

extra_build_file_content = "\n".join([
"java_import(",
" name = \"testonly\",",
" jars = [\"guava-21.0.jar\"],",
" testonly = True,",
")",
]),

jar_sha256 = "972139718abc8a4893fa78cba8cf7b2c903f35c97aaf44fa3031b0669948b480",
)

To generate this BUILD file (therefore my duplication point):

java_import(
name = "com_google_guava",
jars = ["guava-21.0.jar"],
neverlink = True,
)

java_import(
name = "testonly",
jars = ["guava-21.0.jar"],
testonly = True,
)

[1] https://github.com/google/nomulus/blob/3dfd141e0fed650b5eb2631b4345220355221b77/java/google/registry/repositories.bzl#L766-L790

[Philipp Wollermann]

+Ulf Adams FYI

--

You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/078d59c4-ca68-41ff-b1cf-8d18eec45ce8%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Google Germany GmbH
Erika-Mann-Straße 33
80636 München

[Ulf Adams]

Hi all,

are you aware of java_binary.deploy_env?

I'm all for building an automated solution, but it's not clear to me how you want to identify which libraries / jars need to be linked into the final .jar for publishing, and which don't. neverlink is one way to annotate the rules at the individual library level, which is problematic if you have cases where you want both, i.e., one binary that can be deployed with all deps, and one library for upload to Maven (or elsewhere). In that case you end up duplicating the entire dependency graph. In a monorepo, you may actually have multiple different deployment / publishing environments that all require different sets of third-party libraries to be linked in or not linked in. This is the case we've been discussing internally over the years, but we haven't found a good solution yet.

Thanks,

-- Ulf

[Natan Silnitsky]

David, I've tried defining a java_import_external with both neverlink=1 and neverlink=0 without defining extra_build_file_content attribute.
Seems to have worked:

java_import_external(

    name = "com_google_guava",

    licenses = ["notice"],  # Apache 2.0

    jar_urls = [ 

          "https://mirror.bazel.build/repo1.maven.org/maven2/com/google/guava/guava/21.0/guava-21.0.jar", 

          "https://repo1.maven.org/maven2/com/google/guava/guava/21.0/guava-21.0.jar", 

      ],

    jar_sha256 = "972139718abc8a4893fa78cba8cf7b2c903f35c97aaf44fa3031b0669948b480",

    neverlink = True,

    generated_linkable_rule_name="linkable_com_google_guava",

)

the generated BUILD file:

java_import(

    name = "com_google_guava",

    jars = ["guava-21.0.jar"],

    neverlink = True,

)

java_import(

    name = "linkable_com_google_guava",

    jars = ["guava-21.0.jar"],

)