java and strict-deps

[ittai zeidman]

Hi,

We, rules_scala, have a strict-deps solution with a compiler plugin which is both naive (many false negatives) and less performant (compilation action has all transitive ijars as inputs).

We're POCing a different approach which only passes the direct jars and using an external tool to scrape the error messages, extract the needed class and then needed label.

This is showing great success so far in both fronts but I know that often the compiler needs the first level transitive (dependency of dependency) even if it isn't used explicitly (usage of interface which extends interface for example).

I remember, maybe in correctly, that java rules have the following characteristics:

1. They try with direct + one level (limited classpath?) before falling back to entire transitive classpath

2. They try very hard to only fail if they conclude that the usage is "real" and not just because the compiler needed it (see case above).

My questions are:

1. Anyone know how this decision of "real" usage works? Would love links to read and think.

2. Anyone know how the direct + one level works? I think we might need that as well. 

The big reason I'm concerned is that in a many repo setting having many false negative strict-deps can cause many breakages of peer repositories when someone just changes implementation (exception now extends another exception)

Thanks in advance,

Ittai 

[Kevin Bierhoff]

This may be off topic, but you may find this tool useful for checking strict_deps.  It's "stricter" than what Bazel will do for .java files, but it works for any language that compiles to bytecode and doesn't have false negatives.

--
You received this message because you are subscribed to the Google Groups "Bazel/JVM Special Interest Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-sig-jv...@googlegroups.com.
To post to this group, send email to bazel-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-sig-jvm/CAOfK4wULtysN7mhi%2BJ1vnvtiqYiusBCU2y9OqptQ16%2BKXJT0tQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Kevin Bierhoff

Google

[ittai zeidman]

Thanks!

Looks interesting. In what sense is this "stricter"?

Also can you share a bit more details on where you use (or would use) this?

One thought I can think of is to pass direct + one level to compiler and afterwards run this tool to do extra validation. 

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-sig-jvm/CABdRVUbZe0UCXjqDy0mLke-uwRxC3EfeQY2BFv4971NTFGG5sA%40mail.gmail.com.

[Kevin Bierhoff]

On Tue, Dec 11, 2018 at 10:41 PM ittai zeidman <itt...@gmail.com> wrote:

Thanks!

Looks interesting. In what sense is this "stricter"?

The tool requires any and all types mentioned in bytecode as "strict" dependencies, whereas for Java builds we currently use a more lenient heuristic that only requires types mentioned in source.  This can make a difference in something like

T x = a.b().c()

This tool will require b's return type to be defined in a direct dependency, but for Java we currently do not.  The issue is simply that this distinction isn't easily recoverable in bytecode.

 

Also can you share a bit more details on where you use (or would use) this?

You use various flags to define bootclasspath (--bootclasspath_entry), classpath (--classpath_entry), and the subset of the classpath that are direct dependencies (--directdep).  For an example see Bazel's use of this tool here.

 

One thought I can think of is to pass direct + one level to compiler and afterwards run this tool to do extra validation. 

We use the entire transitive classpath but you're welcome to experiment with a subset of that for --classpath.  Please do note that "just" using the Jars from direct dependencies tends to not work, as the tool currently tries to load superclasses etc. as well in order to check that the given inputs are fully consistent.  You could experiment with watering that down somewhat though.

--

Kevin Bierhoff

Google

[ittai zeidman]

Ok interesting. I think it might be too strict for what I’m looking for.
Will this fail if “a” changed implementation and returns a different “b” (that answers the same interface)? Not sure the example is accurate but it just sounds to me like I can still get failures that the end user will be surprised from.

Do you happen to know how the limited classpath works and also where is the tool that does source checking so we can try and learn from it for scala?

[ittai zeidman]

+Liam Miller-Cushon for some words of advice on how java currently works (TL;DR mainly interested in how "limited" classpath works).

Kevin,

I've thought about the tool some more and it sounds like it's a post compilation one (since it needs the bytecode), right?

I'm looking for a tool I can use to build the compilation classpath (rather than trim it) so I can have significantly less entries in the compilation classpath (a lot due to scala and ijar being a poor match).

I am considering somehow passing direct + one level to the compiler and so limit my exposure and then still running the above tool to catch cases where people infringe the strict-deps for the "plus one level" case.

[Kevin Bierhoff]

Apologies, I don't think I had groked you were trying to prune the input classpath before compiling.

Re ijar I'll throw out another maybe-useful pointer: we recently enhanced ijar to not strip methods that carry a custom bytecode attribute "com.google.devtools.ijar.KeepForCompile".  This is probably a bit more involved to take advantage of, but if you can arrange for that attribute to be placed where you need it (naively as a post-processing step) you may get closer to making ijar work for Scala.  Again, this may be involved, or may not be a problem for Scala, but the intention was to allow ijar to be used with compilers that sometimes need method bodies from the classpath.

--

Kevin Bierhoff

Google

[Liam Miller-Cushon]

On Tue, Dec 11, 2018 at 2:25 PM ittai zeidman <itt...@gmail.com> wrote:

1. They try with direct + one level (limited classpath?) before falling back to entire transitive classpath

2. They try very hard to only fail if they conclude that the usage is "real" and not just because the compiler needed it (see case above).

My questions are:

1. Anyone know how this decision of "real" usage works? Would love links to read and think.

The fallback logic isn't very clever, it just looks for a list of diagnostics that could be caused by a too-reduced classpath:

https://github.com/bazelbuild/bazel/blob/bcad22120cfa63cf60368eaabc5dfa4e0eaef9fe/src/java_tools/buildjar/java/com/google/devtools/build/buildjar/ReducedClasspathJavaLibraryBuilder.java#L89

The goal of that is avoid falling back for errors where fallback definitely won't make a difference, since falling back is obviously slower, so it saves a bit of time when iterating on incorrect builds.

The classpath reduction / fallback stuff is distinct from strict deps enforcement; in theory the strict deps errors are the same either way.

 

2. Anyone know how the direct + one level works? I think we might need that as well. 

The reduced classpath heuristic is to take:
* all jars for targets that are direct dependencies
* all jars that JavaBuilder needed to read when compiling those direct dependencies (as reported by that dependency's .jdeps file)

Here's the implementation:

https://github.com/bazelbuild/bazel/blob/bcad22120cfa63cf60368eaabc5dfa4e0eaef9fe/src/java_tools/buildjar/java/com/google/devtools/build/buildjar/javac/plugins/dependency/DependencyModule.java#L246