Setting a JGit dependency to use local development tree fails for me

[Gert van Dijk]

Hi,

In the context of testing the unreleased-but-merged fix in JGit for Issue 9612, I wanted to build Gerrit with it to confirm it's fixed, but I failed.

Here's what I did:

1. Check out Gerrit source tree 'master', recursively. Currently on v2.15.3-4920-g8d052edb22.
2. Clone JGit in another directory outside Gerrit's tree.
3. Check out 'stable-5.0' branch in JGit - this should be only a few changes compared to the 5.0.2 released version as Maven dependency configured.
4. As per developer documentation about Bazel and Gerrit I set LOCAL_JGIT_REPO in lib/jgit/jgit.bzl to point to my local checkout.
5. Run 'bazel build gerrit' and getting different dependency errors everytime:
   
   $ bazel build gerrit
   ERROR: /home/gert/.cache/bazel/_bazel_gert/657d7487a554575c9d983baaafc6bd5d/external/jgit/lib/BUILD:119:1: no such package '@jzlib//jar': The repository could not be resolved and referenced by '@jgit//lib:jzlib'
   ERROR: Analysis of target '//:gerrit' failed; build aborted: Analysis failed
   
   Variations:
   * ERROR: /home/gert/.cache/bazel/_bazel_gert/657d7487a554575c9d983baaafc6bd5d/external/jgit/lib/BUILD:10:1: no such package '@commons_compress//jar': The repository could not be resolved and referenced by '@jgit//lib:commons-compress'
   * ERROR: /home/gert/.cache/bazel/_bazel_gert/657d7487a554575c9d983baaafc6bd5d/external/jgit/lib/BUILD:153:1: no such package '@log_api//jar': The repository could not be resolved and referenced by '@jgit//lib:slf4j-api'

What am I doing wrong?

• It also fails similarly with the JGit version checked out as set in Gerrit/master (v5.0.2.201807311906-r).
• I can build this JGit version perfectly fine with 'bazel build //...' - standalone.)
• Bazel versions used: 0.15.1, 0.16.1.
  

Thanks,

Gert

[David Ostrovsky]

Am Mittwoch, 29. August 2018 10:01:48 UTC+2 schrieb Gert van Dijk:

Hi,

In the context of testing the unreleased-but-merged fix in JGit for Issue 9612, I wanted to build Gerrit with it to confirm it's fixed, but I failed.

Here's what I did:

1. Check out Gerrit source tree 'master', recursively. Currently on v2.15.3-4920-g8d052edb22.
2. Clone JGit in another directory outside Gerrit's tree.
3. Check out 'stable-5.0' branch in JGit - this should be only a few changes compared to the 5.0.2 released version as Maven dependency configured.
4. As per developer documentation about Bazel and Gerrit I set LOCAL_JGIT_REPO in lib/jgit/jgit.bzl to point to my local checkout.
5. Run 'bazel build gerrit' and getting different dependency errors everytime:
   
   

Bazel doesn't support recursive workspaces, so that all transitive dependencies
required by JGit must be resolvable through gerrit's own WORKSPACE file. This
seems currently not to be the case, but was at the time of writing the documentation
that you are referencing to.

1. $ bazel build gerrit
   ERROR: /home/gert/.cache/bazel/_bazel_gert/657d7487a554575c9d983baaafc6bd5d/external/jgit/lib/BUILD:119:1: no such package '@jzlib//jar': The repository could not be resolved and referenced by '@jgit//lib:jzlib'
   ERROR: Analysis of target '//:gerrit' failed; build aborted: Analysis failed
   
   

This doesn't sound familiar. Seems to be a new dependency in JGit and these
lines must be temporarily copied in Gerrit's own WORKSPASE file: [1], and the
documentation should be updated.

1. Variations:
   * ERROR: /home/gert/.cache/bazel/_bazel_gert/657d7487a554575c9d983baaafc6bd5d/external/jgit/lib/BUILD:10:1: no such package '@commons_compress//jar': The repository could not be resolved and referenced by '@jgit//lib:commons-compress'
   * ERROR: /home/gert/.cache/bazel/_bazel_gert/657d7487a554575c9d983baaafc6bd5d/external/jgit/lib/BUILD:153:1: no such package '@log_api//jar': The repository could not be resolved and referenced by '@jgit//lib:slf4j-api'

This is because we harmonized the names of external repositories in Gerrit
in this change: [2], but missed to do the same in JGit project, that why we have
this discrepancy. To rectify, you could adjust build in JGit and harmonize the
names or, alternatively, copy all transitive dependencies with old names from
JGit's WORKSPACE file in Gerrit's WORKSPACE file, e.g.: JGit: [3], Gerrit: [4].

[1] https://github.com/eclipse/jgit/blob/master/WORKSPACE#L36-L40
[2] https://gerrit-review.googlesource.com/c/gerrit/+/183530
[3] https://github.com/eclipse/jgit/blob/master/WORKSPACE#L42-L52
[4] https://github.com/GerritCodeReview/gerrit/blob/master/WORKSPACE#L299-L310

[Gert van Dijk]

Thanks a lot for explaining this, David! Got it to work now, I think.

More inline:

On Wednesday, 29 August 2018 12:11:15 UTC+2, David Ostrovsky wrote:

This doesn't sound familiar. Seems to be a new dependency in JGit and these

lines must be temporarily copied in Gerrit's own WORKSPASE file: [1], and the
documentation should be updated.

What do you think about this approach? https://gerrit-review.googlesource.com/c/gerrit/+/194402

I think we don't need to update the documentation with that change.

This is because we harmonized the names of external repositories in Gerrit
in this change: [2], but missed to do the same in JGit project, that why we have
this discrepancy. To rectify, you could adjust build in JGit and harmonize the
names or, alternatively, copy all transitive dependencies with old names from
JGit's WORKSPACE file in Gerrit's WORKSPACE file, e.g.: JGit: [3], Gerrit: [4].

There seems to be no way in Bazel to allow for a transition period by aliasing the WORKSPACE rules? Found Bazel issue 3219. Ouch.

[Gert van Dijk]

On Wednesday, 29 August 2018 13:10:42 UTC+2, Gert van Dijk wrote:

Thanks a lot for explaining this, David! Got it to work now, I think.

While building works, it fails at runtime now.

This happens when initializing a new site (regardless of version checked out in local JGit repository):

Exception in thread "main" java.lang.NoSuchMethodError: java.nio.ByteBuffer.mark()Ljava/nio/ByteBuffer;
        at org.eclipse.jgit.util.RawParseUtils.decodeNoFallback(RawParseUtils.java:1121)
        at org.eclipse.jgit.util.RawParseUtils.decode(RawParseUtils.java:1087)
        at org.eclipse.jgit.util.RawParseUtils.decode(RawParseUtils.java:1046)
        at org.eclipse.jgit.util.RawParseUtils.decode(RawParseUtils.java:1025)
        at org.eclipse.jgit.storage.file.FileBasedConfig.load(FileBasedConfig.java:173)
        at com.google.gerrit.server.index.GerritIndexStatus.<init>(GerritIndexStatus.java:35)
        at com.google.gerrit.server.index.IndexUtils.setReady(IndexUtils.java:58)
        at com.google.gerrit.pgm.init.InitIndex.run(InitIndex.java:71)
        at com.google.gerrit.pgm.init.SitePathInitializer.run(SitePathInitializer.java:93)
        at com.google.gerrit.pgm.init.BaseInit.run(BaseInit.java:136)
        at com.google.gerrit.pgm.util.AbstractProgram.main(AbstractProgram.java:61)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.google.gerrit.launcher.GerritLauncher.invokeProgram(GerritLauncher.java:224)
        at com.google.gerrit.launcher.GerritLauncher.mainImpl(GerritLauncher.java:120)
        at com.google.gerrit.launcher.GerritLauncher.main(GerritLauncher.java:64)
        at Main.main(Main.java:28)

Any idea?

[David Ostrovsky]

Am Mittwoch, 29. August 2018 14:23:23 UTC+2 schrieb Gert van Dijk:

On Wednesday, 29 August 2018 13:10:42 UTC+2, Gert van Dijk wrote:

Thanks a lot for explaining this, David! Got it to work now, I think.

While building works, it fails at runtime now.

This happens when initializing a new site (regardless of version checked out in local JGit repository):

Exception in thread "main" java.lang.NoSuchMethodError: java.nio.ByteBuffer.mark()Ljava/nio/ByteBuffer;
        at org.eclipse.jgit.util.RawParseUtils.decodeNoFallback(RawParseUtils.java:1121)

Can it be that you are building with Java 9 or higher and running with Java 8?
You can check the target java version by "looking" into the byte code:

  $ javap -v -cp <classpath> org.eclipse.jgit.util.RawParseUtils | grep "major version"

[Duft Markus]

Which java version are you using to build/run? Sound like a Java 9 (+) issue. A quick google brought up this issue:

https://jira.mongodb.org/browse/JAVA-2559

 

Cheers,

Markus

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

SSI Schäfer IT Solutions GmbH | Friesachstrasse 15 | 8114 Friesach | Austria
Registered Office: Friesach | Commercial Register: 49324 K | VAT no. ATU28654300
Commercial Court: Landesgericht für Zivilrechtssachen Graz

[Matthias Sohn]

I ran into the same issue earlier this week since I had played with a Java 11 early-access build and then forgot

to switch back to Java 8 before building Gerrit.

-Matthias

[Gert van Dijk]

On Wednesday, 29 August 2018 15:17:50 UTC+2, David Ostrovsky wrote:

Can it be that you are building with Java 9 or higher and running with Java 8?

Shouldn't Bazel handle selecting the right JDK version? The developer documentation also mentions:

Java 8 is still the default.

But, perhaps it's not set correctly for building the external dependency here (JGit)?

Anyway, my build host JDK selection is 8 and I don't have a newer one installed (I have a newer JRE, though, but not the default as well):

$ java -version
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-0ubuntu0.18.04.1-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

$ javac -version
javac 1.8.0_181

$ javap -version
1.8.0_181

$ update-alternatives --config javac
There is only one alternative in link group javac (providing /usr/bin/javac): /usr/lib/jvm/java-8-openjdk-amd64/bin/javac
Nothing to configure.

As far as I understand Bazel does not use the embedded JDK9/10 for building Java software.

I'm running in a Docker container where only JRE 8 is installed.

 

You can check the target java version by "looking" into the byte code:

  $ javap -v -cp <classpath> org.eclipse.jgit.util.RawParseUtils | grep "major version"

$ javap -v -cp bazel-out/k8-fastbuild/bin/external/jgit/org.eclipse.jgit/_javac/jgit/libjgit-class_classes org.eclipse.jgit.util.RawParseUtils | grep -F "major version"

  major version: 52

This seems to indicate it's all built for Java 8, no?

[David Ostrovsky]

Yes, that's correct. That's strange indeed. I cannot reproduce that problem,

though. I tried on Gerrit and JGit masters with your patch: [1] applied, with

Java 8 and set LOCAL_JGIT_REPO = "/home/davido/projects/jgit" and run:

  $ bazel build :gerrit

Bazel version 0.17.1rc1 produced a valid gerrit.war. I was able to initialize

a vanilla site and run Gerrit without any issue.

Just to confirm, if you build Gerrit from master, without building JGit from

development tree, can you run Gerrit without any issue?

I'm asking because JGit distribution is built with Java 8 as well, so there

shouldn't be any difference whether or not JGit was built locally or not:

  $ javap -v -cp ~/Downloads/org.eclipse.jgit-5.0.2.201807311906-r.jar org.eclipse.jgit.util.RawParseUtils | grep -F "major version"

  major version: 52

Except that JGit distribution was built with Maven and not with Bazel.

I think it would worth to open an issue on Bazel issue tracker.

[1] https://gerrit-review.googlesource.com/c/gerrit/+/194402

[Gert van Dijk]

On Wednesday, 29 August 2018 20:56:10 UTC+2, David Ostrovsky wrote:

Yes, that's correct. That's strange indeed. I cannot reproduce that problem,

though. I tried on Gerrit and JGit masters with your patch: [1] applied, with

Java 8 and set LOCAL_JGIT_REPO = "/home/davido/projects/jgit" and run:

  $ bazel build :gerrit

Bazel version 0.17.1rc1 produced a valid gerrit.war. I was able to initialize

a vanilla site and run Gerrit without any issue.

I tried Bazel 0.17-rc1 and it works now...! whoah. :)

I have no clue if the following lines are of any value, but I found those suspicious while building Gerrit in this context:

INFO: From Executing genrule @bazel_tools//tools/jdk:platformclasspath [for host]:
warning: [options] bootstrap class path not set in conjunction with -source 8
1 warning

INFO: From Rendering asciidoctor files for html_htmlonly:
unsupported Java version "10", defaulting to 1.7

 

Just to confirm, if you build Gerrit from master, without building JGit from

development tree, can you run Gerrit without any issue?

Yes, I could do that perfectly fine with 0.15.1/0.16.1. Those release.war files are running/initializing fine; built on the same host, running in the same container. It must be something with the local git repository that triggers it...

I totally agree it's looking like a JDK API mismatch, though.

Except that JGit distribution was built with Maven and not with Bazel.

I think it would worth to open an issue on Bazel issue tracker.

Yeah, well, apparently 0.17.x is going to fix this for me, so it's probably fixed already.

(I'm also scared about filing bug reports for Bazel - the one I filed to today was considered P0 release blocker. Haha.)

Anyway, thanks a lot for your help, David!

[David Ostrovsky]

On Wednesday, August 29, 2018 at 9:28:45 PM UTC+2, Gert van Dijk wrote:

On Wednesday, 29 August 2018 20:56:10 UTC+2, David Ostrovsky wrote:

Yes, that's correct. That's strange indeed. I cannot reproduce that problem,

though. I tried on Gerrit and JGit masters with your patch: [1] applied, with

Java 8 and set LOCAL_JGIT_REPO = "/home/davido/projects/jgit" and run:

  $ bazel build :gerrit

Bazel version 0.17.1rc1 produced a valid gerrit.war. I was able to initialize

a vanilla site and run Gerrit without any issue.

I tried Bazel 0.17-rc1 and it works now...! whoah. :)

That's crazy, but I was able to reproduce your stack trace on Bazel 0.16.1

and wrote this Bazel issue: [1].

* [1] https://github.com/bazelbuild/bazel/issues/6027

[David Ostrovsky]

Turned out, we are running in a known major Bazel bug: [2], that was only fixed

on upcoming 0.17 release, that causes Bazel to link against JDK9. Some few

methods added co-variants and they cannot be run on Java 9 any more. This

has nothing to do with a way we build JGit at all. The gerrit itself is only not

affected because those problematic methods are not used in gerrit code.

Now I feel better, that we understand the bug *and* the fix ;-)

[2] https://github.com/bazelbuild/bazel/issues/5888

[Gert van Dijk]

On Thursday, 30 August 2018 09:02:27 UTC+2, David Ostrovsky wrote:

Turned out, we are running in a known major Bazel bug: [2], that was only fixed

on upcoming 0.17 release, that causes Bazel to link against JDK9. Some few

methods added co-variants and they cannot be run on Java 9 any more. This

has nothing to do with a way we build JGit at all. The gerrit itself is only not

affected because those problematic methods are not used in gerrit code.

Now I feel better, that we understand the bug *and* the fix ;-)

[2] https://github.com/bazelbuild/bazel/issues/5888

 Oh, wow. Thanks for clarifying it all. :)

[David Pursehouse]

Aside from the known issue with bazel 0.16.1 [1], this should now work as expected with gerrit on master, stable-2.15 and stable-2.14, with the corresponding branches of JGit.

[1] https://github.com/bazelbuild/bazel/issues/5888

--