Can't build Gerrit from sources: NoClassDefFoundError javax/annotation/Generated

[Dariusz Luksza]

Since last week I can't build Gerrit from sources, I've tired building
2.14, 2.15 and current master and always got the same issue:
java.lang.RuntimeException: java.lang.NoClassDefFoundError:
javax/annotation/Generated (full bazel out attached).

I've did tried running `bazel clean --expunge` and removing
~/.gerritcodereview/bazel-cache/, none of it helped.

The same issue occurs for `bazel build gerrit` and `bazel build release`.

Here is my environment
* Bazel version:
Build label: 0.14.1
Build target:
bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Fri Jun 8 12:17:35 2018 (1528460255)
Build timestamp: 1528460255
Build timestamp as int: 1528460255
* Java version:
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)

Any hints what could be wrong?

--
Best regards

Blog: https://luksza.org http://habitchallenge.co
LinkedIn: https://linkedin.com/in/dariuszluksza

[David Ostrovsky]

Am Dienstag, 19. Juni 2018 12:20:29 UTC+2 schrieb Dariusz Łuksza:

Since last week I can't build Gerrit from sources, I've tired building
2.14, 2.15 and current master and always got the same issue:
java.lang.RuntimeException: java.lang.NoClassDefFoundError:
javax/annotation/Generated (full bazel out attached).

This seems to be Java 9 related. Particularly, see this Dagger
issue: [1] and my comment: [2] with the link to exactly the same
stack trace that you are seeing: [3].

My impression was, that the last released rules_closure already includes
the right Dagger version, though. We have switched from the forked
version: [4] to the vanilla upstream rules_closure version recently: [5].

[1] https://github.com/google/dagger/issues/880
[2] https://github.com/google/dagger/issues/880#issuecomment-350564457
[3] http://paste.openstack.org/show/628314
[4] https://gerrit-review.googlesource.com/c/gerrit/+/147231
[5] https://gerrit-review.googlesource.com/c/gerrit/+/181990

[Dariusz Luksza]

I found the root cause, although the $PATH was pointing to java8, the
$JAVA_HOME was pointing to java9. Problem solved ;) Thank you David
for help.

> --
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[David Ostrovsky]

Am Dienstag, 19. Juni 2018 13:03:45 UTC+2 schrieb David Ostrovsky:

Am Dienstag, 19. Juni 2018 12:20:29 UTC+2 schrieb Dariusz Łuksza:

Since last week I can't build Gerrit from sources, I've tired building
2.14, 2.15 and current master and always got the same issue:
java.lang.RuntimeException: java.lang.NoClassDefFoundError:
javax/annotation/Generated (full bazel out attached).

This seems to be Java 9 related. Particularly, see this Dagger
issue: [1] and my comment: [2] with the link to exactly the same
stack trace that you are seeing: [3].

My impression was, that the last released rules_closure already includes
the right Dagger version, though. We have switched from the forked
version: [4] to the vanilla upstream rules_closure version recently: [5].

I was wrong. rules_closure still depends on outdated Dagger version,
that does not support Java 9, unfortunately. I filed this issue upstream: [1].

 [1] https://github.com/bazelbuild/rules_closure/issues/275

[MartinFick]

On Tuesday, June 19, 2018 at 4:20:29 AM UTC-6, Dariusz Łuksza wrote:

Since last week I can't build Gerrit from sources, I've tired building
2.14, 2.15 and current master and always got the same issue:
java.lang.RuntimeException: java.lang.NoClassDefFoundError:
javax/annotation/Generated (full bazel out attached).

I've did tried running `bazel clean --expunge` and removing
~/.gerritcodereview/bazel-cache/, none of it helped.

The same issue occurs for `bazel build gerrit` and `bazel build release`.

I am currently facing this same problem with java 8.

Here is my java setup:

java -version
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-0ubuntu0.16.04.1-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

echo $JAVA_HOME
/usr/lib/jvm/java-8-openjdk-amd64

echo $PATH
/usr2/mfick/bin:/usr/lib/jvm/java-8-openjdk-amd64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/opt/electriccloud/electriccommander/bin

Any thoughts?  Does it maybe only work with earlier versions of java 8?

Thanks,

-Martin

openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-0ubuntu0.16.04.1-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

 java -version
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-0ubuntu0.16.04.1-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-0ubuntu0.16.04.1-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-0ubuntu0.16.04.1-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

[Nasser Grainawi]

On Sep 12, 2018, at 2:47 PM, MartinFick <mf...@codeaurora.org> wrote:

On Tuesday, June 19, 2018 at 4:20:29 AM UTC-6, Dariusz Łuksza wrote:

Since last week I can't build Gerrit from sources, I've tired building
2.14, 2.15 and current master and always got the same issue:
java.lang.RuntimeException: java.lang.NoClassDefFoundError:
javax/annotation/Generated (full bazel out attached).

I've did tried running `bazel clean --expunge` and removing
~/.gerritcodereview/bazel-cache/, none of it helped.

The same issue occurs for `bazel build gerrit` and `bazel build release`.

I am currently facing this same problem with java 8.

Here is my java setup:

java -version
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-0ubuntu0.16.04.1-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

echo $JAVA_HOME
/usr/lib/jvm/java-8-openjdk-amd64

echo $PATH
/usr2/mfick/bin:/usr/lib/jvm/java-8-openjdk-amd64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/opt/electriccloud/electriccommander/bin

Any thoughts?  Does it maybe only work with earlier versions of java 8?

Is your bazel version the latest?

Thanks,

-Martin

openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-0ubuntu0.16.04.1-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

 java -version
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-0ubuntu0.16.04.1-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-0ubuntu0.16.04.1-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-0ubuntu0.16.04.1-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, 
a Linux Foundation Collaborative Project

[mf...@codeaurora.org]

From: Nasser Grainawi <nas...@codeaurora.org>
Sent: Wednesday, September 12, 2018 3:13 PM
To: Martin Fick <mf...@codeaurora.org>
Cc: Repo and Gerrit Discussion <repo-d...@googlegroups.com>
Subject: Re: Can't build Gerrit from sources: NoClassDefFoundError javax/annotation/Generated

 

On Sep 12, 2018, at 2:47 PM, MartinFick <mf...@codeaurora.org> wrote:

On Tuesday, June 19, 2018 at 4:20:29 AM UTC-6, Dariusz Łuksza wrote:

Since last week I can't build Gerrit from sources, I've tired building
2.14, 2.15 and current master and always got the same issue:
java.lang.RuntimeException: java.lang.NoClassDefFoundError:
javax/annotation/Generated (full bazel out attached).

I've did tried running `bazel clean --expunge` and removing
~/.gerritcodereview/bazel-cache/, none of it helped.

The same issue occurs for `bazel build gerrit` and `bazel build release`.

 

I am currently facing this same problem with java 8.

 

Here is my java setup:

java -version
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-0ubuntu0.16.04.1-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

echo $JAVA_HOME
/usr/lib/jvm/java-8-openjdk-amd64

echo $ATH

/usr2/mfick/bin:/usr/lib/jvm/java-8-openjdk-amd64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/opt/electriccloud/electriccommander/bin

Any thoughts?  Does it maybe only work with earlier versions of java 8?

 

Is your bazel version the latest?

 

bazel version

Build label: 0.16.1

Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar

Build time: Mon Aug 13 13:43:36 2018 (1534167816)

Build timestamp: 1534167816

Build timestamp as int: 1534167816

Thanks,

-Martin

[David Ostrovsky]

This Bazel version is broken: [1]. You also have not told us what

version of Gerrit are you trying to build. Gerrit master is 100% JDK9

compatible, though:

  $ git describe

  v2.15.3-5179-g2f6014055d

  $ $JAVA_HOME/bin/java -version

  openjdk version "9.0.4"

  $ bazel version

  Build label: 0.18.0rc3

  $ bazel build \

      --host_java_toolchain=@bazel_tools//tools/jdk:toolchain_java9 \

      --java_toolchain=@bazel_tools//tools/jdk:toolchain_java9 \

      :release

  [...]

  Target //:release up-to-date:

  bazel-bin/release.war

INFO: Elapsed time: 209.428s, Critical Path: 209.24s

INFO: 63 processes: 45 linux-sandbox, 18 worker.

INFO: Build completed successfully, 64 total actions

[1] https://github.com/bazelbuild/bazel/issues/6027

[David Ostrovsky]

My answer wasn't accurate, sorry. The reason you cannot build any more ancient

gerrit versions starting from Bazle 0.16.0 is because of the switch to JDK9 javac.

Gerrit is using two annotation processing libraries: auto-value and Dagger Dependency

Injection through transitiive dependencies on rules_closure. Those outdated versions

cannot be built any more with JDK9 due to removal of some java packages. See this

thread: [1]. These changes must be present on stable-2-14 branch for quite some time

now: [2], [3].

See also that discussion: [4] where I raised this issue you mentioned.

* [1] https://groups.google.com/d/topic/bazel-discuss/zODjrcJtk2E/discussion

* [2] https://gerrit-review.googlesource.com/c/gerrit/+/191186

* [3] https://gerrit-review.googlesource.com/c/gerrit/+/191190

* [4] https://groups.google.com/d/topic/bazel-discuss/1116GHzkNm0/discussion

[mf...@codeaurora.org]

From: repo-d...@googlegroups.com <repo-d...@googlegroups.com> On Behalf Of David Ostrovsky
Sent: Thursday, September 13, 2018 1:02 AM
To: Repo and Gerrit Discussion <repo-d...@googlegroups.com>

 

Sorry, I am trying to build v2.15,

 

-Martin

[David Pursehouse]

You mean the actual tagged v2.15 version?  If so, you might need to locate and cherry-pick some of the bazel-related fixes that have gone into stable-2.15 since that tag.

 

 

-Martin

[David Ostrovsky]

Right, all early tags v2.14.x and v2.15.x cannot be built with Bazel >= 0.16.0.

You would have to cherry-pick auto-value and rules_closure upgrades, that I

mentioned in my previous comment or downgrade Bazel to < 0.16.0.

[Luca Milanesio]

This is something we need to address moving forward: in the past with Buck, the version (SHA1) needed to build Gerrit was recorded in the root Gerrit folder, so that when checking out a version you knew which Buck was right for building it.

Now we should do something similar with Bazel, right?

[David Ostrovsky]

I don't see why this would be needed at that point. We explicitly discourage

to use v2.14 when v2.14.42 is available. v2.14.12 can be built with any Bazel,

the same applies for the tip of stable-2.15 branch. This may change later, when

Bazel will discotinue Java 8 support per default (but would still support JDK8 with

some additional options).

[Luca Milanesio]

You should always be in a position of  rebuilding an old release, and, as you mentioned, we would needed it anyway eventually.

[Edwin Kempin]

This is also important for bisecting issues.

If old commits cannot be built it's difficult to check if it's affected by an issue or not.

[David Pursehouse]

With buck this was easy because part of the installation process was to clone the buck source code, and when using it to build gerrit it would rebuild itself at the correct revision if necessary.

As far as I know there is no similar easy way to switch between bazel versions.

[Luca Milanesio]

Maybe a question for the Bazel Team? I am sure we're not the only project with this problem :-)

[Gert van Dijk]

On Friday, 14 September 2018 11:33:25 UTC+2, David Pursehouse wrote:

With buck this was easy because part of the installation process was to clone the buck source code, and when using it to build gerrit it would rebuild itself at the correct revision if necessary.

As far as I know there is no similar easy way to switch between bazel versions.

Conceptually speaking, versioning your build tooling together with the sources, is also what I like to do.

Bazel seems not really ready for use in long-term stable branches or support for building older tags, so especially for this tooling I think it's even more important. Changes in Bazel are done frequently and only sometimes with warnings, including complete utter nonsense behavior - one of them has been here on the mailing list - with very opinionated (point) release policy. Okay I'll stop my rant here.

Specifically for Jenkins, what I do for projects I run (privately/company), is making it part of the (declarative) Jenkinsfile in each repository. The Docker image is versioned to indicate what was used in CI at the time this commit was passed. This can be considered purely informative for people building locally.

pipeline {
  agent {
    docker {
      image 'registry.domain.tld/bazel:0.16.1'
      // snip, lots of options
    }
  }
  stages {
    // snip
  }
}

(Or, alternatively, build the Docker image used, from a Dockerfile.jenkins in the repository with other agent syntax, so you can manage other tooling as well like code style checkers, separately.)

It would not really address the bisect issue, as you would have to manually switch build tooling. I guess that's the price to pay for a build tool that's not really mature yet (no offence, I think I like Bazel conceptually).

Alternatives I can think of are on a path you don't want to go, e.g. wrapper script running a specific bazel version.

Do, however, realize that this could also turn into an anti-feature. If, for some reason, in the future, a certain <buildtool> <version> doesn't run on <OS> anymore and you could safely upgrade to a new release for building those sources, you will be unnecessarily blocked by that logic.

(FWIW, I *just* see a 0.17.1 tag being pushed a minute ago.)

[Luca Milanesio]

On 14 Sep 2018, at 11:18, Gert van Dijk <gert...@gmail.com> wrote:

On Friday, 14 September 2018 11:33:25 UTC+2, David Pursehouse wrote:

With buck this was easy because part of the installation process was to clone the buck source code, and when using it to build gerrit it would rebuild itself at the correct revision if necessary.

As far as I know there is no similar easy way to switch between bazel versions.

Conceptually speaking, versioning your build tooling together with the sources, is also what I like to do.

Bazel seems not really ready for use in long-term stable branches or support for building older tags, so especially for this tooling I think it's even more important. Changes in Bazel are done frequently and only sometimes with warnings, including complete utter nonsense behavior - one of them has been here on the mailing list - with very opinionated (point) release policy. Okay I'll stop my rant here.

Specifically for Jenkins, what I do for projects I run (privately/company), is making it part of the (declarative) Jenkinsfile in each repository. The Docker image is versioned to indicate what was used in CI at the time this commit was passed. This can be considered purely informative for people building locally.

pipeline {
  agent {
    docker {
      image 'registry.domain.tld/bazel:0.16.1'
      // snip, lots of options
    }
  }
  stages {
    // snip
  }
}

(Or, alternatively, build the Docker image used, from a Dockerfile.jenkins in the repository with other agent syntax, so you can manage other tooling as well like code style checkers, separately.)

We could also NOT install Bazel in the build of the Docker image and, instead, install the right version of Bazel just before starting the build.

It would not really address the bisect issue, as you would have to manually switch build tooling.

If we take the approach of getting the version from Gerrit source, we could will install the "right version" for each run of the bisec operation. 

I guess that's the price to pay for a build tool that's not really mature yet (no offence, I think I like Bazel conceptually).

Alternatives I can think of are on a path you don't want to go, e.g. wrapper script running a specific bazel version.

Do, however, realize that this could also turn into an anti-feature. If, for some reason, in the future, a certain <buildtool> <version> doesn't run on <OS> anymore and you could safely upgrade to a new release for building those sources, you will be unnecessarily blocked by that logic.

(FWIW, I *just* see a 0.17.1 tag being pushed a minute ago.)

[Gert van Dijk]

On Friday, 14 September 2018 13:50:27 UTC+2, lucamilanesio wrote:

(Or, alternatively, build the Docker image used, from a Dockerfile.jenkins in the repository with other agent syntax, so you can manage other tooling as well like code style checkers, separately.)

We could also NOT install Bazel in the build of the Docker image and, instead, install the right version of Bazel just before starting the build.

Well, that approach I tried to illustrate too, e.g.:

Jenkinsfile:

[...]
  agent {
    dockerfile {
      filename 'Dockerfile.jenkins'
      additionalBuildArgs '--pull'
    }
  }
[...]

Dockerfile.jenkins:

FROM registry.domain.tld/ci-image-with-almost-everything:version
RUN ... # install <tool> version a.b.c.
RUN ... # install Bazel version x.y.z.

(This is still fast for any subsequent run on the same machine, because Docker build caching.)

It would not really address the bisect issue, as you would have to manually switch build tooling.

If we take the approach of getting the version from Gerrit source, we could will install the "right version" for each run of the bisec operation. 

You mean like the wrapper script approach which parses a Bazel version from somewhere and downloads and installs it? "Meh." But yeah, you could do that...

 

[Luca Milanesio]

No, don't like wrapper scripts :-)

I mean the Build Flow in Jenkins should have that logic.

That would allow to limit the impact on Gerrit build itself and limit the "smell" to the gerrit-ci-scripts part.

"Meh." But yeah, you could do that...

 

[Martin Fick]

On Friday, September 14, 2018 11:01:21 AM MDT David Pursehouse wrote:
> > On Tuesday, June 19, 2018 at 4:20:29 AM UTC-6, Dariusz Łuksza wrote:
> > Since last week I can't build Gerrit from sources, I've tired building
> > 2.14, 2.15 and current master and always got the same issue:
> > java.lang.RuntimeException: java.lang.NoClassDefFoundError:
> > javax/annotation/Generated (full bazel out attached).
> >
> > I've did tried running `bazel clean --expunge` and removing
> > ~/.gerritcodereview/bazel-cache/, none of it helped.
> >
> > The same issue occurs for `bazel build gerrit` and `bazel build release`.
> >
> > I am currently facing this same problem with java 8.

...

> > Sorry, I am trying to build v2.15,
>
> You mean the actual tagged v2.15 version? If so, you might need to locate
> and cherry-pick some of the bazel-related fixes that have gone into
> stable-2.15 since that tag.

Yes the actual tagged v2.15. The reason for trying to build this version is
so that I can make a plugin depend on the earliest released stable version
that supports the features in the plugin. In other words, I don't want to
force a plugin to depend on some "<stable>.X" version of Gerrit if it doesn't
actually have such a dependency. People who have not upgraded Gerrit should
still be able to build/run the plugin.

Thanks,

-Martin

--
The Qualcomm Innovation Center, Inc. is a member of Code
Aurora Forum, hosted by The Linux Foundation

[David Ostrovsky]

Yes you are right. As other mentioned earlier in that thread, we need to re-add

some means to older Gerrit versions with what Bazel version that Gerrit version

can be built.

Yesterday, I ran myself in exactly this issue trying to bisect non working signed

push rejection. To build very old versions of gerrit to check whether or not it worked

there I had to install Bazel version 0.6.1. I forgot what Bazel version exactly was

needed, so that It took some time with try and error session to figure that out.

In recent Gerrit releases, we tracke the minimum required Bazel version in gerrit tree,

in WORKSPACE file, so that it should be trivial to check that version and install that

bazel version:

  versions.check(minimum_bazel_version = "0.14.0")

Unfortunately Bazel doesn't seem to support multiple version on the same machine

out of the box. I commented on that thread on bazel mailing lost: [1].

[1] https://groups.google.com/d/topic/bazel-discuss/wYocSoGTJOg/discussion

[Dave Borowitz]

I had the same response as Phillip Wollerman in that thread: sad that this isn't supported out of the box, since it's how Blaze works.

Fortunately the last message in the thread is pretty promising, saying it should be easy to script:

https://groups.google.com/d/msg/bazel-discuss/wYocSoGTJOg/hLXYj2E7CAAJ