Push for review rejected after updating to 2.14.7

687 views
Skip to first unread message

janne....@vincit.fi

unread,
Mar 16, 2018, 8:09:29 AM3/16/18
to Repo and Gerrit Discussion
Hi,

After updating to Gerrit 2.14.7 we started to experience a lot of issues when pushing changes to review (no matter if it was new change or update):

Counting objects: 22, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (22/22), done.
Writing objects: 100% (22/22), 3.39 KiB | 1.13 MiB/s, done.
Total 22 (delta 17), reused 1 (delta 0)
remote: Resolving deltas: 100% (17/17)
remote: Processing changes: new: 1, refs: 1, done    
remote: error: internal error while processing changes
To gerrit:gerrit-project.git
 ! [remote rejected] HEAD -> refs/for/master (internal error)
error: failed to push some refs to 'gerrit:gerrit-project.git'


From gerrit logs:

ssh: [2018-03-16 10:30:50,678 +0000] 12345678 gerrit.user a/1000172 LOGIN FROM 195.60.252.249
error: [2018-03-16 10:30:51,789] [SSH git-receive-pack /gerrit-project (gerrit.user)] WARN  com.google.gerrit.server.git.MultiProgressMonitor : MultiProgressMonitor worker did not call end() before returning
error: [2018-03-16 10:30:51,789] [SSH git-receive-pack /gerrit-project (gerrit.user)] WARN  com.google.gerrit.server.git.AsyncReceiveCommits : Error in ReceiveCommits while processing changes for project gerrit-project
error: java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError: com.google.gerrit.server.config.PluginConfigFactory.getProjectPluginConfigWithMergedInheritance(Lcom/google/gerrit/reviewdb/client/Project$NameKey;Ljava/lang/String;)Lorg/eclipse/jgit/lib/Config;
error:         at java.util.concurrent.FutureTask.report(FutureTask.java:122)
error:         at java.util.concurrent.FutureTask.get(FutureTask.java:206)
error:         at com.google.gerrit.server.git.WorkQueue$Task.get(WorkQueue.java:391)
error:         at com.google.gerrit.server.git.MultiProgressMonitor.waitFor(MultiProgressMonitor.java:249)
error:         at com.google.gerrit.server.git.AsyncReceiveCommits.onPreReceive(AsyncReceiveCommits.java:153)
error:         at org.eclipse.jgit.transport.ReceivePack.service(ReceivePack.java:266)
error:         at org.eclipse.jgit.transport.ReceivePack.receive(ReceivePack.java:208)
error:         at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:96)
ssh: [2018-03-16 10:30:51,791 +0000] 12345678 gerrit.user a/1000172 git-receive-pack./gerrit-project 1ms 1072ms 0 git/2.7.4
error:         at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:97)
error:         at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:30)
error:         at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:63)
error:         at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:453)
error:         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
error:         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
error:         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
error:         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
error:         at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:418)
error:         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
error:         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
error:         at java.lang.Thread.run(Thread.java:748)
error: Caused by: java.lang.NoSuchMethodError: com.google.gerrit.server.config.PluginConfigFactory.getProjectPluginConfigWithMergedInheritance(Lcom/google/gerrit/reviewdb/client/Project$NameKey;Ljava/lang/String;)Lorg/eclipse/jgit/lib/Config;
error:         at com.googlesource.gerrit.plugins.reviewers.ReviewersConfig.forProject(ReviewersConfig.java:72)
error:         at com.googlesource.gerrit.plugins.reviewers.Reviewers.getSections(Reviewers.java:147)
error:         at com.googlesource.gerrit.plugins.reviewers.Reviewers.onEvent(Reviewers.java:158)
error:         at com.googlesource.gerrit.plugins.reviewers.Reviewers.onRevisionCreated(Reviewers.java:104)
error:         at com.google.gerrit.server.extensions.events.RevisionCreated.fire(RevisionCreated.java:80)
error:         at com.google.gerrit.server.git.ReplaceOp.postUpdate(ReplaceOp.java:423)
error:         at com.google.gerrit.server.update.ReviewDbBatchUpdate.executePostOps(ReviewDbBatchUpdate.java:908)
error:         at com.google.gerrit.server.update.ReviewDbBatchUpdate.execute(ReviewDbBatchUpdate.java:343)
error:         at com.google.gerrit.server.update.ReviewDbBatchUpdate.execute(ReviewDbBatchUpdate.java:426)
error:         at com.google.gerrit.server.update.ReviewDbBatchUpdate.execute(ReviewDbBatchUpdate.java:421)
error:         at com.google.gerrit.server.git.ReceiveCommits.insertChangesAndPatchSets(ReceiveCommits.java:868)
error:         at com.google.gerrit.server.git.ReceiveCommits.processCommands(ReceiveCommits.java:610)
error:         at com.google.gerrit.server.git.AsyncReceiveCommits$Worker.run(AsyncReceiveCommits.java:81)
error:         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
error:         at com.google.gerrit.server.util.RequestScopePropagator$5.call(RequestScopePropagator.java:217)
error:         at com.google.gerrit.server.util.RequestScopePropagator$4.call(RequestScopePropagator.java:193)
error:         at com.google.gerrit.server.util.ThreadLocalRequestScopePropagator$1.call(ThreadLocalRequestScopePropagator.java:52)
error:         at com.google.gerrit.server.util.RequestScopePropagator$1.call(RequestScopePropagator.java:94)
error:         at com.google.gerrit.server.util.RequestScopePropagator$2.run(RequestScopePropagator.java:125)
error:         ... 8 more
ssh: [2018-03-16 10:30:51,803 +0000] 12345678 gerrit.user a/1000172 LOGOUT


We are running Gerrit in a self-built Docker container which is based on Alpinelinux 3.6 (I also tried with Alpinelinux version 3.7). Gerrit 2.14.6 works fine with our setup.
On Alpine 3.6 Java is 8.131.11 and on 3.7 8.151.12

Any idea what is wrong?

Edwin Kempin

unread,
Mar 16, 2018, 8:11:00 AM3/16/18
to janne....@vincit.fi, Repo and Gerrit Discussion
Looks like you forgot to update the reviewers plugin.
 
error:         at com.googlesource.gerrit.plugins.reviewers.Reviewers.getSections(Reviewers.java:147)
error:         at com.googlesource.gerrit.plugins.reviewers.Reviewers.onEvent(Reviewers.java:158)
error:         at com.googlesource.gerrit.plugins.reviewers.Reviewers.onRevisionCreated(Reviewers.java:104)
error:         at com.google.gerrit.server.extensions.events.RevisionCreated.fire(RevisionCreated.java:80)
error:         at com.google.gerrit.server.git.ReplaceOp.postUpdate(ReplaceOp.java:423)
error:         at com.google.gerrit.server.update.ReviewDbBatchUpdate.executePostOps(ReviewDbBatchUpdate.java:908)
error:         at com.google.gerrit.server.update.ReviewDbBatchUpdate.execute(ReviewDbBatchUpdate.java:343)
error:         at com.google.gerrit.server.update.ReviewDbBatchUpdate.execute(ReviewDbBatchUpdate.java:426)
error:         at com.google.gerrit.server.update.ReviewDbBatchUpdate.execute(ReviewDbBatchUpdate.java:421)
error:         at com.google.gerrit.server.git.ReceiveCommits.insertChangesAndPatchSets(ReceiveCommits.java:868)
error:         at com.google.gerrit.server.git.ReceiveCommits.processCommands(ReceiveCommits.java:610)
error:         at com.google.gerrit.server.git.AsyncReceiveCommits$Worker.run(AsyncReceiveCommits.java:81)
error:         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
error:         at com.google.gerrit.server.util.RequestScopePropagator$5.call(RequestScopePropagator.java:217)
error:         at com.google.gerrit.server.util.RequestScopePropagator$4.call(RequestScopePropagator.java:193)
error:         at com.google.gerrit.server.util.ThreadLocalRequestScopePropagator$1.call(ThreadLocalRequestScopePropagator.java:52)
error:         at com.google.gerrit.server.util.RequestScopePropagator$1.call(RequestScopePropagator.java:94)
error:         at com.google.gerrit.server.util.RequestScopePropagator$2.run(RequestScopePropagator.java:125)
error:         ... 8 more
ssh: [2018-03-16 10:30:51,803 +0000] 12345678 gerrit.user a/1000172 LOGOUT


We are running Gerrit in a self-built Docker container which is based on Alpinelinux 3.6 (I also tried with Alpinelinux version 3.7). Gerrit 2.14.6 works fine with our setup.
On Alpine 3.6 Java is 8.131.11 and on 3.7 8.151.12

Any idea what is wrong?

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

janne....@vincit.fi

unread,
Mar 16, 2018, 8:23:59 AM3/16/18
to Repo and Gerrit Discussion
Hi,

My Dockerfile downloads all the plugins automatically. It downloaded the plugin from https://gerrit-ci.gerritforge.com/job/plugin-reviewers-bazel-stable-2.14/lastSuccessfulBuild/artifact/bazel-genfiles/plugins/reviewers/reviewers.jar

Which seems to be the latest version.

Has anyone got Reviewers plugin to work with 2.14.7?

David Ostrovsky

unread,
Mar 16, 2018, 9:11:32 AM3/16/18
to Repo and Gerrit Discussion

Am Freitag, 16. März 2018 13:23:59 UTC+1 schrieb janne....@vincit.fi:
Hi,

My Dockerfile downloads all the plugins automatically. It downloaded the plugin from https://gerrit-ci.gerritforge.com/job/plugin-reviewers-bazel-stable-2.14/lastSuccessfulBuild/artifact/bazel-genfiles/plugins/reviewers/reviewers.jar

Which seems to be the latest version.

That's the problem here: Plugin is too new, and uses extension point,
that was add on stable-2.14 branch but missed 2.14.7 release, and
is going to be only available in 2.14.8.

 I appologize, that we messed this up. You have two options to recover
from this state:

a) Build gerrit core from stable-2.14 branch, you need this commit: [1]
b) Downgrade the reviewers plugin: you need to get rid of this commit: [2]

* [1] https://gerrit-review.googlesource.com/#/c/gerrit/+/164950
* [2] https://gerrit-review.googlesource.com/#/c/plugins/reviewers/+/165335

janne....@vincit.fi

unread,
Mar 19, 2018, 8:49:23 AM3/19/18
to Repo and Gerrit Discussion
Hi,

Thanks for pointing the solution.

I now compiled Gerrit and all the plugins I use and 2.14.7 seems to run fine now (on my staging environment).

One thing that was quite confusing when we started to use plugins that were not bundled with gerrit.war was that there was no explicit place from where to download plugin JARs. After some digging I found out that I could download plugins from Jenkins and it has worked fine for quite some time :)

Anyway I think I will compile both Gerrit and all the plugins we use in the future so that we can avoid this kind of issues.

David Ostrovsky

unread,
Mar 19, 2018, 2:32:29 PM3/19/18
to Repo and Gerrit Discussion

On Monday, March 19, 2018 at 1:49:23 PM UTC+1, janne....@vincit.fi wrote:
Hi,

Thanks for pointing the solution.

I now compiled Gerrit and all the plugins I use and 2.14.7 seems to run fine now (on my staging environment).

Normally, you shouldn't need to build gerrit from source.


One thing that was quite confusing when we started to use plugins that were not bundled with gerrit.war was that there was no explicit place from where to download plugin JARs. After some digging I found out that I could download plugins from Jenkins and it has worked fine for quite some time :)

Right, this should work in most cases, but in some situations,
you can have a mismatch between gerrit core and plugin versions,
even on the same stable branch. 

Anyway I think I will compile both Gerrit and all the plugins we use in the future so that we can avoid this kind of issues.

This sounds like a good idea. We extended the documentation recently
(Thanks Chad!) how to include custom plugins in gerrit war: [1] with
external dependencies.

Sven Selberg

unread,
Mar 20, 2018, 4:45:33 AM3/20/18
to Repo and Gerrit Discussion

This sounds like a good idea. We extended the documentation recently
(Thanks Chad!) how to include custom plugins in gerrit war: [1] with
external dependencies.

David Ostrovsky

unread,
Mar 20, 2018, 5:23:44 AM3/20/18
to Repo and Gerrit Discussion

Am Dienstag, 20. März 2018 09:45:33 UTC+1 schrieb Sven Selberg:

This sounds like a good idea. We extended the documentation recently
(Thanks Chad!) how to include custom plugins in gerrit war: [1] with
external dependencies.

My name is spelled "Sven"! :-) 

Thanks Sven for adding the documentation in the first place ;-)
I meant this extension: [1].

https://gerrit-review.googlesource.com/#/c/gerrit/+/162511
Reply all
Reply to author
Forward
0 new messages