Internal server error while pushing to gerrit for review

[Raj]

I've gerrit-2.11 installed in my server and I was testing email notification by pushing some changes to gerrit for review. After a number of pushes it started throwing following error.

$ git push origin HEAD:refs/for/master

Counting objects: 4, done.

Writing objects: 100% (3/3), 289 bytes, done.

Total 3 (delta 0), reused 0 (delta 0)

remote: Processing changes: refs: 2, done

To ssh://user@host:29418/project

 ! [remote rejected] HEAD -> refs/for/master (internal server error)

error: failed to push some refs to 'ssh://user@host:29418/project'

I've a number of projects hosted on this server. Now I'm getting the above error for all projects while trying to push something for review. I can see following exceptions in the gerrit error log.

[2015-05-26 08:25:07,797] ERROR com.google.gerrit.server.git.ReceiveCommits : Can't insert change/patchset for project

com.google.gerrit.server.git.InsertException: Error inserting change/patchset

        at com.google.gerrit.server.git.ReceiveCommits$1.apply(ReceiveCommits.java:269)

        at com.google.gerrit.server.git.ReceiveCommits$1.apply(ReceiveCommits.java:260)

        at com.google.common.util.concurrent.Futures$MappingCheckedFuture.mapException(Futures.java:1809)

        at com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:85)

        at com.google.gerrit.server.git.ReceiveCommits.insertChangesAndPatchSets(ReceiveCommits.java:803)

        at com.google.gerrit.server.git.ReceiveCommits.processCommands(ReceiveCommits.java:588)

        at com.google.gerrit.server.git.AsyncReceiveCommits$Worker.run(AsyncReceiveCommits.java:89)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

        at com.google.gerrit.server.util.RequestScopePropagator$5.call(RequestScopePropagator.java:222)

        at com.google.gerrit.server.util.RequestScopePropagator$4.call(RequestScopePropagator.java:201)

        at com.google.gerrit.server.util.ThreadLocalRequestScopePropagator$1.call(ThreadLocalRequestScopePropagator.java:55)

        at com.google.gerrit.server.util.RequestScopePropagator$1.call(RequestScopePropagator.java:98)

        at com.google.gerrit.server.util.RequestScopePropagator$2.run(RequestScopePropagator.java:131)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

        at java.util.concurrent.FutureTask.run(FutureTask.java:262)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)

        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:379)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:745)

Caused by: java.util.concurrent.ExecutionException: java.lang.NullPointerException

        at java.util.concurrent.FutureTask.report(FutureTask.java:122)

        at java.util.concurrent.FutureTask.get(FutureTask.java:188)

        at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:63)

        at com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:78)

        ... 17 more

Caused by: java.lang.NullPointerException

        at com.google.gerrit.server.query.change.EqualsFilePredicate.create(EqualsFilePredicate.java:27)

        at com.google.gerrit.server.query.change.ChangeQueryBuilder.file(ChangeQueryBuilder.java:487)

        at com.google.gerrit.server.query.change.ChangeQueryBuilder.defaultField(ChangeQueryBuilder.java:774)

        at com.google.gerrit.server.query.QueryBuilder.defaultField(QueryBuilder.java:293)

        at com.google.gerrit.server.query.QueryBuilder.toPredicate(QueryBuilder.java:220)

        at com.google.gerrit.server.query.QueryBuilder.parse(QueryBuilder.java:185)

        at com.googlesource.gerrit.plugins.reviewers.ChangeEventListener.filterMatch(ChangeEventListener.java:250)

        at com.googlesource.gerrit.plugins.reviewers.ChangeEventListener.findReviewerSections(ChangeEventListener.java:239)

        at com.googlesource.gerrit.plugins.reviewers.ChangeEventListener.findReviewers(ChangeEventListener.java:223)

        at com.googlesource.gerrit.plugins.reviewers.ChangeEventListener.onEvent(ChangeEventListener.java:160)

        at com.google.gerrit.common.ChangeHookRunner.fireEventForUnrestrictedListeners(ChangeHookRunner.java:706)

        at com.google.gerrit.common.ChangeHookRunner.fireEvent(ChangeHookRunner.java:718)

        at com.google.gerrit.common.ChangeHookRunner.doPatchsetCreatedHook(ChangeHookRunner.java:364)

        at com.google.gerrit.server.change.ChangeInserter.insert(ChangeInserter.java:275)

        at com.google.gerrit.server.git.ReceiveCommits$CreateRequest.insertChange(ReceiveCommits.java:1712)

        at com.google.gerrit.server.git.ReceiveCommits$CreateRequest.access$900(ReceiveCommits.java:1644)

        at com.google.gerrit.server.git.ReceiveCommits$CreateRequest$1.call(ReceiveCommits.java:1675)

        at com.google.gerrit.server.git.ReceiveCommits$CreateRequest$1.call(ReceiveCommits.java:1671)

        at com.google.gerrit.server.util.RequestScopePropagator$1.call(RequestScopePropagator.java:96)

        at java.util.concurrent.FutureTask.run(FutureTask.java:262)

        at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299)

        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132)

        at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:58)

        at com.google.gerrit.server.git.ReceiveCommits$CreateRequest.insertChange(ReceiveCommits.java:1670)

        at com.google.gerrit.server.git.ReceiveCommits.insertChangesAndPatchSets(ReceiveCommits.java:799)

        ... 16 more

I'm a bit new to gerrit. So, I've no idea why this is happening. Can someone help me?

[Raj]

This is actually blocking my activities. Any help is appreciated.

[David Ostrovsky]

Am Dienstag, 26. Mai 2015 15:06:53 UTC+2 schrieb Raj:

This is actually blocking my activities. Any help is appreciated.

On Tuesday, 26 May 2015 14:07:13 UTC+5:30, Raj wrote:

I've gerrit-2.11 installed in my server and I was testing email notification by pushing some changes to gerrit for review. After a number of pushes it started throwing following error.

Looks like the NPE occurs in EqualsFilePredicate that is called by reviewers plugin.

Have you tried to remove the reviewers plugin?

[Raj]

Thanks for your reply.

Yes. I've tried removing the reviewers plugin. Still I'm getting the same error.

[Saša Živkov]

On Wed, May 27, 2015 at 11:36 AM, Raj <vra...@gmail.com> wrote:

Thanks for your reply.

Yes. I've tried removing the reviewers plugin. Still I'm getting the same error.

If the reviewers plugin is removed then the stack trace (from the error_log) cannot be the same.

It cannot contain the "com.googlesource.gerrit.plugins.reviewers" as the plugin is supposed to be removed.

Can you find the error in the error_log and post it here?

 

On Tuesday, 26 May 2015 19:29:31 UTC+5:30, David Ostrovsky wrote:

Am Dienstag, 26. Mai 2015 15:06:53 UTC+2 schrieb Raj:

This is actually blocking my activities. Any help is appreciated.

On Tuesday, 26 May 2015 14:07:13 UTC+5:30, Raj wrote:

I've gerrit-2.11 installed in my server and I was testing email notification by pushing some changes to gerrit for review. After a number of pushes it started throwing following error.

Looks like the NPE occurs in EqualsFilePredicate that is called by reviewers plugin.

Have you tried to remove the reviewers plugin?

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Raj]

My server is down atm. I'll post it asap.

[David Ostrovsky]

Better late than never, I tracked down this bug: [1]. It was really a nice one.

Gerrit plugin loader code isolates a plugin guice injector from gerrit core

(and from other plugins) by copying bean for bean from gerrit core injector

to plugin guice injector. Some beans are filtered and not exposed to

plugin guice injector. This filtering is hiding very serious problem though,

(no wonder, ths bug wasn't discovered for years): when plugin code tries to

access a singleton instance that was exposed in gerrit core guice injector,

but *filtered* and *not* copied to plugin guice injector, the right instance cannot

be accessed (obviously, it wasn't copied) from the plugin guice injector, but

Guice "handles this absence gracefully" for us by creating a new virgin non

initialized (pseudo) singleton instance with unexpected results. Well, in the

case above it was NPE, but this could be anything else.

Big thank to Hugo Arès for giving me a hint where to look.

* [1] https://gerrit-review.googlesource.com/74130