Internal Server error (NullPointerException) after interrupted push

[Roman Karlstetter]

Hi all,

 

I have a problem with one of our gerrit projects, I cannot push anymore and think I have an idea why.

 

I pushed some changes for review and interrupted the push before it was finished (didn't want to push all of them).

This resulted in the following entry in the error_log:

[2016-11-25 18:18:58,221] [SSH git-receive-pack 'MyProject' (myuser)] WARN  com.google.gerrit.server.git.AsyncReceiveCommits : Error in ReceiveCommits while processing changes for project MyProject
java.util.concurrent.ExecutionException: java.lang.InterruptedException
        at com.google.gerrit.server.git.MultiProgressMonitor.waitFor(MultiProgressMonitor.java:211)
        at com.google.gerrit.server.git.AsyncReceiveCommits.onPreReceive(AsyncReceiveCommits.java:162)
        at org.eclipse.jgit.transport.ReceivePack.service(ReceivePack.java:264)
        at org.eclipse.jgit.transport.ReceivePack.receive(ReceivePack.java:206)
        at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:97)
        at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:101)
        at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:32)
        at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:70)
        at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:442)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:417)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.InterruptedException
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:460)
        at java.util.concurrent.TimeUnit.timedWait(TimeUnit.java:348)
        at com.google.gerrit.server.git.MultiProgressMonitor.waitFor(MultiProgressMonitor.java:209)
        ... 16 more
[2016-11-25 18:18:58,227] [SSH git-receive-pack 'MyProject' (myuser)] ERROR com.google.gerrit.sshd.BaseCommand : Internal server error (user myuser account 1000000) during git-receive-pack MyProject
org.apache.sshd.common.SshException: write(ChannelOutputStream[ChannelSession[id=0, recipient=0]-ServerSessionImpl[myuser@/192.168.21.50:54380]] SSH_MSG_CHANNEL_DATA) len=412 - channel already closed
        at org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:104)
        at org.eclipse.jgit.transport.SideBandOutputStream.writeBuffer(SideBandOutputStream.java:171)
        at org.eclipse.jgit.transport.SideBandOutputStream.flushBuffer(SideBandOutputStream.java:127)
        at org.eclipse.jgit.transport.BaseReceivePack.close(BaseReceivePack.java:1710)
        at org.eclipse.jgit.transport.ReceivePack.receive(ReceivePack.java:209)
        at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:97)
        at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:101)
        at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:32)
        at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:70)
        at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:442)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:417)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[2016-11-25 18:18:58,271] [ReceiveCommits-11] WARN  com.google.gerrit.server.git.ReceiveCommits : [MyProject-1480094335495-1196cfea]Skipping change updates on MyProject because ref update failed: REJECTED_OTHER_REASON internal error

 

Then, I tried to do it again, but I got the message: no new commits. Then, I changed my commits, but I cannot push anymore, with the following entry in the error_log:

 

[2016-11-25 18:23:04,783] [ReceiveCommits-2] ERROR com.google.gerrit.server.git.ReceiveCommits : [MyProject-1480094583133-eff195f1]Can't insert change/patch set for MyProject
com.google.gerrit.extensions.restapi.RestApiException: Error inserting change/patchset
        at com.google.gerrit.server.git.ReceiveCommits$1.apply(ReceiveCommits.java:274)
        at com.google.gerrit.server.git.ReceiveCommits$1.apply(ReceiveCommits.java:265)
        at com.google.gerrit.server.git.ReceiveCommits.insertChangesAndPatchSets(ReceiveCommits.java:861)
        at com.google.gerrit.server.git.ReceiveCommits.processCommands(ReceiveCommits.java:602)
        at com.google.gerrit.server.git.AsyncReceiveCommits$Worker.run(AsyncReceiveCommits.java:89)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at com.google.gerrit.server.util.RequestScopePropagator$5.call(RequestScopePropagator.java:221)
        at com.google.gerrit.server.util.RequestScopePropagator$4.call(RequestScopePropagator.java:200)
        at com.google.gerrit.server.util.ThreadLocalRequestScopePropagator$1.call(ThreadLocalRequestScopePropagator.java:55)
        at com.google.gerrit.server.util.RequestScopePropagator$1.call(RequestScopePropagator.java:99)
        at com.google.gerrit.server.util.RequestScopePropagator$2.run(RequestScopePropagator.java:131)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:417)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.google.gerrit.server.git.UpdateException: java.util.concurrent.ExecutionException: java.lang.NullPointerException
        at com.google.gerrit.server.git.BatchUpdate.executeChangeOps(BatchUpdate.java:751)
        at com.google.gerrit.server.git.BatchUpdate.execute(BatchUpdate.java:408)
        at com.google.gerrit.server.git.BatchUpdate.execute(BatchUpdate.java:649)
        at com.google.gerrit.server.git.BatchUpdate.execute(BatchUpdate.java:644)
        at com.google.gerrit.server.git.ReceiveCommits.insertChangesAndPatchSets(ReceiveCommits.java:859)
        ... 16 more
Caused by: java.util.concurrent.ExecutionException: java.lang.NullPointerException
        at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:435)
        at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:79)
        at com.google.gerrit.server.git.BatchUpdate.executeChangeOps(BatchUpdate.java:740)
        ... 20 more
Caused by: java.lang.NullPointerException
        at com.google.gerrit.server.git.ReceiveCommits$UpdateGroupsRequest$1.updateChange(ReceiveCommits.java:2384)
        at com.google.gerrit.server.git.BatchUpdate$ChangeTask.call(BatchUpdate.java:927)
        at com.google.gerrit.server.git.BatchUpdate$ChangeTask.call(BatchUpdate.java:897)
        at com.google.gerrit.server.git.BatchUpdate$ChangeTask.call(BatchUpdate.java:873)
        at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
        at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
        at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
        at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:310)
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
        at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:61)
        at com.google.gerrit.server.git.BatchUpdate.executeChangeOps(BatchUpdate.java:732)
        ... 20 more
 

 

After I tried to push those changes, the output in the shell indicated that the changes were created (and I also can see the changes in the database), but I also see the "....(internal server error: Error inserting change/patchset)...." at the end.

Do you have any idea/suggestion how this problem can be solved so that I can push again?

Do you need any additional information?

 

Kind regards and thanks in advance,

Roman

 

[Gaurav Negi]

See if this helps?

https://groups.google.com/forum/#!topic/repo-discuss/5H2TMIkRG9M

[Roman Karlstetter]

Thanks, but the link does not help me, unfortunately.

 

I'm only having this problem with the one project where I canceled the push operation, so it really seems to be related to this specific project. All other projects work fine.

 

By the way, we're running gerrit 2.13.2.

 

Regards,

Roman

 

Original Message processed by david®

Re: Internal Server error (NullPointerException) after interrupted push (27-Nov-2016 2:33)
From:   Gaurav Negi
To:	Repo and Gerrit Discussion
Cc:	Roman.Ka...@ifta.com

[Roman Karlstetter]

Furthermore, I can push new changes to other branches of the same project without any problems and I can also push new patchsets for existing changes on the "broken" branch for this project.

 

So it seems that only the creation of new changes for the specific Project-Branch combination shows the problem.

 

I assume that there's some inconsistency in the database/git repo for this particular branch. Any idea how this could be fixed?

 

Regards,

Roman

 

Original Message processed by david®

Re-2: Internal Server error (NullPointerException) after interrupted push (28-Nov-2016 10:10)
From:   Roman Karlstetter
To:	repo-d...@googlegroups.com

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Matthias Sohn]

As a workaround you could try to amend the changes you can't push and generate a new ChangeId for each of them.

--
--
To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.

[Roman Karlstetter]

Thanks Matthias, I already tried that. Any change that I try to push for review on this particular branch fails. If I generate a new change ID, the push for review fails with "(internal server error: Error inserting change/patchset)", but the new change (including new change number) appears in the database.

 

Original Message processed by david®

Re: Re-2: Internal Server error (NullPointerException) after interrupted push (28-Nov-2016 11:51)
From:   Matthias Sohn
To:	Roman Karlstetter
Cc:	Repo and Gerrit Discussion

To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

To: matthi...@gmail.com
Cc: repo-d...@googlegroups.com

[Gaurav Negi]

Is it specific to a particular branch or the whole repo?

If just to a branch, a new branch can be created again.

Looks to me some database corruption issue. (not sure though).

To: matthi...@gmail.com
Cc: repo-d...@googlegroups.com

--
--
To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---

You received this message because you are subscribed to a topic in the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/repo-discuss/3TYjH7mx0KY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to repo-discuss+unsubscribe@googlegroups.com.