Re: ![remote rejected] HEAD -> refs/for/master (internal server error: Error inserting change/patchset)

[Hochhaus, Andy]

After further testing, I realized that I am able to push new changes from a fresh "git clone" of the repo. I guess this might be related to some sort of local git corruption. Either way, this is no longer a blocking issue for me.

-Andy

[Matthias Sohn]

On Sun, Aug 13, 2017 at 1:25 AM, Hochhaus, Andy <ahoc...@samegoal.com> wrote:

After further testing, I realized that I am able to push new changes from a fresh "git clone" of the repo. I guess this might be related to some sort of local git corruption. Either way, this is no longer a blocking issue for me.

use git fsck [1] to check if the local repository is corrupt. Check the server's error_log for details

of the internal server error you observed.

[1] https://git-scm.com/docs/git-fsck

-Matthias 

[Hochhaus, Andy]

Hello,

We have successfully been running gerrit 2.14.2 for more than two weeks without issue. Yesterday, however I received the following error while trying to push a new change:

 ! [remote rejected]       HEAD -> refs/for/master (internal server error: Error inserting change/patchset)

Looking on the server at the error_log reveals:

[2017-08-12 00:53:52,392] [ReceiveCommits-1] WARN  com.google.gerrit.server.git.ReceiveCommits : [sg-1502499232040-4acc9ce2]Skipping change updates on sg because ref update failed: REJECTED_OTHER_REASON no new changes

[2017-08-12 02:41:31,376] [SSH git-receive-pack '/sg' (ahochhaus)] WARN  com.google.gerrit.server.git.AsyncReceiveCommits : Error in ReceiveCommits while processing changes for project sg

java.util.concurrent.ExecutionException: java.lang.InterruptedException

        at com.google.gerrit.server.git.MultiProgressMonitor.waitFor(MultiProgressMonitor.java:208)

        at com.google.gerrit.server.git.AsyncReceiveCommits.onPreReceive(AsyncReceiveCommits.java:153)

        at org.eclipse.jgit.transport.ReceivePack.service(ReceivePack.java:266)

        at org.eclipse.jgit.transport.ReceivePack.receive(ReceivePack.java:208)

        at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:96)

        at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:97)

        at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:30)

        at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:63)

        at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:418)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:418)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.InterruptedException

        at java.lang.Object.wait(Native Method)

        at java.lang.Object.wait(Object.java:460)

        at java.util.concurrent.TimeUnit.timedWait(TimeUnit.java:348)

        at com.google.gerrit.server.git.MultiProgressMonitor.waitFor(MultiProgressMonitor.java:206)

        ... 16 more

[2017-08-12 02:41:31,386] [SSH git-receive-pack '/sg' (ahochhaus)] ERROR com.google.gerrit.sshd.BaseCommand : Internal server error (user ahochhaus account 1000000) during git-receive-pack '/sg'

org.apache.sshd.common.SshException: write(ChannelOutputStream[ChannelSession[id=0, recipient=0]-ServerSessionImpl[ahochhaus@/172.56.39.233:36824]] SSH_MSG_CHANNEL_DATA) len=52 - channel already closed

        at org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:106)

        at org.eclipse.jgit.transport.SideBandOutputStream.writeBuffer(SideBandOutputStream.java:171)

        at org.eclipse.jgit.transport.SideBandOutputStream.flushBuffer(SideBandOutputStream.java:127)

        at org.eclipse.jgit.transport.BaseReceivePack.close(BaseReceivePack.java:1790)

        at org.eclipse.jgit.transport.ReceivePack.receive(ReceivePack.java:211)

        at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:96)

        at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:97)

        at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:30)

        at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:63)

        at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:418)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:418)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

[2017-08-12 02:41:31,535] [ReceiveCommits-4] ERROR com.google.gerrit.server.git.ReceiveCommits : [sg-1502505690780-25f6bd83]Can't insert change/patch set for sg

com.google.gerrit.extensions.restapi.RestApiException: Error inserting change/patchset

        at com.google.gerrit.server.git.ReceiveCommits$1.apply(ReceiveCommits.java:282)

        at com.google.gerrit.server.git.ReceiveCommits$1.apply(ReceiveCommits.java:273)

        at com.google.gerrit.server.git.ReceiveCommits.insertChangesAndPatchSets(ReceiveCommits.java:868)

        at com.google.gerrit.server.git.ReceiveCommits.processCommands(ReceiveCommits.java:608)

        at com.google.gerrit.server.git.AsyncReceiveCommits$Worker.run(AsyncReceiveCommits.java:81)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at com.google.gerrit.server.util.RequestScopePropagator$5.call(RequestScopePropagator.java:217)

        at com.google.gerrit.server.util.RequestScopePropagator$4.call(RequestScopePropagator.java:193)

        at com.google.gerrit.server.util.ThreadLocalRequestScopePropagator$1.call(ThreadLocalRequestScopePropagator.java:52)

        at com.google.gerrit.server.util.RequestScopePropagator$1.call(RequestScopePropagator.java:94)

        at com.google.gerrit.server.util.RequestScopePropagator$2.run(RequestScopePropagator.java:125)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:418)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

Caused by: com.google.gerrit.server.update.UpdateException: java.util.concurrent.ExecutionException: java.lang.NullPointerException

        at com.google.gerrit.server.update.ReviewDbBatchUpdate.executeChangeOps(ReviewDbBatchUpdate.java:546)

        at com.google.gerrit.server.update.ReviewDbBatchUpdate.execute(ReviewDbBatchUpdate.java:300)

        at com.google.gerrit.server.update.ReviewDbBatchUpdate.execute(ReviewDbBatchUpdate.java:422)

        at com.google.gerrit.server.update.ReviewDbBatchUpdate.execute(ReviewDbBatchUpdate.java:417)

        at com.google.gerrit.server.git.ReceiveCommits.insertChangesAndPatchSets(ReceiveCommits.java:866)

        ... 16 more

Caused by: java.util.concurrent.ExecutionException: java.lang.NullPointerException

        at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:502)

        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:461)

        at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:78)

        at com.google.gerrit.server.update.ReviewDbBatchUpdate.executeChangeOps(ReviewDbBatchUpdate.java:535)

        ... 20 more

Caused by: java.lang.NullPointerException

        at com.google.gwtorm.jdbc.JdbcAccess.beginTransaction(JdbcAccess.java:46)

        at com.google.gerrit.server.update.ReviewDbBatchUpdate$ChangeTask.call(ReviewDbBatchUpdate.java:714)

        at com.google.gerrit.server.update.ReviewDbBatchUpdate$ChangeTask.call(ReviewDbBatchUpdate.java:692)

        at com.google.gerrit.server.update.ReviewDbBatchUpdate$ChangeTask.call(ReviewDbBatchUpdate.java:666)

        at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)

        at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58)

        at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75)

        at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:260)

        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)

        at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:66)

        at com.google.gerrit.server.update.ReviewDbBatchUpdate.executeChangeOps(ReviewDbBatchUpdate.java:529)

        ... 20 more

What are the correct next steps for me to debug further?

-Andy