internal server error: Error inserting change/patchset

[Lars Vogel]

Dear Gerrit developers,

we recently had to install a new version of Gerrit due to a server crash. We did a fresh installation, create a new project in the new Gerrit instance and pushed an existing local clone to it.

All worked fine for a while but we now getting "internal server error: Error inserting change/patchset" errors during the push.

$ git push

Counting objects: 6, done.

Delta compression using up to 8 threads.

Compressing objects: 100% (6/6), done.

Writing objects: 100% (6/6), 547 bytes | 273.00 KiB/s, done.

Total 6 (delta 5), reused 0 (delta 0)

remote: Resolving deltas: 100% (5/5)

remote: Processing changes: new: 1, refs: 1, done

remote:

remote: New Changes:

remote:   http://Sdevel.eupen.com:8080/#/c/eupen/+/162 xxx:Test neu Branch cccc

remote:

To ssh://Sdevel.eupen.com:29418/eupen

 ! [remote rejected]   HEAD -> refs/for/master (internal server error: Error inserting change/patchset)

error: failed to push some refs to 'ssh://vog...@Sdevel.eupen.com:29418/eupen'^

Below is want he see in the error log.

Any sugestion how to solve this?

Best regards, Lars

2018-04-12 12:56:00,669] [ReceiveCommits-1] ERROR com.google.gerrit.server.git.receive.ReceiveCommits : [eupen-1523530559655-5006c39c]Can't insert change/patch set for eupen

com.google.gerrit.extensions.restapi.RestApiException: Error inserting change/patchset

at com.google.gerrit.server.git.receive.ReceiveCommits$1.apply(ReceiveCommits.java:285)

at com.google.gerrit.server.git.receive.ReceiveCommits$1.apply(ReceiveCommits.java:276)

at com.google.gerrit.server.git.receive.ReceiveCommits.insertChangesAndPatchSets(ReceiveCommits.java:725)

at com.google.gerrit.server.git.receive.ReceiveCommits.processCommands(ReceiveCommits.java:548)

at com.google.gerrit.server.git.receive.AsyncReceiveCommits$Worker.run(AsyncReceiveCommits.java:117)

at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

at com.google.gerrit.server.util.RequestScopePropagator.lambda$cleanup$1(RequestScopePropagator.java:212)

at com.google.gerrit.server.util.RequestScopePropagator.lambda$context$0(RequestScopePropagator.java:191)

at com.google.gerrit.server.util.ThreadLocalRequestScopePropagator.lambda$wrapImpl$0(ThreadLocalRequestScopePropagator.java:50)

at com.google.gerrit.server.util.RequestScopePropagator$1.call(RequestScopePropagator.java:94)

at com.google.gerrit.server.util.RequestScopePropagator$2.run(RequestScopePropagator.java:125)

at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

at java.util.concurrent.FutureTask.run(Unknown Source)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)

at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:435)

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

Caused by: com.google.gerrit.server.update.UpdateException: java.util.concurrent.ExecutionException: java.io.IOException: com.google.gwtorm.server.OrmException: Unable to load change 161

at com.google.gerrit.server.update.BatchUpdate.wrapAndThrowException(BatchUpdate.java:233)

at com.google.gerrit.server.update.NoteDbBatchUpdate.execute(NoteDbBatchUpdate.java:158)

at com.google.gerrit.server.update.NoteDbBatchUpdate.execute(NoteDbBatchUpdate.java:296)

at com.google.gerrit.server.update.BatchUpdate.execute(BatchUpdate.java:283)

at com.google.gerrit.server.git.receive.ReceiveCommits.insertChangesAndPatchSets(ReceiveCommits.java:723)

... 16 more

Caused by: java.util.concurrent.ExecutionException: java.io.IOException: com.google.gwtorm.server.OrmException: Unable to load change 161

at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:503)

at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:482)

at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:79)

at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:62)

at com.google.gerrit.server.update.NoteDbBatchUpdate.execute(NoteDbBatchUpdate.java:141)

... 19 more

Caused by: java.io.IOException: com.google.gwtorm.server.OrmException: Unable to load change 161

at com.google.gerrit.lucene.LuceneChangeIndex.replace(LuceneChangeIndex.java:215)

at com.google.gerrit.lucene.LuceneChangeIndex.replace(LuceneChangeIndex.java:102)

at com.google.gerrit.server.index.change.ChangeIndexer.index(ChangeIndexer.java:207)

at com.google.gerrit.server.index.change.ChangeIndexer$IndexTask.callImpl(ChangeIndexer.java:409)

at com.google.gerrit.server.index.change.ChangeIndexer$IndexTask.callImpl(ChangeIndexer.java:401)

at com.google.gerrit.server.index.change.ChangeIndexer$AbstractIndexTask.call(ChangeIndexer.java:386)

at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)

at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58)

at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75)

... 8 more

Caused by: com.google.gwtorm.server.OrmException: Unable to load change 161

at com.google.gerrit.server.query.change.ChangeData.reloadChange(ChangeData.java:571)

at com.google.gerrit.server.query.change.ChangeData.change(ChangeData.java:558)

at com.google.gerrit.lucene.LuceneChangeIndex.replace(LuceneChangeIndex.java:209)

... 16 more

Caused by: com.google.gerrit.server.project.NoSuchChangeException: 161

at com.google.gerrit.server.notedb.ChangeNotes.onLoad(ChangeNotes.java:703)

at com.google.gerrit.server.notedb.AbstractChangeNotes.load(AbstractChangeNotes.java:167)

at com.google.gerrit.server.notedb.ChangeNotes$Factory.createChecked(ChangeNotes.java:141)

at com.google.gerrit.server.query.change.ChangeData.reloadChange(ChangeData.java:569)

... 18 more

[2018-04-12 13:00:35,588] [ReceiveCommits-1] WARN  com.google.gerrit.server.git.receive.ReceiveCommits : [eupen-1523530835588-5006c39c]Skipping change updates on eupen because ref update failed: REJECTED_OTHER_REASON no new changes

[2018-04-12 13:01:44,728] [Index-Interactive-2] ERROR com.google.gerrit.index.Schema : error getting field added of ChangeData{162}

com.google.gwtorm.server.OrmException: Unable to load change 162

at com.google.gerrit.server.query.change.ChangeData.reloadChange(ChangeData.java:571)

at com.google.gerrit.server.query.change.ChangeData.change(ChangeData.java:558)

at com.google.gerrit.server.query.change.ChangeData.getDiffSummary(ChangeData.java:492)

at com.google.gerrit.server.query.change.ChangeData.computeChangedLines(ChangeData.java:515)

at com.google.gerrit.server.query.change.ChangeData.changedLines(ChangeData.java:527)

at com.google.gerrit.server.index.change.ChangeField.lambda$static$23(ChangeField.java:463)

at com.google.gerrit.index.FieldDef.get(FieldDef.java:138)

at com.google.gerrit.index.Schema$1.apply(Schema.java:185)

at com.google.gerrit.index.Schema$1.apply(Schema.java:180)

at com.google.common.collect.Iterators$5.transform(Iterators.java:757)

at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)

at com.google.common.collect.Iterators$4.computeNext(Iterators.java:623)

at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145)

at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:140)

at com.google.gerrit.lucene.AbstractLuceneIndex.toDocument(AbstractLuceneIndex.java:289)

at com.google.gerrit.lucene.LuceneChangeIndex.replace(LuceneChangeIndex.java:207)

at com.google.gerrit.lucene.LuceneChangeIndex.replace(LuceneChangeIndex.java:102)

at com.google.gerrit.server.index.change.ChangeIndexer.index(ChangeIndexer.java:207)

at com.google.gerrit.server.index.change.ChangeIndexer$IndexTask.callImpl(ChangeIndexer.java:409)

at com.google.gerrit.server.index.change.ChangeIndexer$IndexTask.callImpl(ChangeIndexer.java:401)

at com.google.gerrit.server.index.change.ChangeIndexer$AbstractIndexTask.call(ChangeIndexer.java:386)

at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)

at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58)

at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75)

at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

at java.util.concurrent.FutureTask.run(Unknown Source)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)

at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:435)

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

[Luca Milanesio]

Problems with the indexing?

[2018-04-12 13:01:44,728] [Index-Interactive-2] ERROR com.google.gerrit.index.Schema : error getting field added of ChangeData{162}

Have you tried to shutdown, reindex and restart?

Which version are you using?

Luca.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Lars Vogel]

Yes we did stop and restart several times. Also I reindexed via "ssh -p 29418 vogella@localhost gerrit index start changes --force"

We are using Gerrit 2.15.1 (latest release)

[Matthias Sohn]

these errors look similar

https://groups.google.com/forum/#!topic/repo-discuss/nLXGQzGLYcw

https://bugs.eclipse.org/bugs/show_bug.cgi?id=528863

try rebuilding the index (offline)

https://gerrit-review.googlesource.com/Documentation/pgm-reindex.html

Problems with the indexing?

Luca.

To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.

[Lars Vogel]

Thanks Matthias. We also did this and it revealed changes from yesterday but for the changes we are still receiving errors. 

Looking into the dB see gsql I see that the changes table is empty. I guess that this relates to the new approach for persisting changes in 2.15?

From the stack trace it seems that the code still tried to access the dB.

I'm just guessing of course.

https://gerrit-review.googlesource.com/Documentation/pgm-reindex.html

Problems with the indexing?

Luca.

To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

[Luca Milanesio]

What is your noteDb/changes status?

(gerrit.config and notedb.config)

Luca.

[Saša Živkov]

On Thu, Apr 12, 2018 at 1:19 PM, Lars Vogel <lars....@vogella.com> wrote:

Dear Gerrit developers,

we recently had to install a new version of Gerrit due to a server crash. We did a fresh installation, create a new project in the new Gerrit instance and pushed an existing local clone to it.

 

For me this sounds like you lost notedb of this project as the local clone which you pushed probably didn't contain all refs/changes/* refs?

Specifically for the change 161 (from your error_log): 

Did change 161 exist before the server crash?

Do you have any refs which match refs/changes/61/161/* in that local clone repository?

--
--
To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.

[Matthias Sohn]

On Thu, Apr 12, 2018 at 11:58 PM, Saša Živkov <ziv...@gmail.com> wrote:

On Thu, Apr 12, 2018 at 1:19 PM, Lars Vogel <lars....@vogella.com> wrote:

Dear Gerrit developers,

we recently had to install a new version of Gerrit due to a server crash. We did a fresh installation, create a new project in the new Gerrit instance and pushed an existing local clone to it.

 

For me this sounds like you lost notedb of this project as the local clone which you pushed probably didn't contain all refs/changes/* refs?

If this was cloned using default settings and no additional options it lacks the noteDB notes.

You don't have a backup of the gerrit server's site directory ?

[Lars Vogel]

We didn't install a backup of the previous Gerrit. 

161 was " created" by pushing to the new Gerrit, which resulted in the error on the server. It seems to create something on the server but fails to insert into the database.

The client which pushed used to push to the old Gerrit of the crashed server. Maybe he send old metadata from the old server to the new one which creates an inconsistency.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

[Duft Markus]

Does this sound like the serverId issue discussed in another thread?

SSI Schäfer IT Solutions GmbH | Friesachstrasse 15 | 8114 Friesach | Austria
Registered Office: Friesach | Commercial Register: 49324 K | VAT no. ATU28654300
Commercial Court: Landesgericht für Zivilrechtssachen Graz

[Lars Vogel]

Markus, which other thread?

[Duft Markus]

https://groups.google.com/forum/#!searchin/repo-discuss/serverid%7Csort:date/repo-discuss/uz5rXXeLdcA/UC5i7jmrBwAJ

 

AFAIR, there was something about „if you push commits with a different serverid, the repository is broken” or something like thath. Not sure though J

[Saša Živkov]

To check the stack traces of the errors we need to know which exactly Gerrit version are you using? 

On Thu, Apr 12, 2018 at 1:19 PM, Lars Vogel <lars....@vogella.com> wrote:

--
--
To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.

[Matthias Sohn]

Lars said 2.15.1 in this thread

--
--
To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

[Saša Živkov]

On Fri, Apr 13, 2018 at 7:08 AM, Lars Vogel <lars....@vogella.com> wrote:

We didn't install a backup of the previous Gerrit. 

161 was " created" by pushing to the new Gerrit, which resulted in the error on the server. It seems to create something on the server but fails to insert into the database.

Can you check if any (change) refs were created in this project's repository in Gerrit:

$ cd $GERRIT/git/<project-name>.git

$ git show-ref | grep 161

To check about the serverId issue (mentioned by Markus Duft): if refs/changes/61/161/meta exists run:

$ git log refs/changes/61/161/meta -p

and search for any "serverId" in the output. Compare it with the serverId in your $GERRIT/etc/gerrit.config

 

--
--
To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.

[Michelle Pogado]

Hi zivkov, 

I am encountering the same exact issue.

I have tried doing your instructions and the serverId are the same.

Here are the logs:

com.google.gerrit.extensions.restapi.RestApiException: Error inserting change/patchset

        at com.google.gerrit.server.git.receive.ReceiveCommits$1.apply(ReceiveCommits.java:285)

        at com.google.gerrit.server.git.receive.ReceiveCommits$1.apply(ReceiveCommits.java:276)

        at com.google.gerrit.server.git.receive.ReceiveCommits.insertChangesAndPatchSets(ReceiveCommits.java:725)

        at com.google.gerrit.server.git.receive.ReceiveCommits.processCommands(ReceiveCommits.java:548)

        at com.google.gerrit.server.git.receive.AsyncReceiveCommits$Worker.run(AsyncReceiveCommits.java:117)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at com.google.gerrit.server.util.RequestScopePropagator.lambda$cleanup$1(RequestScopePropagator.java:212)

        at com.google.gerrit.server.util.RequestScopePropagator.lambda$context$0(RequestScopePropagator.java:191)

        at com.google.gerrit.server.util.ThreadLocalRequestScopePropagator.lambda$wrapImpl$0(ThreadLocalRequestScopePropagator.java:50)

        at com.google.gerrit.server.util.RequestScopePropagator$1.call(RequestScopePropagator.java:94)

        at com.google.gerrit.server.util.RequestScopePropagator$2.run(RequestScopePropagator.java:125)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:435)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

Caused by: com.google.gerrit.server.update.UpdateException: java.io.IOException: Permission denied

        at com.google.gerrit.server.update.BatchUpdate.wrapAndThrowException(BatchUpdate.java:233)

        at com.google.gerrit.server.update.NoteDbBatchUpdate.execute(NoteDbBatchUpdate.java:158)

        at com.google.gerrit.server.update.NoteDbBatchUpdate.execute(NoteDbBatchUpdate.java:296)

        at com.google.gerrit.server.update.BatchUpdate.execute(BatchUpdate.java:283)

        at com.google.gerrit.server.git.receive.ReceiveCommits.insertChangesAndPatchSets(ReceiveCommits.java:723)

        ... 16 more

Caused by: java.io.IOException: Permission denied

        at java.io.UnixFileSystem.createFileExclusively(Native Method)

        at java.io.File.createNewFile(File.java:1012)

        at org.eclipse.jgit.util.FS_POSIX.createNewFile(FS_POSIX.java:367)

        at org.eclipse.jgit.internal.storage.file.LockFile.lock(LockFile.java:171)

        at org.eclipse.jgit.internal.storage.file.RefDirectory.pack(RefDirectory.java:774)

        at org.eclipse.jgit.internal.storage.file.RefDirectory.pack(RefDirectory.java:701)

        at org.eclipse.jgit.internal.storage.file.PackedBatchRefUpdate.execute(PackedBatchRefUpdate.java:172)

        at org.eclipse.jgit.lib.BatchRefUpdate.execute(BatchRefUpdate.java:611)

        at com.google.gerrit.server.update.RefUpdateUtil.executeChecked(RefUpdateUtil.java:37)

        at com.google.gerrit.server.notedb.NoteDbUpdateManager.execute(NoteDbUpdateManager.java:638)

        at com.google.gerrit.server.notedb.NoteDbUpdateManager.execute(NoteDbUpdateManager.java:604)

        at com.google.gerrit.server.update.NoteDbBatchUpdate$ChangesHandle.execute(NoteDbBatchUpdate.java:353)

        at com.google.gerrit.server.update.NoteDbBatchUpdate.execute(NoteDbBatchUpdate.java:98)

        ... 19 more

Is this more an error of the change references rather than user/serverId issue?

Thanks :D

--
--
To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

[David Pursehouse]

It looks like a file permission error.  Check the contents of the git repository to make sure the gerrit user has write permission on them.

[Michelle Pogado]

Hi David, 


Indeed it is file permission error.
I needed to run a script to check that all changes are available after a fluke in our replication.
So, the folders created by the script was owned by root, and not gerrit.

Thank you so much!