Problems upgrading to 3.5.2 and running copy-approvals

[Chung Wu]

Hi all,

I'm using the official Gerrit Docker image. Yesterday, I upgraded to 3.5.2, and have been trying to run copy-approvals as instructed.

If I run via ssh, I see: 

gerrit_1  | [2022-09-08T04:27:52.712Z] [SSH gerrit copy-approvals  (chungwu)] ERROR com.google.gerrit.sshd.BaseCommand : Internal server error (user chungwu account 1000000) during gerrit copy-approvals
gerrit_1  | java.lang.IllegalArgumentException: Approval that should be copied is not copied.
gerrit_1  |     at com.google.common.base.Preconditions.checkArgument(Preconditions.java:142)
gerrit_1  |     at com.google.gerrit.server.notedb.ChangeUpdate.putCopiedApproval(ChangeUpdate.java:285)
gerrit_1  |     at com.google.gerrit.server.approval.ApprovalsUtil.persistCopiedApprovals(ApprovalsUtil.java:390)
gerrit_1  |     at com.google.gerrit.server.approval.RecursiveApprovalCopier$PersistCopiedVotesOp.updateChange(RecursiveApprovalCopier.java:105)
gerrit_1  |     at com.google.gerrit.server.update.BatchUpdate.executeChangeOps(BatchUpdate.java:680)
gerrit_1  |     at com.google.gerrit.server.update.BatchUpdate.execute(BatchUpdate.java:147)
gerrit_1  |     at com.google.gerrit.server.update.BatchUpdate.execute(BatchUpdate.java:434)
gerrit_1  |     at com.google.gerrit.server.approval.RecursiveApprovalCopier.persist(RecursiveApprovalCopier.java:77)
gerrit_1  |     at com.google.gerrit.sshd.commands.CopyApprovalsCommand.run(CopyApprovalsCommand.java:75)
gerrit_1  |     at com.google.gerrit.sshd.SshCommand.lambda$start$1(SshCommand.java:79)
gerrit_1  |     at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:494)
gerrit_1  |     at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:113)
gerrit_1  |     at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
gerrit_1  |     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
gerrit_1  |     at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
gerrit_1  |     at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:612)
gerrit_1  |     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
gerrit_1  |     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
gerrit_1  |     at java.base/java.lang.Thread.run(Thread.java:829)

If I bash into the running gerrit container, and run it with `gerrit.war`, I see:

    at com.google.inject.internal.InternalProvisionException.toProvisionException(InternalProvisionException.java:251)
    at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1103)
    at com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:878)
    at com.sun.proxy.$Proxy28.create(Unknown Source)
    at com.google.gerrit.server.index.SingleVersionModule$SingleVersionListener.start(SingleVersionModule.java:98)
    at com.google.gerrit.server.index.SingleVersionModule$SingleVersionListener.start(SingleVersionModule.java:79)
    at com.google.gerrit.lifecycle.LifecycleManager.start(LifecycleManager.java:95)
    at com.google.gerrit.pgm.CopyApprovals.run(CopyApprovals.java:70)
    at com.google.gerrit.pgm.util.AbstractProgram.main(AbstractProgram.java:61)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at com.google.gerrit.launcher.GerritLauncher.invokeProgram(GerritLauncher.java:224)
    at com.google.gerrit.launcher.GerritLauncher.mainImpl(GerritLauncher.java:120)
    at com.google.gerrit.launcher.GerritLauncher.main(GerritLauncher.java:65)
    at Main.main(Main.java:28)
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock held by another program: /var/gerrit/index/accounts_0011/write.lock
    at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:118)
    at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41)
    at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45)
    at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:800)
    at com.google.gerrit.lucene.AutoCommitWriter.<init>(AutoCommitWriter.java:35)
    at com.google.gerrit.lucene.AutoCommitWriter.<init>(AutoCommitWriter.java:31)
    at com.google.gerrit.lucene.AbstractLuceneIndex.<init>(AbstractLuceneIndex.java:138)
    at com.google.gerrit.lucene.LuceneAccountIndex.<init>(LuceneAccountIndex.java:101)
    at com.google.gerrit.lucene.LuceneAccountIndex$$FastClassByGuice$$199127807.GUICE$TRAMPOLINE(<generated>)
    at com.google.gerrit.lucene.LuceneAccountIndex$$FastClassByGuice$$199127807.apply(<generated>)
    at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:82)
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:296)
    at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1100)
    ... 15 more

(maybe because Gerrit is running at the time?)

I'm not sure how to use the Docker image to run copy-approvals without also starting the Gerrit service...

Any ideas how I should proceed?

Thanks!

Chung

[Stefan Fritsch]

Hi,

> I'm using the official Gerrit Docker image. Yesterday, I upgraded to
> 3.5.2, and have been trying to run copy-approvals as instructed.
>
> If I run via ssh, I see:
>
> gerrit_1 | [2022-09-08T04:27:52.712Z] [SSH gerrit copy-approvals
> (chungwu)] ERROR com.google.gerrit.sshd.BaseCommand : Internal server
> error (user chungwu account 1000000) during gerrit copy-approvals
> gerrit_1 | java.lang.IllegalArgumentException: Approval that should be
> copied is not copied.

is there any news on this issue? It is preventing us from upgrading. I
have the impression that this happens mainly on projects that have been
imported using the "importer" plugin.

The ticket https://bugs.chromium.org/p/gerrit/issues/detail?id=15978 is
still in state AwaitingInformation but the requested backtrace has been
provided in June.

Cheers,
Stefan

[Luca Milanesio]

Can you upload a sample repository that reproduces the issue?

The stack trace doesn’t tell much IMHO.

Luca.

[Stefan Fritsch]

Am 10.10.22 um 14:20 schrieb Luca Milanesio:

>
> Can you upload a sample repository that reproduces the issue?
>
> The stack trace doesn’t tell much IMHO.

That's difficult. All the repos where it happens are non-public and I
cannot share them. If someone else with this issue has a repo that he
can share, that would be nice.

Also, the importer plugin does not seem to have been updated beyond
2.15, so it is probably difficult to create a new reproducer. That is if
the importer plugin is the cause of the issue.

Cheers,
Stefan

[Luca Milanesio]

> On 10 Oct 2022, at 13:49, Stefan Fritsch <s...@sfritsch.de> wrote:
>
>
>
> Am 10.10.22 um 14:20 schrieb Luca Milanesio:
>> Can you upload a sample repository that reproduces the issue?
>> The stack trace doesn’t tell much IMHO.
>
> That's difficult. All the repos where it happens are non-public and I cannot share them.

If you are concerned about sharing confidential information, have you considered an Enterprise Support option? [1]

> If someone else with this issue has a repo that he can share, that would be nice.
>
> Also, the importer plugin does not seem to have been updated beyond 2.15, so it is probably difficult to create a new reproducer. That is if the importer plugin is the cause of the issue.

How do you believe we could reproduce the issue? Can you help with some simple steps?
If we cannot access your repository, and there isn’t another repository we can use to see the issue, having some steps to reproduce it is the bare minimum to start IMHO.

Luca.

[1] https://www.gerritcodereview.com/support.html#enterprise-support

[doug.r...@wandisco.com]

Stefan:

The traceback apparently dies here:

    at com.google.common.base.Preconditions.checkArgument(Preconditions.java:142)

I'm assuming that you have redacted the account name in the traceback.

With the account name AND the source code for 3.5.2 for the "Preconditions.java" file you

should be able to work out what the code was checking for and compare that to the data

available for the account in question.  You should then be able to determine what data for

that account is wrong and/or missing.  Then fix it in NoteDB.

Hope that helps.

Cheers.

Doug

[Stefan Fritsch]

Am 10.10.22 um 21:09 schrieb 'doug.r...@wandisco.com' via Repo and
Gerrit Discussion:

> Stefan:
>
> The traceback apparently dies here:
>
>     at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:142)
>
> I'm assuming that you have redacted the account name in the traceback.

yes

>
> With the account name AND the source code for 3.5.2 for the
> "Preconditions.java" file you
> should be able to work out what the code was checking for and compare
> that to the data
> available for the account in question.  You should then be able to
> determine what data for
> that account is wrong and/or missing.  Then fix it in NoteDB.

Thanks for the suggestion, but the only thing one sees is that
copiedPatchSetApproval.copied() returns false, meaning the copied flag
is false. But it is not clear how it gets into that state.

Cheers,
Stefan

[Chung Wu]

BTW I noticed that I can pass in specific projects to run copy-approvals for in the command, so I tried all my projects one by one. Turns out the only one that failed was All-Projects... 🤔

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to a topic in the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/repo-discuss/fK4zqyUi7sA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/e115c5e9-1ad9-dcec-7a9c-338d8e0dceb5%40sfritsch.de.