replication issue after upgrading to 3.6.3

[Aaron Cline]

Hello everyone:

I'm trying to flush out the remaining issue from a recent upgrade from 3.1.x to 3.6.3.  Our replication seems to not be working.  None of our recent merges to the master branch are able to sync to our remote copy.  We use replication simply as a backup copy to a  remote copy (private github repo if that matters).  For whatever reason, I get a bunch of errors for failed tasks, and since the errors are all very similar I will only paste one below.  The files that the plugin is trying to move do not seem to exist in either of the locations specified and also do not exist anywhere else in the data/replication/ folder structure.

Also, this may have been not working completely well as I see numerous other files hanging around under data/replication/

There's approximately 28750 files in data/replication/ref-updates/

There's approximately 85 files in data/replication/waiting

All of these have older timestamps ranging back weeks to years.

Prior to the upgrade on 12-22-2022, most, if not all, commits were being properly sync'd to the remote repo.

Is there a recommended source of action here?  Again, the remote repo is just meant to be a warm backup that is never directly written to and doesn't get changed or sync'd back to our primary gerrit.  We use it for searching primarily.

Thank you for any help.

Aaron

Error:

[2023-01-03T22:14:06.134Z] [ReplicateTo-github-1[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@4f3d90b9[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@45c5fbc9[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@4b020d01]]]] ERROR com.googlesource.gerrit.plugins.replication.ReplicationTasksStorage : Error while renaming task 77bcb554fd31e274b653ab596fd322eef01d86af [CONTEXT pushOneId="56bd3a01" request="SSH" ]
java.nio.file.NoSuchFileException: /var/lib/gerrit/review_site/data/replication/ref-updates/waiting/77bcb554fd31e274b653ab596fd322eef01d86af -> /var/lib/gerrit/review_site/data/replication/ref-updates/running/77bcb554fd31e274b653ab596fd322eef01d86af
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:417)
        at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:267)
        at java.base/java.nio.file.Files.move(Files.java:1422)
        at com.googlesource.gerrit.plugins.replication.ReplicationTasksStorage$Task.rename(ReplicationTasksStorage.java:411)
        at com.googlesource.gerrit.plugins.replication.ReplicationTasksStorage$Task.start(ReplicationTasksStorage.java:383)
        at com.googlesource.gerrit.plugins.replication.ReplicationTasksStorage.start(ReplicationTasksStorage.java:178)
        at com.googlesource.gerrit.plugins.replication.Destination.requestRunway(Destination.java:606)
        at com.googlesource.gerrit.plugins.replication.PushOne.doRunPushOperation(PushOne.java:414)
        at com.googlesource.gerrit.plugins.replication.PushOne.runPushOperation(PushOne.java:405)
        at com.googlesource.gerrit.plugins.replication.PushOne.lambda$run$2(PushOne.java:391)
        at com.google.gerrit.server.util.RequestScopePropagator.lambda$cleanup$1(RequestScopePropagator.java:186)
        at com.google.gerrit.server.util.RequestScopePropagator.lambda$context$0(RequestScopePropagator.java:174)
        at com.google.gerrit.server.git.PerThreadRequestScope$Propagator.lambda$scope$0(PerThreadRequestScope.java:70)
        at com.googlesource.gerrit.plugins.replication.PushOne.run(PushOne.java:394)
        at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:113)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:612)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)

[Nasser Grainawi]

On Tue, Jan 3, 2023 at 3:31 PM 'Aaron Cline' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:

Hello everyone:

I'm trying to flush out the remaining issue from a recent upgrade from 3.1.x to 3.6.3.  Our replication seems to not be working.  None of our recent merges to the master branch are able to sync to our remote copy.  We use replication simply as a backup copy to a  remote copy (private github repo if that matters).  For whatever reason, I get a bunch of errors for failed tasks, and since the errors are all very similar I will only paste one below.  The files that the plugin is trying to move do not seem to exist in either of the locations specified and also do not exist anywhere else in the data/replication/ folder structure.

The 3.1 version of the replication plugin stored the task files in a different format than the 3.6 version does, so you probably want to delete everything under data/replication when you stop Gerrit next. I thought that was in the 3.2+ release notes, but I don't see a mention of it, sorry.

Nasser

 

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/b3745d71-7b1e-4129-b9a8-6de45493152fn%40googlegroups.com.

[Aaron Cline]

On Tuesday, January 3, 2023 at 4:54:10 PM UTC-6 nasser....@linaro.org wrote:

On Tue, Jan 3, 2023 at 3:31 PM 'Aaron Cline' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:

Hello everyone:

I'm trying to flush out the remaining issue from a recent upgrade from 3.1.x to 3.6.3.  Our replication seems to not be working.  None of our recent merges to the master branch are able to sync to our remote copy.  We use replication simply as a backup copy to a  remote copy (private github repo if that matters).  For whatever reason, I get a bunch of errors for failed tasks, and since the errors are all very similar I will only paste one below.  The files that the plugin is trying to move do not seem to exist in either of the locations specified and also do not exist anywhere else in the data/replication/ folder structure.

The 3.1 version of the replication plugin stored the task files in a different format than the 3.6 version does, so you probably want to delete everything under data/replication when you stop Gerrit next. I thought that was in the 3.2+ release notes, but I don't see a mention of it, sorry.

Nasser

 

Thank you!

Aaron