Change number falls back to an old value without rebooting the gerrit sevice

[Ted Chan]

Hi, All

We have a problem when our gerrit service is running that the change nubmers fall back to older values and the changes  with duplicated change number are not imported from other servers. wo found some logs when the duplication occured at the first time like this:

[ReceiveCommits-188[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@7e6fad[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@7af15981[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@395a7b9c]]]-for-SSH git-receive-pack //device/xiaomi/tanzanite (p-wushijian)] WARN org.eclipse.jgit.internal.storage.file.PackDirectory : Pack file /home/work/repositories/miui/All-Projects.git/objects/pack/pack-eba25844d351f0109e33ef18ef20b498e1110c15.pack was deleted, removing it from pack list

java.io.FileNotFoundException: /home/work/repositories/miui/All-Projects.git/objects/pack/pack-eba25844d351f0109e33ef18ef20b498e1110c15.pack (No such file or directory)

at java.base/java.io.RandomAccessFile.open0(Native Method)

at java.base/java.io.RandomAccessFile.open(RandomAccessFile.java:346)

at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:260)

at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:215)

at org.eclipse.jgit.internal.storage.file.Pack.doOpen(Pack.java:646)

at org.eclipse.jgit.internal.storage.file.Pack.beginWindowCache(Pack.java:626)

at org.eclipse.jgit.internal.storage.file.WindowCache.load(WindowCache.java:510)

at org.eclipse.jgit.internal.storage.file.WindowCache.getOrLoad(WindowCache.java:602)

at org.eclipse.jgit.internal.storage.file.WindowCache.get(WindowCache.java:385)

at org.eclipse.jgit.internal.storage.file.WindowCursor.pin(WindowCursor.java:335)

at org.eclipse.jgit.internal.storage.file.WindowCursor.copy(WindowCursor.java:234)

at org.eclipse.jgit.internal.storage.file.Pack.readFully(Pack.java:602)

at org.eclipse.jgit.internal.storage.file.Pack.load(Pack.java:785)

at org.eclipse.jgit.internal.storage.file.Pack.get(Pack.java:273)

at org.eclipse.jgit.internal.storage.file.PackDirectory.open(PackDirectory.java:219)

at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedObject(ObjectDirectory.java:420)

at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:384)

at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObjectWithoutRestoring(ObjectDirectory.java:375)

at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject(ObjectDirectory.java:360)

at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:140)

at com.google.gerrit.server.notedb.IntBlob.parse(IntBlob.java:66)

at com.google.gerrit.server.notedb.IntBlob.parse(IntBlob.java:56)

at com.google.gerrit.server.notedb.RepoSequence.acquire(RepoSequence.java:271)

at com.google.gerrit.server.notedb.RepoSequence.lambda$next$1(RepoSequence.java:227)

at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)

at com.github.rholder.retry.Retryer.call(Retryer.java:160)

at com.google.gerrit.server.notedb.RepoSequence.next(RepoSequence.java:221)

at com.google.gerrit.server.notedb.Sequences.nextChangeIds(Sequences.java:141)

at com.google.gerrit.server.git.receive.ReceiveCommits.selectNewAndReplacedChangesFromMagicBranch(ReceiveCommits.java:2536)

at com.google.gerrit.server.git.receive.ReceiveCommits.processCommandsUnsafe(ReceiveCommits.java:788)

at com.google.gerrit.server.git.receive.ReceiveCommits.processCommands(ReceiveCommits.java:684)

at com.google.gerrit.server.git.receive.AsyncReceiveCommits.lambda$preReceive$2(AsyncReceiveCommits.java:395)

at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264).

The logs may suggest that a gc was runing when gerrit aquired the change number and after the gc, the refs/sequences/changes pointed to an old blob( the history of refs/sequences/changes supports).

Maybe a solution of this problem is to do some check between change number in memory and the refs/sequences/changes pointed at to prevent falling back.

The duplication of change numbers makes a lot of problems like bad change queries.

Look forward to all replies.

[Ted Chan]

The Gerrit version is 3.7.2

[Fabio Ponciroli]

Hi Ted,

The problem you are describing has been resolved in 3.9 [1].

Considering 3.9 is EOL, I suggest upgrading to a version >= 3.10.

HTH,

Ponch

[1]: https://gerrit-review.googlesource.com/c/gerrit/+/448401

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/f5a2f934-7e74-4d8c-8a56-0d17b3d9939bn%40googlegroups.com.

[Nasser Grainawi]

On Wed, Jul 2, 2025 at 3:17 AM Fabio Ponciroli <fabio.p...@gmail.com> wrote:

Hi Ted,

The problem you are describing has been resolved in 3.9 [1].

That's only true if someone ran `gerrit sequence set`, correct? If this happened without anyone running that command, then I think it's a different issue. We've seen this happen twice now, once on 3.5 and once on 3.9 and we didn't have a clear root cause in either case. We're planning to add a safety check to just fail if a sequence tries to go backwards, but otherwise we're not sure how best to approach it.

Nasser

 

To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/CAPYXC5MmJ5%3DRtSuZzZBxD5uX04b%2BMszMMh2-Ze0TZwYftG%2B%2Biw%40mail.gmail.com.

[Ted Chan]

HI, Fabio

As Nasser's suggestion, The patch(https://gerrit-review.googlesource.com/c/gerrit/+/448401) seems to fix problems of setting change number from command line. 

That seems not to be  the case. 

As  the logs from our Gerrit service suggests, the problem occured when acquiring change number for the lost of pack file（ maybe a concurrently running of gc of all-projects, but why gc remote the blob the refs/sequences/changes pointed at?）.

[Luca Milanesio]

On 3 Jul 2025, at 09:24, Ted Chan <ted.c...@gmail.com> wrote:

HI, Fabio

As Nasser's suggestion, The patch(https://gerrit-review.googlesource.com/c/gerrit/+/448401) seems to fix problems of setting change number from command line. 

That seems not to be  the case. 

As  the logs from our Gerrit service suggests, the problem occured when acquiring change number for the lost of pack file（ maybe a concurrently running of gc of all-projects, but why gc remote the blob the refs/sequences/changes pointed at?）.

I believe you have GC settings issues: the GC process should *never ever* remove packfiles that are referenced and “live” with pointers to it, unless you’re doing something wrong.

Q1: Are you using Git GC or JGit GC?

Q2: How do you prune (or preserve) packfiles in the GC phase?

Luca.

To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/395772c5-9ab2-4efe-bcc5-4db2f8d908b1n%40googlegroups.com.

[Ted Chan]

Hi, Luca

The git gc configuration looks like:

[gc]
    auto = 0
    autoDetach = false
    pruneexpire = 2.hour.ago
    prunepackexpire = 30.minutes.ago
    #writeCommitGraph = true

From the configuration, autogc is turn off. 

Q1: We use JGit GC ( Gerrit service does GCs everyday)

Q2: As the configuration suggests, The GC will  prune objects/pack : pruneexpire = 2.hour.ago, prunepackexpire = 30.minutes.ago

Look forward to your replies. 

Ted

[Luca Milanesio]

On 7 Jul 2025, at 10:51, Ted Chan <ted.c...@gmail.com> wrote:

Hi, Luca

The git gc configuration looks like:

[gc]
    auto = 0
    autoDetach = false
    pruneexpire = 2.hour.ago
    prunepackexpire = 30.minutes.ago
    #writeCommitGraph = true

Is this the configuration of All-Projects.git/config?

How often you perform GC on All-Projects.git?

Luca.

To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/15c75a8d-37cc-4e88-b5b3-be14ba12e80cn%40googlegroups.com.

[kaushikl]

We have also seen changes and accounts sequence rolling back to an older value. I have proposed https://gerrit-review.googlesource.com/c/gerrit/+/488941, which serves as a safety net to help prevent this issue. While it doesn’t cover all scenarios (multi-primary setups for example), it’s still a valuable safeguard to have in place.