Upgrade Gerrit on Docker-hosted instances

[Maria Helena Braga]

What is the correct process to upgrade a Gerrit instance that is running on a docker container?

I followed the steps below, but I keep getting NoteDB errors.

• Stop Gerrit container
• Backup the existing indexes in case a rollback is required
• Ensure all installed plugins are compatible with the new API
• Edit the gerrit.yml file with the new image version
• Edit gerrit.config and remove all java variables from "container" section
• Uncomment the line #command: init from gerrit.yml
• Run the container on "Init" mode configured previously
• Wait for execution complete --> This step fails with InvalidServerIdException. Even if I edit the serverId to the value requested, the next time I execute the container in init mode, the exceptions start again.
  
  Thanks in advance!

[Luca Milanesio]

Hi Maria,

On 30 Apr 2025, at 19:30, Maria Helena Braga <mariab...@gmail.com> wrote:

What is the correct process to upgrade a Gerrit instance that is running on a docker container?

I followed the steps below, but I keep getting NoteDB errors.

• Stop Gerrit container
• Backup the existing indexes in case a rollback is required
• Ensure all installed plugins are compatible with the new API
• Edit the gerrit.yml file with the new image version

Is the ‘gerrit.yml’ coming from an open-source project?

• Edit gerrit.config and remove all java variables from "container" section
• Uncomment the line #command: init from gerrit.yml

Where is this process coming from?

I don’t recall to have described this type of upgrade process.

• Run the container on "Init" mode configured previously

That depends on from which from and to version you are considering.

“init” may not be needed at all.

• Wait for execution complete --> This step fails with InvalidServerIdException. Even if I edit the serverId to the value requested, the next time I execute the container in init mode, the exceptions start again.
  

I believe you must have some bugs in your deployment process as you don’t persist the server id and it’s getting regenerated.

Gerrit upgrade isn’t your problem, you are not saving the server-id, that’s the issue.

HTH

Luca.

[Maria Helena Braga]

Hi Luca,

1. The gerrit.yml I created based on the "Using Gerrit in production" example here: https://hub.docker.com/r/gerritcodereview/gerrit?ref=awsmfoss.com
2. The process described is an attempt to reproduce the offline upgrade steps described here: https://www.gerritcodereview.com/3.12.html 
   1. The "Run init" step, being executed when uncommenting the line #command: init from gerrit.yml
   2. When the container is started on "Init" mode, I see it also runs the reindex step.
3. I'm trying to upgrade from Gerrit 3.10.1 to 3.12.0-rc4. Should I upgrade to version 3.11 before going to 3.12 ?
4. I'm keeping the serverId persisted on my gerrit.config file, but when I start the container in "Init mode", the serverId gets overwritten, and when the reindex step is running I get several InvalidServerIdException.

You mentioned that depending on the version, “init” may not be needed at all. I also tried to skip the init step and just bring down Gerrit 3.10.1 container, update the image on gerrit.yml to 3.12.0-rc4 and start the container again. When I do this, the Gerrit container gets stuck and nothing is written on the container logs or in error_log.

Do you have any suggestions on which other steps I should try?

Thanks, 

Maria

[Luca Milanesio]

> On 2 May 2025, at 13:46, Maria Helena Braga <mariab...@gmail.com> wrote:
>
> Hi Luca,
>
> • The gerrit.yml I created based on the "Using Gerrit in production" example here: https://hub.docker.com/r/gerritcodereview/gerrit?ref=awsmfoss.com

Ack.
> • The process described is an attempt to reproduce the offline upgrade steps described here: https://www.gerritcodereview.com/3.12.html

> • The "Run init" step, being executed when uncommenting the line #command: init from gerrit.yml
> • When the container is started on "Init" mode, I see it also runs the reindex step.

Ack.
> • I'm trying to upgrade from Gerrit 3.10.1 to 3.12.0-rc4. Should I upgrade to version 3.11 before going to 3.12 ?
If you want to avoid offline reindexing, then yes, you should go through the v3.11 upgrade first.
Otherwise, you should re-generate all the indexes offline, as described at https://www.gerritcodereview.com/3.12.html#offline-upgrade

> • I'm keeping the serverId persisted on my gerrit.config file, but when I start the container in "Init mode", the serverId gets overwritten, and when the reindex step is running I get several InvalidServerIdException.

No, it does not get overwritten at all.
You may have forgotten to mount the $GERRIT_SITE/etc maybe?

volumes:
- /external/gerrit/etc:/var/gerrit/etc

> You mentioned that depending on the version, “init” may not be needed at all. I also tried to skip the init step and just bring down Gerrit 3.10.1 container, update the image on gerrit.yml to 3.12.0-rc4 and start the container again. When I do this, the Gerrit container gets stuck and nothing is written on the container logs or in error_log.

See the above: you cannot perform a direct upgrade from v3.10.x to v3.12.x as described on [1].
If you just upgrade going through intermediate releases, then yes, the ‘init’ may not be needed at all. But in your case, it requires a downtime and a full reindexing.

HTH

Luca.

[1] https://www.gerritcodereview.com/3.12.html

> --
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/a6a8c77f-a885-4960-babf-f4509bc9157bn%40googlegroups.com.

[Maria Helena Braga]

Hi Luca,

I'm following the offline upgrade steps now, but I keep receiving the same InvalidServerIdException errors.

My site_path is /var/gerrit, and the steps I followed were:

1. Stop Gerrit-3.10
2. Download Gerrit-3.12-0-rc4.war
3. Backup folders
4. Move Gerrit-3.12-0-rc4.war to /var/gerrit/bin
5. Run init: java -jar Gerrit-3.12-0-rc4.war init -d /var/gerrit --batch
6. Run Reindex: java -jar Gerrit-3.12-0-rc4.war reindex -d /var/gerrit, which threw the error below:
   [2025-05-02 12:16:24,362] [Index-Batch-127[Index changes slice 10/52 of project eit/project/test5][eit/project/test5-10]] WARN  com.google.gerrit.server.index.change.AllChangesIndexer : Failed to read change 172910 for indexing
   com.google.gerrit.server.notedb.InvalidServerIdException: invalid server id, expected b36d7cad-dd42-4c16-9552-f6090c9c46c2: actual: a90c279c-648d-4cb3-b0a6-b8fe2f0b392e

This was my gerrit.config file when I ran the init and reindex commands:
[gerrit]
  basePath = git
        canonicalWebUrl = https://gerrit.test.com/
        serverId = b36d7cad-dd42-4c16-9552-f6090c9c46c2

[index]
  type = LUCENE

After this error, I also tried to modify the serverId value to a90c279c-648d-4cb3-b0a6-b8fe2f0b392e and run the reindex command again, which also threw the error:

[2025-05-02 12:51:39,335] [Index-Batch-190[Index changes slice 14/65 of project eit/project/test][eit/project/test-14]] WARN  com.google.gerrit.server.index.change.AllChangesIndexer : Failed to read change 104404 for indexing
com.google.gerrit.server.notedb.InvalidServerIdException: invalid server id, expected a90c279c-648d-4cb3-b0a6-b8fe2f0b392e: actual: b36d7cad-dd42-4c16-9552-f6090c9c46c2

I keep getting this loop of invalid serverIds errors, when my serverId is b36d7cad-dd42-4c16-9552-f6090c9c46c2, the error says the value should be a90c279c-648d-4cb3-b0a6-b8fe2f0b392e and vice versa.

I'm lost :( Is there any other step I should do to run the offline reindex?

Regards,

Maria H.

[Luca Milanesio]

> On 2 May 2025, at 21:27, Maria Helena Braga <mariab...@gmail.com> wrote:
>
>
> Hi Luca,
>

> I'm following the offline upgrade steps now, but I keep receiving the same InvalidServerIdException errors.
> My site_path is /var/gerrit, and the steps I followed were:
>
> • Stop Gerrit-3.10
> • Download Gerrit-3.12-0-rc4.war
> • Backup folders
> • Move Gerrit-3.12-0-rc4.war to /var/gerrit/bin
> • Run init: java -jar Gerrit-3.12-0-rc4.war init -d /var/gerrit --batch
> • Run Reindex: java -jar Gerrit-3.12-0-rc4.war reindex -d /var/gerrit, which threw the error below:
> [2025-05-02 12:16:24,362] [Index-Batch-127[Index changes slice 10/52 of project eit/project/test5][eit/project/test5-10]] WARN com.google.gerrit.server.index.change.AllChangesIndexer : Failed to read change 172910 for indexing
> com.google.gerrit.server.notedb.InvalidServerIdException: invalid server id, expected b36d7cad-dd42-4c16-9552-f6090c9c46c2: actual: a90c279c-648d-4cb3-b0a6-b8fe2f0b392e
>
> This was my gerrit.config file when I ran the init and reindex commands:
> [gerrit]
> basePath = git
> canonicalWebUrl = https://gerrit.test.com/
> serverId = b36d7cad-dd42-4c16-9552-f6090c9c46c2
>
> [index]
> type = LUCENE

The serverId you have *now* is b36d7cad-dd42-4c16-9552-f6090c9c46c2, however, the change ‘172910’ of the project `eit/project/test5` was created when the serverId was a90c279c-648d-4cb3-b0a6-b8fe2f0b392e.
That means that *someone* has amended in the past the serverId and created this inconsistency.

The problem *IS NOT* the upgrade process, but that operation that happened in the past and created this inconsistency on the change ‘172910’ of the project `eit/project/test5`.

> After this error, I also tried to modify the serverId value to a90c279c-648d-4cb3-b0a6-b8fe2f0b392e and run the reindex command again, which also threw the error:
> [2025-05-02 12:51:39,335] [Index-Batch-190[Index changes slice 14/65 of project eit/project/test][eit/project/test-14]] WARN com.google.gerrit.server.index.change.AllChangesIndexer : Failed to read change 104404 for indexing
> com.google.gerrit.server.notedb.InvalidServerIdException: invalid server id, expected a90c279c-648d-4cb3-b0a6-b8fe2f0b392e: actual: b36d7cad-dd42-4c16-9552-f6090c9c46c2

Ah, that’s the mistake you've made: it isn’t that the upgrade process has changed the serverId, but you’ve modified it manually when trying to fix the above inconsistency.
There is nothing you can do in the Gerrit’s serverId: if you care about the change ‘172910’ of the project `eit/project/test5` you have to rewrite its history in the /meta ref and amend that serverId.

> I keep getting this loop of invalid serverIds errors, when my serverId is b36d7cad-dd42-4c16-9552-f6090c9c46c2, the error says the value should be a90c279c-648d-4cb3-b0a6-b8fe2f0b392e and vice versa.

Because you have changes in the repository with a mix of serverIds: that’s not a situation that you’ve created now with the upgrade but possibly was done in the past and now comes up because of the offline reindex process.

> I'm lost :( Is there any other step I should do to run the offline reindex?

The offline reindex is fine, you’re not doing anythign wrong.
The data you have is inconsistent, that’s the issue.

You *could* use a hack to fix the situation: importedServerId (see [1]).

Adding b36d7cad-dd42-4c16-9552-f6090c9c46c2 as importedServerId in your gerrit.config, you can “pretend” that the change was coming from another Gerrit server and the error should disappear.
NOTE: This is a hack, you should understand why the serverId was different when that change was created.

HTH

Luca.

[1] https://www.gerritcodereview.com/3.7.html#import-projects-changes-from-other-gerrit-servers

> To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/22641eb2-0e5d-416a-9d1c-fc3f466f5d5cn%40googlegroups.com.

[Maria Helena Braga]

Hi Luca,

I'm following the offline reindex steps, but I still see the same InvalidServerIdException errors when I try to reindex my instance. 

My steps were:

1. Stop Gerrit 3.10.1
2. Download Gerrit 3.12-0-rc4.war file
3. Go to folder /var/gerrit and cp gerrit 3.12-0-rc4.war  /var/gerrit/bin
4. Run init:  java -jar gerrit 3.12-0-rc4.war init -d /var/gerrit --batch
   WARNING: A restricted method in java.lang.foreign.Linker has been called
   WARNING: java.lang.foreign.Linker::downcallHandle has been called by the unnamed module
   WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for this module
   
   May 02, 2025 12:32:34 PM org.apache.lucene.internal.vectorization.VectorizationProvider lookup
   WARNING: Java vector incubator module is not readable. For optimal vector performance, pass '--add-modules jdk.incubator.vector' to enable Vector API.
   Initialized /var/gerrit
   
   
5. Run reindex: java -jar gerrit 3.12-0-rc4.war reindex -d /var/gerrit

1. 
   [2025-05-02 12:16:24,362] [Index-Batch-127[Index changes slice 10/52 of project eit/project/test5][eit/project/test5-10]] WARN  com.google.gerrit.server.index.change.AllChangesIndexer : Failed to read change 172910 for indexing
   com.google.gerrit.server.notedb.InvalidServerIdException: invalid server id, expected b36d7cad-dd42-4c16-9552-f6090c9c46c2: actual: a90c279c-648d-4cb3-b0a6-b8fe2f0b392e

This was my gerrit.config when I executed the steps:

[gerrit]
  basePath = git
        canonicalWebUrl = https://gerrit.test.com/
        serverId = b36d7cad-dd42-4c16-9552-f6090c9c46c2

[index]
  type = LUCENE

After the error, I tried to change the serverId to a90c279c-648d-4cb3-b0a6-b8fe2f0b392e, and run the reindex command again, which lead to following error:

[2025-05-02 12:43:36,509] [Index-Batch-59[Index changes slice 5/37 of project eit/project/test2][eit/project/test2-5]] WARN  com.google.gerrit.server.index.change.AllChangesIndexer : Failed to read change 264703 for indexing

com.google.gerrit.server.notedb.InvalidServerIdException: invalid server id, expected a90c279c-648d-4cb3-b0a6-b8fe2f0b392e: actual: b36d7cad-dd42-4c16-9552-f6090c9c46c2

I keep getting this loop of errors, when serverId is b36d7cad-dd42-4c16-9552-f6090c9c46c2, Gerrit expects it to be a90c279c-648d-4cb3-b0a6-b8fe2f0b392e and vice-versa.

Is there anything else I should do to make the reindex work?

Thanks,

Maria H.

On Friday, May 2, 2025 at 10:08:13 AM UTC-3 Luca Milanesio wrote:

[Luca Milanesio]

> On 2 May 2025, at 18:50, Maria Helena Braga <mariab...@gmail.com> wrote:
>
> Hi Luca,
>

I believe you’re missing my last reply to this issue: did you try my suggestions on [2].
Please read the answer carefully and follow the suggestions, they’ll work if you follow them.

HTH

Luca.

[2] https://groups.google.com/g/repo-discuss/c/dA3sxnWckPA/m/IUY34x4-EAAJ

> To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/2c9b0913-d91d-4f0f-a50a-f70131f7fcd8n%40googlegroups.com.

[Maria Helena Braga]

Hi Luca,

Thanks, your suggestion to add the importedServerId helped me move forward with the upgrade. :D

But now I'm facing a different error. I'm able to run the reindex command up to 98% of completion. But then, I start getting StackOverflowError exceptions.
Do you know what could be causing this issue? I attached the logs output here.

Thanks for all the help so far!!