Gerrit upgrade 3.4.1 server id changes impacting many changes

[Arun]

Hi Experts,

I need your suggestions to fix the issue.

 

After the upgrade from 3.1.4  to 3.4.1 there are some potential issues which I am facing & its impacting several development activities

1. The server id on gerrit.config file had changed (automatically after the upgrade)

I realized it a day after some users complaining on open changes not visible 

So i reverted back the server id & lucky that only a day changes were missed.

I am not sure how to get back those changes

This is the error message

2. The users are not able to push the code to remote repo as the server id associated is not matching.

I get the below error message

[2021-12-08T09:38:07.211+01:00] [HTTP GET /changes/rmp-gds%2Fui~52757/revisions/current/ported_drafts/ (a247878 from 10.147.12.45)] ERROR com.google.gerrit.httpd.restapi.RestApiServlet : Error in GET /changes/rmp-gds%2Fui~52757/revisions/current/ported_drafts/: InvalidServerIdException
com.google.gerrit.server.notedb.InvalidServerIdException: invalid server id, expected a99d02c0-ea9d-41b2-8e4d-385d93ab1233: actual: f780c99d-8fc7-4fed-867a-b08e519688a9
        at com.google.gerrit.server.notedb.ChangeNotes.onLoad(ChangeNotes.java:585)

 

Best Regards

Arun

[Luca Milanesio]

On 8 Dec 2021, at 13:02, Arun <arun....@gmail.com> wrote:

Hi Experts,

I need your suggestions to fix the issue.

 

After the upgrade from 3.1.4  to 3.4.1 there are some potential issues which I am facing & its impacting several development activities

1. The server id on gerrit.config file had changed (automatically after the upgrade)

Can you reproduce the issue? If you run a Gerrit upgrade to v3.4.1 from v3.1.4, so you see the server-id updated?

Luca.

I realized it a day after some users complaining on open changes not visible 

So i reverted back the server id & lucky that only a day changes were missed.

I am not sure how to get back those changes

This is the error message

2. The users are not able to push the code to remote repo as the server id associated is not matching.

I get the below error message

[2021-12-08T09:38:07.211+01:00] [HTTP GET /changes/rmp-gds%2Fui~52757/revisions/current/ported_drafts/ (a247878 from 10.147.12.45)] ERROR com.google.gerrit.httpd.restapi.RestApiServlet : Error in GET /changes/rmp-gds%2Fui~52757/revisions/current/ported_drafts/: InvalidServerIdException
com.google.gerrit.server.notedb.InvalidServerIdException: invalid server id, expected a99d02c0-ea9d-41b2-8e4d-385d93ab1233: actual: f780c99d-8fc7-4fed-867a-b08e519688a9
        at com.google.gerrit.server.notedb.ChangeNotes.onLoad(ChangeNotes.java:585)

 

Best Regards

Arun

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/21e973ee-6266-45e5-a992-c74cd82ee89bn%40googlegroups.com.
<Server_Error500.png>

[Nasser Grainawi]

On Dec 8, 2021, at 9:13 AM, Luca Milanesio <luca.mi...@gmail.com> wrote:

On 8 Dec 2021, at 13:02, Arun <arun....@gmail.com> wrote:

Hi Experts,

I need your suggestions to fix the issue.

 

After the upgrade from 3.1.4  to 3.4.1 there are some potential issues which I am facing & its impacting several development activities

1. The server id on gerrit.config file had changed (automatically after the upgrade)

Can you reproduce the issue? If you run a Gerrit upgrade to v3.4.1 from v3.1.4, so you see the server-id updated?

Related, can you share the steps (commands) you ran to do this upgrade? I would think you’d need to run init in a new site and then copy/mount repos into that new site to have a problem like this.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/11802241-FD50-422D-A572-785B576753F3%40gmail.com.

[Arun]

Hi Luca

Thanks for your interest / response. I would like to explain a bit more. 

Yes the gerrit.config file got updated with the value

(After the upgrade)       serverId = f780c99d-8fc7-4fed-867a-b08e519688a9

1: Because of the new server id reference number all the "OPEN" changes lost the reference & upon opening the change it popped up Error message 500.

 During this time, some teams started to make changes (on 06 Dec'21) to gerrit & pushed it against the above server id & only those changes were visible

2. So to fix the larger issue i have to revert to the server id as you see below 

    (before the upgrade/ current state)      serverId = a99d02c0-ea9d-41b2-8e4d-385d93ab1233  ()

This enabled the other changes to be visible but the changes made on 06 Dec'21 are lost & not visible.

3.  Those who made the changes on 06 Dec, those repos are impacted & the team members are not able to push the changes to the remote repo. 

4. I would to share this point / issue as well although its fixed or you may suggest the best option to this issue

For the code, which needs to be reviewed the after selecting the "CodeReview button+2" it should turn to "Submit"  button but it disappeared. 

 To fix this I added "Label Verify" under the Access section for the user group & the "Submit" button appeared.

The post "https://groups.google.com/g/repo-discuss/c/blVMWCrauTs" worked out but i am nor sure why its behaving strange on prod but on QA it was smooth & it didnt through these error / confusion 

Let me know if you need more details. Your support will be highly appreciated at this time :)

BR
Arun

[Arun]

Hi 

Thanks for your reply. Here is the detailed list of steps followed. I also added some comments to make it clear

After reverting the server id, i haven't executed the reindexing I am not sure whether it will solve the issue or create another one. 

Grateful for your support

vi gerrit.config
// The upgrade had the serverid as"f780c99d-8fc7-4fed-867a-b08e519688a9"
[gerrit]
        basePath = /gerrit/site_dir/git
        canonicalWebUrl = https://git.it.volvo.net/
        serverId = f780c99d-8fc7-4fed-867a-b08e519688a9
[container]


./gerrit.sh start

// Gerrit started well & basic functionalities were working

After a day the "OPEN" changes made before the upgrade (04 Dec'21) were not working / not visible.
It was throwing an error message "Error 500"

To fix the "OPEN" changes the server id was replaced- which was the earlier copy(before the upgrade)
vi gerrit.config
serverId = a99d02c0-ea9d-41b2-8e4d-385d93ab1233
(this was the server id which was present before the upgrade)

./gerrit.sh stop
./gerrit.sh start


All the "OPEN" changes made before the upgarde 04Dec'21 came back alive
BUT the "OPEN" changes made on 06 Dec'21 the initial commits made by some teams are lost

The repos affected are not able to push the code
The error log has the below details

[2021-12-07T06:49:16.724+01:00] [HTTP GET /changes/rmp-gds%2Fui~52785/robotcomments (a390675 from 10.147.229.14)] ERROR com.google.gerrit.httpd.restapi.RestApiServlet : Error in GET /changes/rmp-gds%2Fui~52785/robotcomments: InvalidServerIdException
com.google.gerrit.server.notedb.InvalidServerIdException:

 invalid server id, expected a99d02c0-ea9d-41b2-8e4d-385d93ab1233: actual: f780c99d-8fc7-4fed-867a-b08e519688a9
        at com.google.gerrit.server.notedb.ChangeNotes.onLoad(ChangeNotes.java:585)

        at com.google.gerrit.server.notedb.AbstractChangeNotes.load(AbstractChangeNotes.java:156)
        at com.google.gerrit.server.notedb.ChangeNotes$Factory.createChecked(ChangeNotes.java:120)

BR
Arun

[Sven Selberg]

The commits on the meta branches $CHANGE_REF/meta (refs/changes/57/52757 in the case above) contains the server-id in the Author field:

    Author:  Some Name <$ACCOUNT_ID@$SERVER_ID>

    Date:       Thu Oct 11 11:14:31 2018 +0000

    Commit message subject of some...

To make them "work" again it "should" work to rewrite the history of those refs and replace the $SERVER_ID part with your actual server-id.

 

 

Best Regards

Arun

[Arun]

Hi  Sven 

Thanks for your suggestion. If you dont mind can you please highlight where (in which file)  I should check & get the details of the author & the server id

 Author:  Some Name <$ACCOUNT_ID@$SERVER_ID>

    Date:       Thu Oct 11 11:14:31 2018 +0000

Under the ".git" folder i see a file  packed-refs  whihc has come strings & text char

Merge branch 'develop' of ssh://git.it.####.net:29418/rmp-gds/ui into develop
Change-Id: I347fd7a7ff3c5acae69604a6b64887d299a5f7d8
¼[…í˜ x ˆ wþtree 93a622a009cb0977d9cbb94717aee8bcb271d4c0
parent 57ea0219b7092796616eafbfa79fe1d9eb3822ba
author account_id  <user_name@consultant.###.com> 1638955360 +0530
committer  account_id  <user_name@consultant.###.com> 1638955360 +0530

  

And many more entries similar to this

BR
Arun

[Sven Selberg]

On Thursday, December 9, 2021 at 2:46:00 PM UTC+1 arun....@gmail.com wrote:

Hi  Sven 

Thanks for your suggestion. If you dont mind can you please highlight where (in which file)  I should check & get the details of the author & the server id

 Author:  Some Name <$ACCOUNT_ID@$SERVER_ID>

    Date:       Thu Oct 11 11:14:31 2018 +0000

Under the ".git" folder i see a file  packed-refs  whihc has come strings & text char

You need to do this on the server.
In the case above:
[2021-12-07T06:49:16.724+01:00] [HTTP GET /changes/rmp-gds%2Fui~52785/robotcomments (a390675 from 10.147.229.14)] ERROR com.google.gerrit.httpd.restapi.RestApiServlet : Error in GET /changes/rmp-gds%2Fui~52785/robotcomments: InvalidServerIdException

You go to the server running gerrit and (GERRIT_SITE is were gerrit is installed) assuming that gerrit.basePath is set to the default ("git"):

$ cd $GERRIT_SITE/git/rmp-gds/ui.git

ui.git$ git log refs/changes/58/52785/meta

Those are the commits were server-id is present in Author field.

You can then use filter-branch [1] to rewrite the commits with an Author-filed with the correct server-id and move the ref (refs/changes/58/52785/meta) to this new commit-stack.
I recommend that you do this in some sort of staging environment first.

/Sven

[1] https://git-scm.com/docs/git-filter-branch

[Arun]

Hi Sven,

Thanks for the suggestions. I wasnt very sure to execute the git-filter-ranch command & with some wild guess i retried reindexing command which solved the issue. The changes made on 06 Dec was lost but it was present on local repos so the teams were able to get back the changes.

Importantly the Push to remote started to work

BR
Arun