Gerrit U[grade v2.14 to v3.2 - Post Migration Issues

148 views
Skip to first unread message

Nandha Kumar Nagarajan

unread,
Feb 17, 2021, 9:50:16 AM2/17/21
to Repo and Gerrit Discussion
Hi Team,

Requesting your opinions on below queries

Query 1: After upgrading to v3.2, I was checking the sshd_log file and noticed the following logs. In the log, there is this word (port=22) present is all lines. I am sure I have configured and mentioned Gerrrit SSH port as 29418. Any idea why it is showing as port=22 in the logs. 

eg: 
[2021-02-15T17:02:19.145+0000] c2952765 [sshd-SshDaemon[254a9f65](port=22)-nio2-thread-2]

But I have ensured, communication is happening via port 29418 only. In logs alone, it is mentioned as port=22. In gerrit.config file also I have ensured ListenAddress=*:29418 and AdvertiseAddress is not defined (by default takes value of ListenAddress )
Any suggestions?

  Query 2: After upgrading our Gerrit staging env from v2.14 -> v2.16 -> v3.2.2, we noticed deadlock occurred a couple of times. So just wanted to understand are they any possible reasons / causes for this ?

Note:
- We have checked gerrit logs and Java thread dump and don't see anything unusual
- Just thought of updating this as well. Our Gerrit Staging env is running with 2 replicas attached to it. During the time of this deadlock occurrence, only master was upgraded to v3.2.2 and replicas were still running in v2.14

Thanks,
Nandha Kumar N

Matthias Sohn

unread,
Feb 17, 2021, 10:44:30 AM2/17/21
to Nandha Kumar Nagarajan, Repo and Gerrit Discussion
On Wed, Feb 17, 2021 at 3:50 PM Nandha Kumar Nagarajan <nandh...@gmail.com> wrote:
Hi Team,

Requesting your opinions on below queries

Query 1: After upgrading to v3.2, I was checking the sshd_log file and noticed the following logs. In the log, there is this word (port=22) present is all lines. I am sure I have configured and mentioned Gerrrit SSH port as 29418. Any idea why it is showing as port=22 in the logs. 

eg: 
[2021-02-15T17:02:19.145+0000] c2952765 [sshd-SshDaemon[254a9f65](port=22)-nio2-thread-2]

But I have ensured, communication is happening via port 29418 only. In logs alone, it is mentioned as port=22. In gerrit.config file also I have ensured ListenAddress=*:29418 and AdvertiseAddress is not defined (by default takes value of ListenAddress )
Any suggestions?

  Query 2: After upgrading our Gerrit staging env from v2.14 -> v2.16 -> v3.2.2, we noticed deadlock occurred a couple of times. So just wanted to understand are they any possible reasons / causes for this ?

Note:
- We have checked gerrit logs and Java thread dump and don't see anything unusual

how did you detect there is a deadlock ? Any details about what was in deadlock ?
Why didn't you upgrade to the latest service release of the 3.2.x series which is 3.2.7 ?
Look at the many fixes and improvements between 3.2.2 and 3.2.7:
 
- Just thought of updating this as well. Our Gerrit Staging env is running with 2 replicas attached to it. During the time of this deadlock occurrence, only master was upgraded to v3.2.2 and replicas were still running in v2.14

replication shouldn't depend on the version of the replica as long as it can process git protocol
 
Thanks,
Nandha Kumar N

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/a1c95eec-cef8-4401-8b75-56d48c1bf90cn%40googlegroups.com.

Luca Milanesio

unread,
Feb 17, 2021, 10:47:06 AM2/17/21
to Repo and Gerrit Discussion, Luca Milanesio

On 17 Feb 2021, at 15:44, Matthias Sohn <matthi...@gmail.com> wrote:

On Wed, Feb 17, 2021 at 3:50 PM Nandha Kumar Nagarajan <nandh...@gmail.com> wrote:
Hi Team,

Requesting your opinions on below queries

Query 1: After upgrading to v3.2, I was checking the sshd_log file and noticed the following logs. In the log, there is this word (port=22) present is all lines. I am sure I have configured and mentioned Gerrrit SSH port as 29418. Any idea why it is showing as port=22 in the logs. 

eg: 
[2021-02-15T17:02:19.145+0000] c2952765 [sshd-SshDaemon[254a9f65](port=22)-nio2-thread-2]

But I have ensured, communication is happening via port 29418 only. In logs alone, it is mentioned as port=22. In gerrit.config file also I have ensured ListenAddress=*:29418 and AdvertiseAddress is not defined (by default takes value of ListenAddress )
Any suggestions?

  Query 2: After upgrading our Gerrit staging env from v2.14 -> v2.16 -> v3.2.2, we noticed deadlock occurred a couple of times. So just wanted to understand are they any possible reasons / causes for this ?

Note:
- We have checked gerrit logs and Java thread dump and don't see anything unusual

how did you detect there is a deadlock ? Any details about what was in deadlock ?

I believe JavaMelody gives you that information.
There are also metrics exposed for that.

Why didn't you upgrade to the latest service release of the 3.2.x series which is 3.2.7 ?
Look at the many fixes and improvements between 3.2.2 and 3.2.7:

That’s a very good question :-)

People should *always* upgrade to the latest patch-level available.

Luca.

 
- Just thought of updating this as well. Our Gerrit Staging env is running with 2 replicas attached to it. During the time of this deadlock occurrence, only master was upgraded to v3.2.2 and replicas were still running in v2.14

replication shouldn't depend on the version of the replica as long as it can process git protocol
 
Thanks,
Nandha Kumar N

-- 
-- 
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/a1c95eec-cef8-4401-8b75-56d48c1bf90cn%40googlegroups.com.

-- 
-- 
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

Nandha Kumar Nagarajan

unread,
Feb 25, 2021, 5:43:12 AM2/25/21
to Luca Milanesio, Repo and Gerrit Discussion
Hi Team,

As suggested, I have upgraded gerrit to the latest patch version 3.2.7 and we are testing it. In one of the testing we noticed, "ref-replicated" stream events are not coming

I mean, following command is not returning any output. But when I check the replication log, it shows the replication is completed. Any suggestions please let me know

$ ssh -p 24466 gerritqa.com gerrit stream-events -s ref-replicated

Whereas, I can able to see the output for following event
$  ssh -p 24466 gerritqa.com gerrit stream-events -s ref-replication-scheduled


To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/1D776A2D-A1F6-4BF7-AE5B-7C2779C2A420%40gmail.com.

Luca Milanesio

unread,
Feb 25, 2021, 8:43:26 AM2/25/21
to Repo and Gerrit Discussion
On 25 Feb 2021, at 10:42, Nandha Kumar Nagarajan <nandh...@gmail.com> wrote:

Hi Team,

As suggested, I have upgraded gerrit to the latest patch version 3.2.7 and we are testing it. In one of the testing we noticed, "ref-replicated" stream events are not coming

I mean, following command is not returning any output. But when I check the replication log, it shows the replication is completed. Any suggestions please let me know

$ ssh -p 24466 gerritqa.com gerrit stream-events -s ref-replicated

Whereas, I can able to see the output for following event
$  ssh -p 24466 gerritqa.com gerrit stream-events -s ref-replication-scheduled

Have you checked the replication_log for any errors?
It could well be that the replication failed and therefore the ref-replicated event isn’t produced.

HTH

Luca.

Nandha Kumar Nagarajan

unread,
Mar 9, 2021, 3:15:05 AM3/9/21
to Luca Milanesio, Repo and Gerrit Discussion
Hi Team,

Thanks for the update on ref-replicated events. Yes, it is working as suggested.

I have the following 2 queries regarding NoteDB
1) We are not able to find any docs to gain a deeper understanding of how NoteDB works other than below link. Previously when we have postgresql, we had separate log file to look for errors and Javamelody shows Postgresql connections and metrics

But after we moved to NoteDB completely(Postgresql is disabled), we are kind of not sure how to troubleshoot and monitor(Java melody not showing any info on Notedb) in case of any failures. Any information will be greatly helpful

2) While going through gerrit config documentation (https://gerrit.com/Documentation/config-gerrit.html#accountPatchReviewDb), we noticed this section "accountPatchReviewDb" and it shows supported DB types as H2, POSTGRESQL, MARIADB, and MYSQL. 

I checked our existing gerrit configuration (gerrit.config) and there is no section like "accountPatchReviewDb". So just wanted to understand what this section is and is this mandatory for v3.2.7 ?

On Mon, Mar 1, 2021 at 3:38 PM Luca Milanesio <luca.mi...@gmail.com> wrote:


On 1 Mar 2021, at 09:54, Luca Milanesio <luca.mi...@gmail.com> wrote:



On 1 Mar 2021, at 05:38, Nandha Kumar Nagarajan <nandh...@gmail.com> wrote:

Hi Luca,

I can see this ref-replicated events when I push a new change to gerrit

But when I run `start replication <repo>` and if the repo is already in sync with replicas, then I am not seeing  ref-replicated events. Hope that is expected one

If a ref was updated and it was replicated, you will get the ref-replicated event, otherwise, you won’t.

Actually, if the ref-update failed you still get a ref-replicated event with the ’status=failed’ field.

Luca.


HTH

Luca.


Jacek Centkowski

unread,
Mar 9, 2021, 3:59:37 AM3/9/21
to Repo and Gerrit Discussion
Hi Nandha,

I'm answering your second question :)
As mentioned in documentation accountPatchReviewDb is responsible for storing reviewed/not reviewed bit for files put in the review. It was decided that it cannot be implemented in the NoteDB (it didn't make sense to keep history of reviewed/not reviewed files in git forever) hence it had to stay in regular DB.
You have two choices:
1. since you are migrating from older version it means that you used to have real DB backing it up therefore you can create DB for accountPatchReviewDb and pass it as configuration parameter
(check documentation on how to do taht and how to migrate existing data)
2. let it be the default one (H2) wich is OK for small shops but it is way to slow for bigger instances (see option 1 for more performant solution)

Regards
Jacek

Luca Milanesio

unread,
Mar 9, 2021, 4:19:56 AM3/9/21
to Repo and Gerrit Discussion

On 9 Mar 2021, at 08:59, Jacek Centkowski <geminica...@gmail.com> wrote:

Hi Nandha,

I'm answering your second question :)
As mentioned in documentation accountPatchReviewDb is responsible for storing reviewed/not reviewed bit for files put in the review. It was decided that it cannot be implemented in the NoteDB (it didn't make sense to keep history of reviewed/not reviewed files in git forever) hence it had to stay in regular DB.
You have two choices:
1. since you are migrating from older version it means that you used to have real DB backing it up therefore you can create DB for accountPatchReviewDb and pass it as configuration parameter
(check documentation on how to do taht and how to migrate existing data)
2. let it be the default one (H2) wich is OK for small shops but it is way to slow for bigger instances (see option 1 for more performant solution)

We have a medium-sized setup (45k repos, 25k users) and the H2 is still good enough for us.
For much bigger setups (e.g. Eclipse Foundation with over 100k+ users) the option 1) could be better as you suggested.

Luca.

Nandha Kumar Nagarajan

unread,
Mar 9, 2021, 5:10:36 AM3/9/21
to Luca Milanesio, Repo and Gerrit Discussion
Thanks Jacek and Luca. That info is really helpful.
I will check which option will be best suitable for our environment

Meanwhile, team please any thoughts on below query
1) We are not able to find any docs to gain a deeper understanding of how NoteDB works other than below link. Previously when we have postgresql, we had a separate log file to look for errors and Javamelody shows Postgresql connections and metrics

But after we moved to NoteDB completely(Postgresql is disabled), we are kind of not sure how to troubleshoot and monitor(Java melody not showing any info on Notedb) in case of any failures. Any information will be greatly helpful
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/6CEC474F-6479-407F-A28B-9B2ADEAB06BB%40gmail.com.

Luca Milanesio

unread,
Mar 9, 2021, 7:03:26 AM3/9/21
to Nandha Kumar Nagarajan, Luca Milanesio, Repo and Gerrit Discussion

On 9 Mar 2021, at 10:10, Nandha Kumar Nagarajan <nandh...@gmail.com> wrote:

Thanks Jacek and Luca. That info is really helpful.
I will check which option will be best suitable for our environment

Meanwhile, team please any thoughts on below query
1) We are not able to find any docs to gain a deeper understanding of how NoteDB works other than below link. Previously when we have postgresql, we had a separate log file to look for errors and Javamelody shows Postgresql connections and metrics

But after we moved to NoteDB completely(Postgresql is disabled), we are kind of not sure how to troubleshoot and monitor(Java melody not showing any info on Notedb) in case of any failures. Any information will be greatly helpful

Just forget about the “DB” part of the name: it is a set of “notes” inside the Git repository.
There is no external DB to manage or monitor.

HTH

Luca.

Nandha Kumar Nagarajan

unread,
Mar 9, 2021, 12:04:16 PM3/9/21
to Repo and Gerrit Discussion
Thanks for the update Luca

Reg the usage of H2 DB for AccountPatchReviewDb, I have 2 queries
- May I know how H2 data is being synchronized between main and replica servers?
- I guess we can follow the steps provided in below doc for migrating H2 to any external DB in future if needed. This will be performed on master. Do we need to do same in replica servers ? Like installing postgresql DB and migrating data or linking replica postgresql DB to master postgresql DB ?

https://gerrit.com/Documentation/pgm-MigrateAccountPatchReviewDb.html

Luca Milanesio

unread,
Mar 9, 2021, 12:09:09 PM3/9/21
to Nandha Kumar Nagarajan, Luca Milanesio, Repo and Gerrit Discussion

On 9 Mar 2021, at 17:04, Nandha Kumar Nagarajan <nandh...@gmail.com> wrote:

Thanks for the update Luca

Reg the usage of H2 DB for AccountPatchReviewDb, I have 2 queries

Nandha,
I believe we are potentially heading to an infinite level of nesting on this topic called “post migration issues”.
I would like to remind you that this channel is for helping the *whole community* and making the topics focussed help other people benefiting from this discussion.

I will answer this last question here, but *please* keep “one topic = one issue” and not “one topic = multiple discussions”.

- May I know how H2 data is being synchronized between main and replica servers?

Why do you need to synchronise them?
Are you running a multi-server setup?
Replicas are just for Git data and they do not serve the GUI

- I guess we can follow the steps provided in below doc for migrating H2 to any external DB in future if needed. This will be performed on master. Do we need to do same in replica servers ?

See my question above: why would you need them?

Like installing postgresql DB and migrating data or linking replica postgresql DB to master postgresql DB ?

See above.

HTH

Luca.

Reply all
Reply to author
Forward
0 new messages