Gerrit Replication Plugin thread-lock issue

[Rikard Almgren]

Hello,

We have a few mirrors, some internal, and some external, and we use the replication plugin to replicate to these.
This works fairly well in most cases, however, when a mirror, usually an external one, experiences connection issues,
or goes down for whatever reason, replication to *all* mirrors stops.

We have separate remote-sections to all our mirrors, with slightly different rules/replication delays, but the pattern we noticed is:

Example:

4 Remote sections, 8 threads per section.

Replication works fine to section 1-2-3, but the mirror in section 4 experiences network instability:

Once the threads in section 4 reaches 8 threads failing replication, they lock replication.
But they don't only lock replication to that section.

They stop replication to all sections, even though the other sections each have their own 8 threads and aren't filled.

Visual example in the style of

gerrit show-queue -w -q

Once it reaches this state, target_01, target_02 and target_03 will lock together with the failing target_04,
and eventually fill up their own queues without attempting to replicate.

Queue: ReplicateTo-target_01
87e071af              12:20:18.277      [278e65f1] push ssh://git@target_01/<repo_7>
c102d988              12:20:20.407      [44611748] push ssh://git@target_01/<repo_8>
------------------------------------------------------------------------------
  2 tasks, 8 worker threads

Queue: ReplicateTo-target_02
87e071af              12:20:18.277      [278e65f1] push ssh://git@target_02/<repo_7>
c102d988              12:20:20.407      [44611748] push ssh://git@target_02/<repo_8>
------------------------------------------------------------------------------
  2 tasks, 8 worker threads

Queue: ReplicateTo-target_03
87e071af              12:20:18.277      [278e65f1] push ssh://git@target_03/<repo_7>
c102d988              12:20:20.407      [44611748] push ssh://git@target_03/<repo_8>
------------------------------------------------------------------------------
  2 tasks, 8 worker threads

Queue: ReplicateTo-target_04
87e071af              12:20:18.277      [278e65f1] push ssh://git@target_04/<repo_9>
c102d981              12:20:20.407      [44611741] push ssh://git@target_04/<repo_1>
c102d982              12:20:20.407      [44611742] push ssh://git@target_04/<repo_2>
c102d983              12:20:20.407      [44611743] push ssh://git@target_04/<repo_3>
c102d984              12:20:20.407      [44611744] push ssh://git@target_04/<repo_4>
c102d985              12:20:20.407      [44611745] push ssh://git@target_04/<repo_5>
c102d986              12:20:20.407      [44611746] push ssh://git@target_04/<repo_6>
c102d987              12:20:20.407      [44611747] push ssh://git@target_04/<repo_7>
c102d988 Waiting      12:20:20.407      [44611748] push ssh://git@target_04/<repo_8>
------------------------------------------------------------------------------
  9 tasks, 8 worker threads

Our configuration:

  target_01:
      url: ssh://git@target_01/srv/gerrit/git/${name}.git
      threads: 8
      mirror: true
      authGroup: Registered Users
      replicateProjectDeletions: true
      replicationDelay: 10
      rescheduleDelay: 5
  target_02:
      url: ssh://git@target_02/srv/gerrit/git/${name}.git
      threads: 8
      mirror: true
      replicateProjectDeletions: true
      replicationDelay: 5
      rescheduleDelay: 5
  target_03:
      url: ssh://git@target_03/srv/gerrit/git/${name}.git
      threads: 8
      mirror: true
      authGroup:
        - blub
        - Registered Users
      replicateProjectDeletions: true
      replicationDelay: 11
      rescheduleDelay: 11
      timeout: 3000
  target_04:
      url: ssh://git@target_04/srv/gerrit/git/${name}.git
      threads: 8
      mirror: true
      authGroup:
        - blub
        - Registered Users
      replicateProjectDeletions: true
      replicationDelay: 20
      rescheduleDelay: 20
      replicationRetry: 2
      timeout: 5000

Is this a known problem? Is there potentially an error in our configuration that causes it?

BR

Rikard Almgren

[Luca Milanesio]

It depends which version of Gerrit you are referring to.

Luca.

BR

Rikard Almgren

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/01fd0b96-cf8b-445e-afd1-89a2848e835c%40googlegroups.com.

[Rikard Almgren]

We are running Gerrit 2.16.12-93-g27ca7c6, Replication Plugin is v2.16.11.1-15-g04bd527

To unsubscribe, email rep...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-d...@googlegroups.com.

[luca.mi...@gmail.com]

Sent from my iPhone

On 16 Apr 2020, at 11:55, Rikard Almgren <rika...@axis.com> wrote:



We are running Gerrit 2.16.12-93-g27ca7c6, Replication Plugin is v2.16.11.1-15-g04bd527

Very strange release numbers: are you running a fork?

Luca

To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/67dfe843-10c9-4054-bae3-9adb86f52277%40googlegroups.com.

[Rikard Almgren]

I accidentally got the version number wrong. It's supposed to be 2.16.12-91-g27ca7c6.
We have a local fork with two commits on top though, yes.

To unsubscribe, email repo-d...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-d...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/67dfe843-10c9-4054-bae3-9adb86f52277%40googlegroups.com.

[Rikard Almgren]

The reason for the odd number is that we run off the latest on stable-2.16 (at the point of deployment), not the pre-built ones.

[Saša Živkov]

I don't think this is a known problem.

Could you please open an issue for that and then, when replication to all remotes hang again due to the issue you described,

create a thread dump and post it in that ticket?

 

BR

Rikard Almgren

--
--

To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

[Rikard Almgren]

I created https://bugs.chromium.org/p/gerrit/issues/detail?id=12596

Fortunately, the issue with a mirror going down is quite rare, so it may take an indeterminate amount of time until the next occasion,

but I will try to keep it in mind and not just plow through in panic.

To unsubscribe, email repo-d...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-d...@googlegroups.com.

[Martin Fick]

This IS a known issue with jgit ssh, at least with older versions, I am not sure that this has been fixed yet? The known issue is that the jgit implementation of ssh, jshell is broken and not very thread working. It will basically serialize your operations. This is the reason many people have switched to git: or https for replication,

-Martin

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[pete neal]

We had a problem with SSH connection deadlocking in Gerrit, particularly when using multiple threads and under load.

for us, we fixed it by adding this to the Gerrit config files.

[sshd]

   rekeyBytesLimit = 0

   rekeyTimeLimit = 0

Might be worth a try.

[Sven Selberg]

On Thursday, April 16, 2020 at 4:19:49 PM UTC+2, MartinFick wrote:

This IS a known issue with jgit ssh, at least with older versions, I am not sure that this has been fixed yet? The known issue is that the jgit implementation of ssh, jshell is broken and not very thread working. It will basically serialize your operations. This is the reason many people have switched to git: or https for replication,

That's really unfortunate. Are we the only ones using SSH?

Initially I thought that setting GIT_SSH should remedy that, but then I remembered that jgit doesn't work when setting GIT_SSH either:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=529463
https://bugs.chromium.org/p/gerrit/issues/detail?id=11043

We saw some amazing performance improvements in replication by setting controlmaster but unfortunately this blew up intermittently in production so we had to abandon it.

-Martin

To unsubscribe, email repo-d...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-d...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/01fd0b96-cf8b-445e-afd1-89a2848e835c%40googlegroups.com.

[David Ostrovsky]

Am Donnerstag, 16. April 2020 16:28:45 UTC+2 schrieb pete neal:

We had a problem with SSH connection deadlocking in Gerrit, particularly when using multiple threads and under load.

for us, we fixed it by adding this to the Gerrit config files.

[sshd]

   rekeyBytesLimit = 0

   rekeyTimeLimit = 0

Probably related this upstream issue: [1].

[1] https://issues.apache.org/jira/projects/SSHD/issues/SSHD-966

[Matthias Sohn]

Since jgit 5.2 there is an alternative ssh implementation in jgit based on mina sshd [1].

In 5.3 we switched the default for EGit to this new implementation [2].

I think this new implementation can be considered stable.

Advantages are that mina sshd is actively developed and maintained

in a real open source setup and we can contribute patches which is

not the case for jsch. The code also looks a lot cleaner than jsch.

Find more details in [3]. Kudos to Thomas Wolf for this implementation.

We may consider to use this in gerrit as well. The jgit master branch

currently uses mina-sshd 2.4.0 while Gerrit master uses version 2.3.0

for its sshd implementation.

[1] https://wiki.eclipse.org/JGit/New_and_Noteworthy/5.2

https://projects.eclipse.org/projects/technology.jgit/releases/5.2.0

[2] https://wiki.eclipse.org/EGit/New_and_Noteworthy/5.3

[3] https://bugs.eclipse.org/bugs/show_bug.cgi?id=520927

-Matthias

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/35B1690E-46C8-4B8E-9C32-129C47555F0B%40codeaurora.org.

[Luca Milanesio]

On 16 Apr 2020, at 16:47, Matthias Sohn <matthi...@gmail.com> wrote:

Since jgit 5.2 there is an alternative ssh implementation in jgit based on mina sshd [1].

In 5.3 we switched the default for EGit to this new implementation [2].

I think this new implementation can be considered stable.

Advantages are that mina sshd is actively developed and maintained

in a real open source setup and we can contribute patches which is

not the case for jsch. The code also looks a lot cleaner than jsch.

Find more details in [3]. Kudos to Thomas Wolf for this implementation.

We may consider to use this in gerrit as well.

+1 to that. Can you add it to the Gerrit roadmap?

Luca.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/CAKSZd3S%3Ds0LVaBJ-UzgMv25GU-FP5shRUEm7-vFPk8vmPne3Tg%40mail.gmail.com.

[Matthias Sohn]

On Thu, Apr 16, 2020 at 9:03 PM Luca Milanesio <luca.mi...@gmail.com> wrote:

On 16 Apr 2020, at 16:47, Matthias Sohn <matthi...@gmail.com> wrote:

Since jgit 5.2 there is an alternative ssh implementation in jgit based on mina sshd [1].

In 5.3 we switched the default for EGit to this new implementation [2].

I think this new implementation can be considered stable.

Advantages are that mina sshd is actively developed and maintained

in a real open source setup and we can contribute patches which is

not the case for jsch. The code also looks a lot cleaner than jsch.

Find more details in [3]. Kudos to Thomas Wolf for this implementation.

We may consider to use this in gerrit as well.

+1 to that. Can you add it to the Gerrit roadmap?

do you think this could be done in 3.2 or rather in 3.3 ?

[Luca Milanesio]

On 16 Apr 2020, at 22:17, Matthias Sohn <matthi...@gmail.com> wrote:

On Thu, Apr 16, 2020 at 9:03 PM Luca Milanesio <luca.mi...@gmail.com> wrote:

On 16 Apr 2020, at 16:47, Matthias Sohn <matthi...@gmail.com> wrote:

Since jgit 5.2 there is an alternative ssh implementation in jgit based on mina sshd [1].

In 5.3 we switched the default for EGit to this new implementation [2].

I think this new implementation can be considered stable.

Advantages are that mina sshd is actively developed and maintained

in a real open source setup and we can contribute patches which is

not the case for jsch. The code also looks a lot cleaner than jsch.

Find more details in [3]. Kudos to Thomas Wolf for this implementation.

We may consider to use this in gerrit as well.

+1 to that. Can you add it to the Gerrit roadmap?

do you think this could be done in 3.2 or rather in 3.3 ?

We haven’t cut the stable-3.2 branch yet, so I would go for v3.2. What do you think?

The current Git/SSH client support in Gerrit is very unstable and flaky anyway, so I don’t believe we are going to lose stability with the upgrade.

Luca.

[Matthias Sohn]

On Thu, Apr 16, 2020 at 11:21 PM Luca Milanesio <luca.mi...@gmail.com> wrote:

On 16 Apr 2020, at 22:17, Matthias Sohn <matthi...@gmail.com> wrote:

On Thu, Apr 16, 2020 at 9:03 PM Luca Milanesio <luca.mi...@gmail.com> wrote:

On 16 Apr 2020, at 16:47, Matthias Sohn <matthi...@gmail.com> wrote:

Since jgit 5.2 there is an alternative ssh implementation in jgit based on mina sshd [1].

In 5.3 we switched the default for EGit to this new implementation [2].

I think this new implementation can be considered stable.

Advantages are that mina sshd is actively developed and maintained

in a real open source setup and we can contribute patches which is

not the case for jsch. The code also looks a lot cleaner than jsch.

Find more details in [3]. Kudos to Thomas Wolf for this implementation.

We may consider to use this in gerrit as well.

+1 to that. Can you add it to the Gerrit roadmap?

do you think this could be done in 3.2 or rather in 3.3 ?

We haven’t cut the stable-3.2 branch yet, so I would go for v3.2. What do you think?

sounds good

 

The current Git/SSH client support in Gerrit is very unstable and flaky anyway, so I don’t believe we are going to lose stability with the upgrade.

https://gerrit-review.googlesource.com/c/homepage/+/263255

[Sven Selberg]

On Thursday, April 16, 2020 at 11:52:39 PM UTC+2, Matthias Sohn wrote:

On Thu, Apr 16, 2020 at 11:21 PM Luca Milanesio <luca.m...@gmail.com> wrote:

On 16 Apr 2020, at 22:17, Matthias Sohn <matthi...@gmail.com> wrote:

On Thu, Apr 16, 2020 at 9:03 PM Luca Milanesio <luca.m...@gmail.com> wrote:

On 16 Apr 2020, at 16:47, Matthias Sohn <matthi...@gmail.com> wrote:

Since jgit 5.2 there is an alternative ssh implementation in jgit based on mina sshd [1].

In 5.3 we switched the default for EGit to this new implementation [2].

I think this new implementation can be considered stable.

Advantages are that mina sshd is actively developed and maintained

in a real open source setup and we can contribute patches which is

not the case for jsch. The code also looks a lot cleaner than jsch.

Find more details in [3]. Kudos to Thomas Wolf for this implementation.

We may consider to use this in gerrit as well.

+1 to that. Can you add it to the Gerrit roadmap?

do you think this could be done in 3.2 or rather in 3.3 ?

We haven’t cut the stable-3.2 branch yet, so I would go for v3.2. What do you think?

sounds good

Am I understanding correctly or am I confused? You are talking about switching to a jgit that uses Mina, or? 

Luca.

 

Luca.

To unsubscribe, email repo-discuss+unsub...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-d...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/01fd0b96-cf8b-445e-afd1-89a2848e835c%40googlegroups.com.

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

-- 
-- 
To unsubscribe, email repo-discuss+unsub...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-d...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/35B1690E-46C8-4B8E-9C32-129C47555F0B%40codeaurora.org.

-- 
-- 
To unsubscribe, email rep...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-d...@googlegroups.com.

[Matthias Sohn]

On Fri, Apr 17, 2020 at 9:31 AM Sven Selberg <sven.s...@axis.com> wrote:

On Thursday, April 16, 2020 at 11:52:39 PM UTC+2, Matthias Sohn wrote:

On Thu, Apr 16, 2020 at 11:21 PM Luca Milanesio <luca.m...@gmail.com> wrote:

On 16 Apr 2020, at 22:17, Matthias Sohn <matthi...@gmail.com> wrote:

On Thu, Apr 16, 2020 at 9:03 PM Luca Milanesio <luca.m...@gmail.com> wrote:

On 16 Apr 2020, at 16:47, Matthias Sohn <matthi...@gmail.com> wrote:

Since jgit 5.2 there is an alternative ssh implementation in jgit based on mina sshd [1].

In 5.3 we switched the default for EGit to this new implementation [2].

I think this new implementation can be considered stable.

Advantages are that mina sshd is actively developed and maintained

in a real open source setup and we can contribute patches which is

not the case for jsch. The code also looks a lot cleaner than jsch.

Find more details in [3]. Kudos to Thomas Wolf for this implementation.

We may consider to use this in gerrit as well.

+1 to that. Can you add it to the Gerrit roadmap?

do you think this could be done in 3.2 or rather in 3.3 ?

We haven’t cut the stable-3.2 branch yet, so I would go for v3.2. What do you think?

sounds good

Am I understanding correctly or am I confused? You are talking about switching to a jgit that uses Mina, or? 

yes, since jgit 5.2 you can choose if you want to use jsch or mina-sshd for ssh client support in jgit.

The default in jgit is still jsch, to select mina-sshd you need to add org.eclipse.jgit.ssh.apache to the dependencies

and set the SshSessionFactory:

import org.eclipse.jgit.transport.SshSessionFactory;

import org.eclipse.jgit.transport.sshd.DefaultProxyDataFactory;

import org.eclipse.jgit.transport.sshd.JGitKeyCache;

import org.eclipse.jgit.transport.sshd.SshdSessionFactory;

SshdSessionFactory f = new SshdSessionFactory(new JGitKeyCache(), new DefaultProxyDataFactory());

SshSessionFactory.setInstance(factory)

There is a pending change https://git.eclipse.org/r/#/c/156153/ which removes that jsch is the default

ssh client and moves the code needed for using jsch to a separate library in order to enable that it can

be removed from the jgit dependencies if mina-sshd is used instead of jsch. This change is almost ready.

It needs to be split since the current change also separates code depending on bouncycastle into a separate

library which should be moved to a separate change.

-Matthias

[David Ostrovsky]

Am Freitag, 17. April 2020 13:36:24 UTC+2 schrieb Matthias Sohn:

Thanks for the pointer. Quoting the comment on this CL:

• add Hex and its tests
• add TeeOutputStream and its tests
• extract jsch implementation
• extract bouncycastle gpg implementation
  

Would it mean that Gerrit would add yet another 3 JGit modules to the build process:

1. jgit.gpg

2. jgit.jsch

3. jgit.mina

And allow to switch between JGit SSH client backends?

[Matthias Sohn]

we could first provide both options (jsch or mina-sshd) so that users can choose

which ssh client they want to use. In a later version I'd remove jsch.

jgit uses sshd-osgi which is a shaded combination of sshd-common and sshd-core to

avoid issues with split packages which bite in an osgi runtime like Eclipse.

In gerrit I think we would need to 

- upgrade to the mina version 2.4 which jgit master is using

- use sshd-osgi instead of sshd-common and sshd-core

- add a dependency to org.eclipse.jgit.ssh.apache

- add sshd-sftp to the dependencies

- eddsa 0.3 is already in Gerrit dependencies

- add a dependency to org.eclipse.jgit.ssh.jsch when https://git.eclipse.org/r/#/c/156153/ is submitted

- set the SshdSessionFactory to enable switching implementations based on some new gerrit config option

- org.eclipse.jgit.gpg.bc would be only required if there is any use case for creating signed commits in gerrit

-Matthias

[Luca Milanesio]

That looks a very safe option.

jgit uses sshd-osgi which is a shaded combination of sshd-common and sshd-core to

avoid issues with split packages which bite in an osgi runtime like Eclipse.

In gerrit I think we would need to 

- upgrade to the mina version 2.4 which jgit master is using

- use sshd-osgi instead of sshd-common and sshd-core

- add a dependency to org.eclipse.jgit.ssh.apache

- add sshd-sftp to the dependencies

- eddsa 0.3 is already in Gerrit dependencies

- add a dependency to org.eclipse.jgit.ssh.jsch when https://git.eclipse.org/r/#/c/156153/ is submitted

- set the SshdSessionFactory to enable switching implementations based on some new gerrit config option

- org.eclipse.jgit.gpg.bc would be only required if there is any use case for creating signed commits in gerrit

Wow, what a list ! Would using the new SSH client backend (Mina) easier to implement?

Luca.

-Matthias

-- 
-- 
To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/CAKSZd3RhuEQFa3eteeO6s3qAFUc00E8k8qhmQNLgCHO_u6%3DXZQ%40mail.gmail.com.

[David Ostrovsky]

With: [1] and with this config option in gerrit.config:

    ssh.clientImplementation=MINA`

it is possible to switch to using MINA ssh client in JGit.

[1] https://gerrit-review.googlesource.com/c/gerrit/+/263692