Replication plugin configuration

[Jigar R]

Hello Gerrit community,

I have setup 2 gerrit docker images to test multi-site plugin. If I create a repository on gerrit_1/gerrit_2 then I see bunch of errors in replication.log stating Invalid privatekey. On the other hand I can run "ssh gerrit_2 gerrit" command successfully. So I am confused on how to fix this issues. Please guide me on how to tackle this.

Here is the configuration of each host:

Gerrit_1

    $ cat replication.config
    [remote "Replication"]
        url = ssh://gerrit_2:29418/var/gerrit/git/${name}.git
        push = +refs/*:refs/*
        timeout = 600
        rescheduleDelay = 15
        replicationDelay = 5
        mirror = true
        createMissingRepositories = true
        replicateProjectDeletions = true
        replicateHiddenProjects = true
    [gerrit]
        autoReload = true
        replicateOnStartup = false
    [replication]
        lockErrorMaxRetries = 5
    maxRetries = 5

    $ ssh gerrit_2 gerrit
    fatal: Available commands of gerrit are:

       apropos              Search in Gerrit documentation
       ban-commit           Ban a commit from a project's repository
       ..........

    See 'gerrit COMMAND --help' for more information.

    $ tail -f /var/gerrit/logs/replication.log
    [2021-01-06 18:16:56,959] Replication to ssh://gerrit_2:29418/var/gerrit/git/jrtest2.git started... [CONTEXT pushOneId="bcd60f96" ]
    [2021-01-06 18:16:56,962] Push to ssh://gerrit_2:29418/var/gerrit/git/jrtest2.git references: RemoteRefUpdate{refSpec=refs/meta/config:refs/meta/config, status=NOT_ATTEMPTED, id=(null)..AnyObjectId[c2833c07c9f10b61cff36da134b96914abc63c9f], force=yes, delete=no, ffwd=no} [CONTEXT pushOneId="bcd60f96" ]
    [2021-01-06 18:16:56,966] Cannot replicate to ssh://gerrit_2:29418/var/gerrit/git/jrtest2.git [CONTEXT pushOneId="bcd60f96" ]
    org.eclipse.jgit.errors.TransportException: ssh://gerrit_2:29418/var/gerrit/git/jrtest2.git: invalid privatekey: [B@ad3f2f0
        at org.eclipse.jgit.transport.JschConfigSessionFactory.getSession(JschConfigSessionFactory.java:162)
        at org.eclipse.jgit.transport.SshTransport.getSession(SshTransport.java:107)
        at org.eclipse.jgit.transport.TransportGitSsh$SshPushConnection.<init>(TransportGitSsh.java:313)
        at org.eclipse.jgit.transport.TransportGitSsh.openPush(TransportGitSsh.java:150)
        at org.eclipse.jgit.transport.PushProcess.execute(PushProcess.java:127)
        at org.eclipse.jgit.transport.Transport.push(Transport.java:1335)
        at org.eclipse.jgit.transport.Transport.push(Transport.java:1381)
        at com.googlesource.gerrit.plugins.replication.PushOne.pushVia(PushOne.java:512)
        at com.googlesource.gerrit.plugins.replication.PushOne.runImpl(PushOne.java:483)
        at com.googlesource.gerrit.plugins.replication.PushOne.doRunPushOperation(PushOne.java:367)
        at com.googlesource.gerrit.plugins.replication.PushOne.runPushOperation(PushOne.java:335)
        at com.googlesource.gerrit.plugins.replication.PushOne.lambda$run$0(PushOne.java:321)
        at com.google.gerrit.server.util.RequestScopePropagator.lambda$cleanup$1(RequestScopePropagator.java:182)
        at com.google.gerrit.server.util.RequestScopePropagator.lambda$context$0(RequestScopePropagator.java:170)
        at com.google.gerrit.server.git.PerThreadRequestScope$Propagator.lambda$scope$0(PerThreadRequestScope.java:70)
        at com.googlesource.gerrit.plugins.replication.PushOne.run(PushOne.java:324)
        at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:87)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:610)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
    Caused by: com.jcraft.jsch.JSchException: invalid privatekey: [B@ad3f2f0
        at com.jcraft.jsch.KeyPair.load(KeyPair.java:664)
        at com.jcraft.jsch.KeyPair.load(KeyPair.java:561)
        at com.jcraft.jsch.IdentityFile.newInstance(IdentityFile.java:40)
        at com.jcraft.jsch.JSch.addIdentity(JSch.java:407)
        at com.jcraft.jsch.JSch.addIdentity(JSch.java:367)
        at org.eclipse.jgit.transport.JschConfigSessionFactory.getJSch(JschConfigSessionFactory.java:383)
        at org.eclipse.jgit.transport.JschConfigSessionFactory.createSession(JschConfigSessionFactory.java:317)
        at org.eclipse.jgit.transport.JschConfigSessionFactory.createSession(JschConfigSessionFactory.java:184)
        at org.eclipse.jgit.transport.JschConfigSessionFactory.getSession(JschConfigSessionFactory.java:108)
        ... 23 more

[Jigar R]

I made following changes and now getting different error:

- created PEM based ssh key by running "ssh-keygen -m PEM -t rsa -C"

- added ssh key to gerit user

- updated replication.config as following:

$ cat /var/gerrit/etc/replication.config
[remote "Replication"]
    url = gerrit_2:/var/gerrit/git/${name}.git

    push = +refs/*:refs/*
    timeout = 600
    rescheduleDelay = 15
    replicationDelay = 5
    mirror = true
    createMissingRepositories = true
    replicateProjectDeletions = true
    replicateHiddenProjects = true
[gerrit]
    autoReload = true
    replicateOnStartup = false
[replication]
    lockErrorMaxRetries = 5
    maxRetries = 5

$ cat /var/gerrit/.ssh/config
Host gerrit_2
  Port 29418
  PreferredAuthentications publickey

  IdentityFile /var/gerrit/.ssh/id_rsa

replication log :

[2021-01-06 20:02:43,935] Replication to gerrit_2:/var/gerrit/git/test1.git started... [CONTEXT pushOneId="13d3090c" ]
[2021-01-06 20:02:43,940] Push to gerrit_2:/var/gerrit/git/test1.git references: RemoteRefUpdate{refSpec=refs/meta/config:refs/meta/config, status=NOT_ATTEMPTED, id=(null)..AnyObjectId[459d876b89ad80c118f410f778d52a431093fd18], force=yes, delete=no, ffwd=no} [CONTEXT pushOneId="13d3090c" ]
[2021-01-06 20:02:44,383] Created remote repository: gerrit_2:/var/gerrit/git/test1.git [CONTEXT pushOneId="13d3090c" ]
[2021-01-06 20:02:44,385] Missing repository created; retry replication to gerrit_2:/var/gerrit/git/test1.git [CONTEXT pushOneId="13d3090c" ]

[2021-01-06 20:03:44,390] Replication to gerrit_2:/var/gerrit/git/test1.git started... [CONTEXT pushOneId="13d3090c" ]
[2021-01-06 20:03:44,392] Push to gerrit_2:/var/gerrit/git/test1.git references: RemoteRefUpdate{refSpec=refs/meta/config:refs/meta/config, status=NOT_ATTEMPTED, id=(null)..AnyObjectId[459d876b89ad80c118f410f778d52a431093fd18], force=yes, delete=no, ffwd=no} [CONTEXT pushOneId="13d3090c" ]
[2021-01-06 20:03:45,038] Created remote repository: gerrit_2:/var/gerrit/git/test1.git [CONTEXT pushOneId="13d3090c" ]
[2021-01-06 20:03:45,039] Missing repository created; retry replication to gerrit_2:/var/gerrit/git/test1.git [CONTEXT pushOneId="13d3090c" ]

[2021-01-06 20:04:45,046] Replication to gerrit_2:/var/gerrit/git/test1.git started... [CONTEXT pushOneId="13d3090c" ]
[2021-01-06 20:04:45,050] Push to gerrit_2:/var/gerrit/git/test1.git references: RemoteRefUpdate{refSpec=refs/meta/config:refs/meta/config, status=NOT_ATTEMPTED, id=(null)..AnyObjectId[459d876b89ad80c118f410f778d52a431093fd18], force=yes, delete=no, ffwd=no} [CONTEXT pushOneId="13d3090c" ]
[2021-01-06 20:04:45,467] Created remote repository: gerrit_2:/var/gerrit/git/test1.git [CONTEXT pushOneId="13d3090c" ]
[2021-01-06 20:04:45,468] Missing repository created; retry replication to gerrit_2:/var/gerrit/git/test1.git [CONTEXT pushOneId="13d3090c" ]

[Luca Milanesio]

On 6 Jan 2021, at 20:05, Jigar R <jigarra...@gmail.com> wrote:

I made following changes and now getting different error:

- created PEM based ssh key by running "ssh-keygen -m PEM -t rsa -C"

- added ssh key to gerit user

Yeah, that is what it was missing.

Where is the error you are having?

Luca.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/89866213-a391-49e4-8099-ef6bb4c7e5b6n%40googlegroups.com.

[Jigar R]

Here, gerrit_1 is trying to replicate test1.git on gerrit_2. After, gerrit_1 reaches max retry limit then it cancels the operation.

[Luca Milanesio]

Have you tried to SSH manually to the other box and run a ‘git init --bare’ of a new repository?
That is what Gerrit for creating a new repository.

HTH

Luca.

> To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/8a77cfc5-f517-474f-8e39-f71e83fd0a47n%40googlegroups.com.

[Jigar R]

If I am on gerrit_1 then I create repository on gerrit_2 by using "ssh gerrit_2 gerrit create-project test".  Does replication plugin work on port 29418 or 22?

If I run "ssh gerrit_2" then it gives me following output:

$ ssh gerrit_2
The authenticity of host '[gerrit_2]:29418 ([192.168.96.7]:29418)' can't be established.
ECDSA key fingerprint is SHA256:R+W8G9JrIiR0Lzg6GPcTN/0ooOG4/06vPIchTLYPMEM.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[gerrit_2]:29418,[192.168.96.7]:29418' (ECDSA) to the list of known hosts.

  ****    Welcome to Gerrit Code Review    ****

  Hi gerritadmin, you have successfully connected over SSH.

  Unfortunately, interactive shells are disabled.
  To clone a hosted Git repository, use:

  git clone ssh://gerritadmin@localhost:29418/REPOSITORY_NAME.git

Connection to gerrit_2 closed by remote host.
Connection to gerrit_2 closed.

[Luca Milanesio]

No, that’s not the way the replication plugin works.
The replication plugin will run a remote SSH session (a proper one on port 22) and will issue a command.

> If I run "ssh gerrit_2" then it gives me following output:
> $ ssh gerrit_2
> The authenticity of host '[gerrit_2]:29418 ([192.168.96.7]:29418)' can't be established.
> ECDSA key fingerprint is SHA256:R+W8G9JrIiR0Lzg6GPcTN/0ooOG4/06vPIchTLYPMEM.
> Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
> Warning: Permanently added '[gerrit_2]:29418,[192.168.96.7]:29418' (ECDSA) to the list of known hosts.
>
> **** Welcome to Gerrit Code Review ****
>
> Hi gerritadmin, you have successfully connected over SSH.
>
> Unfortunately, interactive shells are disabled.
> To clone a hosted Git repository, use:
>
> git clone ssh://gerritadmin@localhost:29418/REPOSITORY_NAME.git
>
> Connection to gerrit_2 closed by remote host.
> Connection to gerrit_2 closed.

That’s not what the replication plugin does.
Try again by SSH to port 22 and see if you can create the repository under $GERRIT_SITE/git

HTH

Luca.

> To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/0d803d85-6b43-48c9-b45c-1c61c8aa503an%40googlegroups.com.

[jiga...@gmail.com]

I tried to setup docker image with gerrit and opensshd. After multiple attempts, I gave up on it. I ended up setting up 2 gerrit instances on AWS. I needed event-brokers.jar to make multi-site work ( https://repo1.maven.org/maven2/com/gerritforge/events-broker/ ). I assume this is the source code of events-broker (https://gerrit.googlesource.com/modules/events-broker/). Am I correct?

[Luca Milanesio]

This topic was about replication plugin configuration: are you OK with the answers we gave?

If you would like to get some help on the events-broker, it would be best if you could open a separate topic, so that it would be easier for other people to follow the discussion.

Thanks.

Luca.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/1506ee03-4854-431d-8284-f3ea0f70b017n%40googlegroups.com.

[jiga...@gmail.com]

It was 100% helpful. I would create another thread. Thanks Luca.