Re: Gerrit Replication problem - "Cannot replicate .. reject HostKey"

[Shawn Pearce]

On Sun, Jul 1, 2012 at 7:07 AM, Weber <webm...@zazma.com> wrote:

After upgrading Gerrit to 2.4.1, replication stopped working for us.  Searching the logs, this mailing list, and Google, yielded no solution, so we're
asking for your help.

Before the upgrade, replication worked like a charm.

Now, the log shows:

[2012-07-01 13:44:02,772] ERROR com.google.gerrit.server.git.PushReplication : Cannot replicate to g...@git.assembla.com:mainrepo.git
org.eclipse.jgit.errors.TransportException: g...@git.assembla.com:mainrepo.git: reject HostKey: git.assembla.com
        at org.eclipse.jgit.transport.JschConfigSessionFactory.getSession(JschConfigSessionFactory.java:141)
        at org.eclipse.jgit.transport.SshTransport.getSession(SshTransport.java:121)
        at org.eclipse.jgit.transport.TransportGitSsh$SshFetchConnection.<init>(TransportGitSsh.java:248)
        at org.eclipse.jgit.transport.TransportGitSsh.openFetch(TransportGitSsh.java:147)
        at com.google.gerrit.server.git.PushOp.listRemote(PushOp.java:379)
        at ........


The ssh keys or known_hosts seem to not be the problem, as apparently we can ssh to the target machine (when logged-in as gerrit2):

I think Gerrit isn't able to read the ~/.ssh/known_hosts file. Are you running it as the correct user? Does it have the right HOME environment variable? Did the known_hosts file suddenly wind up readable only by root?

[Weber]

Thanks Shawn

I think we're okay on all those points:

$ ps -ef | grep GerritCodeReview
gerrit2   4815   544  0 06:23 pts/0    00:00:00 grep --color=auto GerritCodeReview
gerrit2  32083     1  0 Jul01 ?        00:03:35 GerritCodeReview -jar /home/gerrit2/r/bin/gerrit.war daemon -d /home/gerrit2/r --run-id=1341146209.32062
$ ls -ld ~gerrit2/
drwxr-xr-x 6 gerrit2 gerrit2 4096 Jul  1 13:44 /home/gerrit2/
$ ls -ld ~gerrit2/.ssh
drwx------ 2 gerrit2 gerrit2 4096 Jul  1 12:58 /home/gerrit2/.ssh
$ ls -l ~gerrit2/.ssh
total 16
-rw------- 1 gerrit2 gerrit2 1675 Jul  1 08:18 id_rsa
-rw-r--r-- 1 gerrit2 gerrit2  412 Jul  1 08:19 id_rsa.pub
-rw-r--r-- 1 gerrit2 gerrit2  444 Jul  1 12:58 known_hosts
$ cat /etc/default/gerritcodereview
export GERRIT_SITE=/home/gerrit2/r
$

Any other ideas or pointers on how to debug this?

Thanks

[Shawn Pearce]

On Sun, Jul 1, 2012 at 11:27 PM, Weber <webm...@zazma.com> wrote:

Any other ideas or pointers on how to debug this?

Was there any other relevant portion of the error message that you maybe clipped?

Did the host key change?

[Weber]

No and no, respectively..  but it seems that I found the problem now:   We had also updated the Gerrit host OS, and this caused a subtle change
in the contents of the known_hosts file.   It now used ecdsa-sha2-nistp256 keys, which apparently Jsch is not happy to work with.

After restoring the old known_hosts file from backup (with same keys, but stored as ssh-rsa), replication works again.

2 things worth mentioning:

- I had to restore from backup since I didn't find a way to convince ssh(1) to create the keys with anything other then ecdsa-sha2-nistp256

- This means that following the Gerrit documentation on setting up replication, is now apparently bound to fail on a recent system (well, Ubuntu Server at least), since
using the command "sudo su -c 'ssh mirror1.us.some.org echo' gerrit2" (from here) will create a bad (from Jsch point of view) known_hosts file.


Thanks for the help,

--zw

[David Pursehouse]

On Wednesday, August 8, 2012 9:42:35 PM UTC+9, Marcel Huber wrote:

I just encountered similar problems but was not able to restore an old version of the known_hosts file.

The solution to get gerrit replicating again is quite simple:

# remove any ecdsa entries in the known_hosts file using known hostnames:

$ ssh-keygen -R hostname -f ~/.ssh/known_hosts

# or manually delete offending lines from the file

#  to find the correct line number, you could pass -v to an ssh call and grep for known_hosts:

$ ssh -v gerrit2@hostname 2>&1 | grep known_hosts:

 debug1: Found key in /home/gerrit2/.ssh/known_hosts:4

$ sed -i 'd4' ~/.ssh/known_hosts # if it was line number 4

# explicitly add rsa key from host:

$ ssh-keyscan -t rsa hostname >> ~/.ssh/known_hosts

# optionally you can re-hash the contents of the known_hosts file:

$ ssh-keygen -H -f ~/.ssh/known_hosts

It's worth noting that after fixing up the known_hosts file, you have to restart Gerrit.  It's not enough to simply reload the replication plugin.

[georgey]

Remove ecdsa entries for your remote from ~/.ssh/known_hosts
Add something like this to your ~/.ssh/config
#===
Host remote-alias remote-alias.someplace.com
  Hostname real-remote.someplace.com
  Protocol 2
  HostKeyAlgorithms ssh-rsa,ssh-dss
#===
The re-run the manual connection to the remote and accept the RSA or DSA version of the published remote SSH hostkey