ssh at port 29418 fails (Permission denied (publickey))

[adrien...@gmail.com]

Hi Guys,

We just enabled a high available version of Gerrit 3.1.8.

As we were doing an initial round of test, we can pull source code and thought that we can also push code.

However, the PUSH part is showing an error on the sshd log as follow:

gerrit - AUTH FAILURE FROM 127.0.0.1 no-matching-key

So when we tried to check the gerrit accounts capability to do ssh (port 29418) at localhost , we found the below error shown below.

What baffles us is that SSH seems to still look for other keys (id_ecdsa , id_ed25519 and id_xmss) even when both id_rsa and id_dsa were already present in the .ssh folder.

Just to another info, when we stood up the new servers with RHEL8. The .ssh folder was copied from the current production  (non-ha RHEL7) server. I believe that it has been the practice here since that when moving Gerrit to a new server with new OS, the .ssh is just being copied in order to preserve integration with Jira and Jenkins).

I just took over this project so I am just looking for help in this issue.

 

===================================================

[gerrit@mrkxxxx]$ ssh -vvvv -p 29418 localhost

OpenSSH_8.0p1, OpenSSL 1.1.1g FIPS  21 Apr 2020

debug1: Reading configuration data /home/gerrit/.ssh/config

debug1: Reading configuration data /etc/ssh/ssh_config

debug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0

debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf

debug2: checking match for 'final all' host localhost originally localhost

debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 3: not matched 'final'

debug2: match not found

debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)

debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config

debug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-]

debug3: kex names ok: [curve25519-sha256,curve255...@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]

debug1: configuration requests final Match pass

debug1: re-parsing configuration

debug1: Reading configuration data /home/gerrit/.ssh/config

debug1: Reading configuration data /etc/ssh/ssh_config

debug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0

debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf

debug2: checking match for 'final all' host localhost originally localhost

debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 3: matched 'final'

debug2: match found

debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1

debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config

debug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-]

debug3: kex names ok: [curve25519-sha256,curve255...@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]

debug2: resolving "localhost" port 29418

debug2: ssh_connect_direct

debug1: Connecting to localhost [127.0.0.1] port 29418.

debug1: Connection established.

debug1: identity file /home/gerrit/.ssh/id_rsa type 0

debug1: identity file /home/gerrit/.ssh/id_rsa-cert type -1

debug1: identity file /home/gerrit/.ssh/id_dsa type 1

debug1: identity file /home/gerrit/.ssh/id_dsa-cert type -1

debug1: identity file /home/gerrit/.ssh/id_ecdsa type -1

debug1: identity file /home/gerrit/.ssh/id_ecdsa-cert type -1

debug1: identity file /home/gerrit/.ssh/id_ed25519 type -1

debug1: identity file /home/gerrit/.ssh/id_ed25519-cert type -1

debug1: identity file /home/gerrit/.ssh/id_xmss type -1

debug1: identity file /home/gerrit/.ssh/id_xmss-cert type -1

debug1: Local version string SSH-2.0-OpenSSH_8.0

debug1: Remote protocol version 2.0, remote software version GerritCodeReview_3.1.8 (APACHE-SSHD-2.3.0)

debug1: no match: GerritCodeReview_3.1.8 (APACHE-SSHD-2.3.0)

debug2: fd 5 setting O_NONBLOCK

debug1: Authenticating to localhost:29418 as 'gerrit'

debug3: put_host_port: [localhost]:29418

debug3: hostkeys_foreach: reading file "/home/gerrit/.ssh/known_hosts"

debug3: record_hostkey: found key type ECDSA in file /home/gerrit/.ssh/known_hosts:1

debug3: load_hostkeys: loaded 1 keys from [localhost]:29418

debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nis...@openssh.com,ecdsa-sha2-nis...@openssh.com,ecdsa-sha2-nis...@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521

debug3: send packet: type 20

debug1: SSH2_MSG_KEXINIT sent

debug3: receive packet: type 20

debug1: SSH2_MSG_KEXINIT received

debug2: local client KEXINIT proposal

debug2: KEX algorithms: curve25519-sha256,curve255...@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c

debug2: host key algorithms: ecdsa-sha2-nis...@openssh.com,ecdsa-sha2-nis...@openssh.com,ecdsa-sha2-nis...@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed2551...@openssh.com,rsa-sha2-5...@openssh.com,rsa-sha2-2...@openssh.com,ssh-rsa-...@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa

debug2: ciphers ctos: aes25...@openssh.com,chacha20...@openssh.com,aes256-ctr,aes256-cbc,aes12...@openssh.com,aes128-ctr,aes128-cbc

debug2: ciphers stoc: aes25...@openssh.com,chacha20...@openssh.com,aes256-ctr,aes256-cbc,aes12...@openssh.com,aes128-ctr,aes128-cbc

debug2: MACs ctos: hmac-sha...@openssh.com,hmac-s...@openssh.com,umac-1...@openssh.com,hmac-sha...@openssh.com,hmac-sha2-256,hmac-sha1,umac...@openssh.com,hmac-sha2-512

debug2: MACs stoc: hmac-sha...@openssh.com,hmac-s...@openssh.com,umac-1...@openssh.com,hmac-sha...@openssh.com,hmac-sha2-256,hmac-sha1,umac...@openssh.com,hmac-sha2-512

debug2: compression ctos: none,zl...@openssh.com,zlib

debug2: compression stoc: none,zl...@openssh.com,zlib

debug2: languages ctos:

debug2: languages stoc:

debug2: first_kex_follows 0

debug2: reserved 0

debug2: peer server KEXINIT proposal

debug2: KEX algorithms: ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group18-sha512,diffie-hellman-group17-sha512,diffie-hellman-group16-sha512,diffie-hellman-group15-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1

debug2: host key algorithms: ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa

debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,aes192-cbc,aes256-cbc

debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,aes192-cbc,aes256-cbc

debug2: MACs ctos: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha2-512,hmac-sha1-96,hmac-md5-96

debug2: MACs stoc: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha2-512,hmac-sha1-96,hmac-md5-96

debug2: compression ctos: none

debug2: compression stoc: none

debug2: languages ctos:

debug2: languages stoc:

debug2: first_kex_follows 0

debug2: reserved 0

debug1: kex: algorithm: ecdh-sha2-nistp256

debug1: kex: host key algorithm: ecdsa-sha2-nistp256

debug1: kex: server->client cipher: aes256-ctr MAC: hmac-sha2-256 compression: none

debug1: kex: client->server cipher: aes256-ctr MAC: hmac-sha2-256 compression: none

debug1: kex: ecdh-sha2-nistp256 need=32 dh_need=32

debug1: kex: ecdh-sha2-nistp256 need=32 dh_need=32

debug3: send packet: type 30

debug1: expecting SSH2_MSG_KEX_ECDH_REPLY

debug3: receive packet: type 31

debug1: Server host key: ecdsa-sha2-nistp256 SHA256:Z34AY+Lusw05nijXL0qdtWi7ycoRTCFdvBANcHm7K/c

debug3: put_host_port: [127.0.0.1]:29418

debug3: put_host_port: [localhost]:29418

debug3: hostkeys_foreach: reading file "/home/gerrit/.ssh/known_hosts"

debug3: record_hostkey: found key type ECDSA in file /home/gerrit/.ssh/known_hosts:1

debug3: load_hostkeys: loaded 1 keys from [localhost]:29418

debug1: Host '[localhost]:29418' is known and matches the ECDSA host key.

debug1: Found key in /home/gerrit/.ssh/known_hosts:1

debug3: send packet: type 21

debug2: set_newkeys: mode 1

debug1: rekey out after 4294967296 blocks

debug1: SSH2_MSG_NEWKEYS sent

debug1: expecting SSH2_MSG_NEWKEYS

debug3: receive packet: type 21

debug1: SSH2_MSG_NEWKEYS received

debug2: set_newkeys: mode 0

debug1: rekey in after 4294967296 blocks

debug1: Skipping ssh-dss key /home/gerrit/.ssh/id_dsa - not in PubkeyAcceptedKeyTypes

debug1: Will attempt key: /home/gerrit/.ssh/id_rsa RSA SHA256:L4mO8iOJ3+xyr3b3GVUNNrOgFVHS0V+huFN8r4P6NGI

debug1: Will attempt key: /home/gerrit/.ssh/id_ecdsa

debug1: Will attempt key: /home/gerrit/.ssh/id_ed25519

debug1: Will attempt key: /home/gerrit/.ssh/id_xmss

debug2: pubkey_prepare: done

debug3: send packet: type 5

debug3: receive packet: type 6

debug2: service_accept: ssh-userauth

debug1: SSH2_MSG_SERVICE_ACCEPT received

debug3: send packet: type 50

debug3: receive packet: type 51

debug1: Authentications that can continue: publickey

debug3: start over, passed a different list publickey

debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password

debug3: authmethod_lookup publickey

debug3: remaining preferred: keyboard-interactive,password

debug3: authmethod_is_enabled publickey

debug1: Next authentication method: publickey

debug1: Offering public key: /home/gerrit/.ssh/id_rsa RSA SHA256:L4mO8iOJ3+xyr3b3GVUNNrOgFVHS0V+huFN8r4P6NGI

debug3: send packet: type 50

debug2: we sent a publickey packet, wait for reply

debug3: receive packet: type 51

debug1: Authentications that can continue: publickey

debug1: Trying private key: /home/gerrit/.ssh/id_ecdsa

debug3: no such identity: /home/gerrit/.ssh/id_ecdsa: No such file or directory

debug1: Trying private key: /home/gerrit/.ssh/id_ed25519

debug3: no such identity: /home/gerrit/.ssh/id_ed25519: No such file or directory

debug1: Trying private key: /home/gerrit/.ssh/id_xmss

debug3: no such identity: /home/gerrit/.ssh/id_xmss: No such file or directory

debug2: we did not send a packet, disable method

debug1: No more authentication methods to try.

gerrit@localhost: Permission denied (publickey).

[Sven Selberg]

You seem to have an rsa and a dsa key in /home/gerrit/.ssh/
Could it be that the only ssh key that is added to "gerrit"''s account in Gerrit is the dsa key that is no longer accepted?
(check with https://gerrit-review.googlesource.com/Documentation/rest-api-accounts.html#list-ssh-keys)
See below: "debug1: Skipping ssh-dss key /home/gerrit/.ssh/id_dsa - not in PubkeyAcceptedKeyTypes"

[adrien...@gmail.com]

Hi Sven,

It is also what I was thinking initially, but I do remember adding the id_rsa. 

Moreover, is the skipping of the ssh-dss key has something to do with OpenSSH_8 default in RHEL8? Because in production we still have RHEL7 which uses OpenSSH_7.

I will try and add the id_rsa once more and check whether the issue persist.

 

One more thing, as I wanted to list the sshkeys for the gerrit account, I tried to list my own account's sshkeys first thru curl (I logged into remote server prior) but I get an unauthorized message below. Could this be related to a missing permission?

   server:~ (username)$curl -k -u username:rest_key -X GET https://server:8443/accounts/username/sshkeys/

   modify account not permitted

Regards,

Adrien

[Sven Selberg]

On Monday, May 31, 2021 at 5:30:02 PM UTC+2 adrien...@gmail.com wrote:

Hi Sven,

It is also what I was thinking initially, but I do remember adding the id_rsa. 

Moreover, is the skipping of the ssh-dss key has something to do with OpenSSH_8 default in RHEL8? Because in production we still have RHEL7 which uses OpenSSH_7.

I'm not to familiar with Red Hat but that seem to me like the most likely explanation. As I remembered it dsa was removed already in OpenSSH_7 but perhaps it was just deprecated. 

I will try and add the id_rsa once more and check whether the issue persist.

 

One more thing, as I wanted to list the sshkeys for the gerrit account, I tried to list my own account's sshkeys first thru curl (I logged into remote server prior) but I get an unauthorized message below. Could this be related to a missing permission?

   server:~ (username)$curl -k -u username:rest_key -X GET https://server:8443/accounts/username/sshkeys/

   modify account not permitted

Don't know exactly why you get an error and I don't see why they would be related, but try:

    $ curl -k -u username:rest_key -X GET https://server:8443/a/accounts/self/sshkeys

[adrien...@gmail.com]

Hi Sven,

The below works for my account and I was able to see my sshkey.

       curl -k -u username:rest_key -X GET https://server:8443/a/accounts/self/sshkeys

While my account is also localized on the server (as an admin), it is based from AD just like any other users who uses it to login to the Codereview UI (browser) and later register the ssh public key.

However, and hope you don't find this uncommon, the so called "gerrit" account is a local account in the unix server and non AD based. So we do not use it to login to Codereview UI (browser). But to register its public key, as a Gerrit admin, we can execute the following admin command thru GitBash.

       cat id_rsa.pub | ssh -p 29418 server gerrit set-account --add-ssh-key - gerrit

And so I realized that I cannot use the curl command to list the sshkeys of the gerrit account since it does not have a rest_key nor the password does not exist in AD.

Circling back to the original reported issue. If we can only find a way to query the sskeys of the gerrit account, we may be able to pinpoint what is registered (cached) so we can try to remove the sshkey and reregister the id_rsa instead of id_dsa.

Hope to hear from you soon.

Regards,

Adrien

[Sven Selberg]

On Tuesday, June 1, 2021 at 8:29:54 PM UTC+2 adrien...@gmail.com wrote:

Hi Sven,

The below works for my account and I was able to see my sshkey.

       curl -k -u username:rest_key -X GET https://server:8443/a/accounts/self/sshkeys

While my account is also localized on the server (as an admin), it is based from AD just like any other users who uses it to login to the Codereview UI (browser) and later register the ssh public key.

However, and hope you don't find this uncommon, the so called "gerrit" account is a local account in the unix server and non AD based. So we do not use it to login to Codereview UI (browser). But to register its public key, as a Gerrit admin, we can execute the following admin command thru GitBash.

       cat id_rsa.pub | ssh -p 29418 server gerrit set-account --add-ssh-key - gerrit

And so I realized that I cannot use the curl command to list the sshkeys of the gerrit account since it does not have a rest_key nor the password does not exist in AD.

Circling back to the original reported issue. If we can only find a way to query the sskeys of the gerrit account, we may be able to pinpoint what is registered (cached) so we can try to remove the sshkey and reregister the id_rsa instead of id_dsa.

As an admin you should be able to list ssh keys for all users:

    curl -k -u username:rest_key -X GET https://server:8443/a/accounts/gerrit/sshkeys

If not you can set the http-password for the "gerrit" user and query for ssh-keys as that user

    ssh -p 29418 server gerrit set-account gerrit  --http-password $SECRET
    curl -k -u gerrit:$SECRET -X GET https://server:8443/a/accounts/self/sshkeys

[adrien...@gmail.com]

Hi Sven,

Thanks for the help. 

Last night I was able to list the ssh keys and found that indeed only the id_dsa.pub is registered for the gerrit user. So I tried your suggested REST command and later used the DELETE REST API to delete the existing sshkey. 

After that, I added the id_rsa.pub using the below admin command (non REST) and then I was able to ssh on both ports 22 and 29418 . I will try today your suggestion to register the http-password for the "gerrit" user.

     cat id_rsa.pub | ssh -p 29418 server gerrit set-account --add-ssh-key - gerrit

We were now able to move over this hurdle.

There's a new one and has something to do with our gerrit hooks that tries to call/interface with Jira. I will investigate further and will check for any historical conversations here that could be of help. 

Thanks and Regards,

Adrien