Invalid DER: object is not an OID: SEQUENCE

[Лёша М]

Hello Repo and Gerrit Discussion:

After I upgraded to 2.15 I'm seeing this all over creation:

[sshd-SshServer[67689c81]-nio2-thread-2] WARN  org.apache.sshd.common.keyprovider.FileKeyPairProvider : Failed (StreamCorruptedException) to load key resource=/home/gerrit2/g/etc/ssh_host_ecdsa_key: Invalid DER: object is not an OID: SEQUENCE

[sshd-SshServer[67689c81]-nio2-thread-2] WARN  org.apache.sshd.common.keyprovider.FileKeyPairProvider : Failed (StreamCorruptedException) to load key resource=/home/gerrit2/g/etc/ssh_host_ecdsa_384_key: Invalid DER: object is not an OID: SEQUENCE

[sshd-SshServer[67689c81]-nio2-thread-2] WARN  org.apache.sshd.common.keyprovider.FileKeyPairProvider : Failed (StreamCorruptedException) to load key resource=/home/gerrit2/g/etc/ssh_host_ecdsa_521_key: Invalid DER: object is not an OID: SEQUENCE

[sshd-SshServer[67689c81]-nio2-thread-2] WARN  org.apache.sshd.common.keyprovider.FileKeyPairProvider : Failed (StreamCorruptedException) to load key resource=/home/gerrit2/g/etc/ssh_host_ecdsa_key: Invalid DER: object is not an OID: SEQUENCE

[sshd-SshServer[67689c81]-nio2-thread-2] WARN  org.apache.sshd.common.keyprovider.FileKeyPairProvider : Failed (StreamCorruptedException) to load key resource=/home/gerrit2/g/etc/ssh_host_ecdsa_384_key: Invalid DER: object is not an OID: SEQUENCE

[sshd-SshServer[67689c81]-nio2-thread-2] WARN  org.apache.sshd.common.keyprovider.FileKeyPairProvider : Failed (StreamCorruptedException) to load key resource=/home/gerrit2/g/etc/ssh_host_ecdsa_521_key: Invalid DER: object is not an OID: SEQUENCE

[sshd-SshServer[67689c81]-nio2-thread-2] WARN  org.apache.sshd.common.keyprovider.FileKeyPairProvider : Failed (StreamCorruptedException) to load key resource=/home/gerrit2/g/etc/ssh_host_ecdsa_key: Invalid DER: object is not an OID: SEQUENCE

According to timestamps of those files it looks like gerrit upgrade created them; I did not do anything special about it.

How can I fix it?

Thank you.

[Gert van Dijk]

Gerrit will generate SSH host keys with the use of your host's ssh-keygen application. Then the Apache MINA SSHD Java library in Gerrit will read them as well as the Jsch library in Gerrit. Apparently MINA fails to read them in your case, for three ECDSA host key formats only.

I suspect something went wrong with the creation of the SSH host keys now or in the past. Could you share which exact version of ssh-keygen you are using on your host OS? On Debian-like systems you could perform steps like this:

$ dpkg -S $(which ssh-keygen)  # which package? 
openssh-client: /usr/bin/ssh-keygen

$ apt-cache policy openssh-client  # which version of the package do we have? 
openssh-client:
 Installed: 1:7.2p2-4ubuntu2.4

You could also try the following. Temporarily move the three host keys out of the way as a backup and then run the init step again (java -jar gerrit.war init ...). It should create new SSH host keys, only those of the type you have moved. Please report what you see when you start Gerrit. Same warning message? -> still reproducible with new keys; probably hitting a bug. Warning message disappeared? -> probably your keys were generated the wrong way in the past, your problem is solved now and hard to reproduce now.

What Gerrit version were you running before you upgraded to 2.15? And is the creation timestamp of the SSH host keys the same as your upgrade time? I ask because Gerrit created ECDSA host keys for a while already, before 2.15.

[Gert van Dijk]

Just to be sure I wanted to add that I was referring to Gerrit's SSH host keys, in the Gerrit-site 'etc' dir, NOT your actual SSH daemon's host keys. 

[Лёша М]

I have a backup of my previous install, it was 2.14.0, I do not see ECDSA keys there at all. It only has rsa and dsa keys from 2014 when I generated them (there were a few upgrades to get me to 2.14).

I think it is some version of Redhat this VM is running; yum list openssh returns: 5.3p1-123.el6_9.

I can try generating ECDSA keys with a more modern version of openssh on a different machine. If I stop gerrit, copy over these files into gerrit's etc directory and start gerrit again, that should be OK, right? Any particular version of openssh that would work?

Appreciate your help!

[Gert van Dijk]

That version of OpenSSH indicates you're running RHEL/CentOS 6. Although 5.3 is really old and came without support for ECDSA keys, this feature is backported by RedHat since 5.3p1-95 and it indeed appears to work with openssh-5.3p1-123.el6_9.x86_64 in my Docker container.

Gerrit will actually attempt to generate all the types of keys and if it fails, just ignore it (source). It could be that somehow the generated key is incompatible with Gerrit. As I'm pretty familiar with SSH, I would like to do some more research on that another time.

What Java JRE version are you using on your host, by the way? (It could be a subtle bug there with the cryptographic libraries that Gerrit ships with in combination with your JRE.)
For RHEL/CentOS 6 they provide java-1.8.0-openjdk-headless which I think is best for Gerrit.

You could indeed, as a workaround, try to generate new SSH keys on another host and transfer them to your Gerrit machine in the site directory.

Hope this helps. :)

[David Pursehouse]

On Fri, Apr 13, 2018 at 5:03 PM Gert van Dijk <gert...@gmail.com> wrote:

That version of OpenSSH indicates you're running RHEL/CentOS 6. Although 5.3 is really old and came without support for ECDSA keys, this feature is backported by RedHat since 5.3p1-95 and it indeed appears to work with openssh-5.3p1-123.el6_9.x86_64 in my Docker container.

Gerrit will actually attempt to generate all the types of keys and if it fails, just ignore it (source). It could be that somehow the generated key is incompatible with Gerrit.

I wonder if that's what causes this warning in the log:

[main] WARN  com.google.gerrit.sshd.SshDaemon : Cannot format SSHD host key [EdDSA]: invalid key type

 

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[David Pursehouse]

On Fri, Apr 13, 2018 at 9:09 PM David Pursehouse <david.pu...@gmail.com> wrote:

On Fri, Apr 13, 2018 at 5:03 PM Gert van Dijk <gert...@gmail.com> wrote:

That version of OpenSSH indicates you're running RHEL/CentOS 6. Although 5.3 is really old and came without support for ECDSA keys, this feature is backported by RedHat since 5.3p1-95 and it indeed appears to work with openssh-5.3p1-123.el6_9.x86_64 in my Docker container.

Gerrit will actually attempt to generate all the types of keys and if it fails, just ignore it (source). It could be that somehow the generated key is incompatible with Gerrit.

I wonder if that's what causes this warning in the log:

[main] WARN  com.google.gerrit.sshd.SshDaemon : Cannot format SSHD host key [EdDSA]: invalid key type

Now I've just realised it's the same warning as mentioned (by you) here:

https://bugs.chromium.org/p/gerrit/issues/detail?id=6505

[Fabio Porcedda]

Have you fixed the issue?

I'm asking because I've the same problems.

Thanks

[Fabio Porcedda]

I'm using Gerrit 2.14.8 in a CentOS 6.9 and openssh 5.3p1 123.el6_9.

For solving the issue I've triggered the regeneration of ssh_host_ecsda* files doing the following steps:

- Stopped gerrit

- Remove etc/ssh_host_ecsda* files

- java -jar gerrit.war init -d /opt/gerrit2/

- Start gerrit

Unfortunately the error messages are still present...

Maybe i can try to do it on a Centos 7 and after that copy back the ssh_host_ecsda* files to the Centos 6 server.

[Fabio Porcedda]

I've fixed the issue using keys regenerated on a Centos 7 server.  

[Kamalakar Sharma]

Hi Porcedda,

How did you generated in Centos 7 ?

Are you referring generated keys in Gerrit's SSH host keys, in the Gerrit-site 'etc' dir, or actual SSH daemon's host keys in centos 7