ssh_host_key

[Phil Longstaff]

Recently, I upgraded from gerrit 3.2.6 to gerrit 3.4.0.  I was able to access the UI, but when our developers tried to use "git fetch" to update their local repo, they got a report that the remote host identification had failed. It was now an ECDSA key, whereas previously we had an RSA key. I noticed that "etc/ssh_host_key" had a date matching the date of installation.

I wasn't able to find anything in either the 3.3 or 3.4 release notes about this. Is this expected?

Thanks,

Phil

[Luca Milanesio]

On 25 May 2021, at 22:51, 'Phil Longstaff' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:

Recently, I upgraded from gerrit 3.2.6 to gerrit 3.4.0.  I was able to access the UI, but when our developers tried to use "git fetch" to update their local repo, they got a report that the remote host identification had failed. It was now an ECDSA key, whereas previously we had an RSA key.

I have run the Docker image of Gerrit v3.2.6 and this is the list of keys generated:

-rw------- 1 gerrit gerrit  634 May 25 21:56 ssh_host_ecdsa_384_key

-rw-r--r-- 1 gerrit gerrit  237 May 25 21:56 ssh_host_ecdsa_384_key.pub

-rw------- 1 gerrit gerrit  756 May 25 21:56 ssh_host_ecdsa_521_key

-rw-r--r-- 1 gerrit gerrit  285 May 25 21:56 ssh_host_ecdsa_521_key.pub

-rw------- 1 gerrit gerrit  525 May 25 21:56 ssh_host_ecdsa_key

-rw-r--r-- 1 gerrit gerrit  193 May 25 21:56 ssh_host_ecdsa_key.pub

-rw------- 1 gerrit gerrit  432 May 25 21:56 ssh_host_ed25519_key

-rw-r--r-- 1 gerrit gerrit  113 May 25 21:56 ssh_host_ed25519_key.pub

-rw------- 1 gerrit gerrit 2622 May 25 21:56 ssh_host_rsa_key

-rw-r--r-- 1 gerrit gerrit  585 May 25 21:56 ssh_host_rsa_key.pub

As you can see, there is the ECDSA key in there.

I noticed that "etc/ssh_host_key" had a date matching the date of installation.

For some reasons your keys were re-created in your case.

Have you kept the logs of the upgrades?

I wasn't able to find anything in either the 3.3 or 3.4 release notes about this. Is this expected?

Not directly expected, that why it wasn’t mentioned.

Luca.

Thanks,

Phil

CONFIDENTIALITY NOTICE: This email and any attachments are for the sole use of the intended recipient and may contain material that is proprietary, confidential, privileged or otherwise legally protected or restricted under applicable government laws. Any review, disclosure, distributing or other use without expressed permission of the sender is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies without reading, printing, or saving. 

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/194ffb4c-d45c-4fc3-92f1-52ff114439ffn%40googlegroups.com.

[Peter Bruin]

I have the same issue, is there a way to prevent this?

This is the log of the server starting up, nothing special

```

Running Gerrit ...

Running Gerrit Code Review:

WARNING: An illegal reflective access operation has occurred

WARNING: Illegal reflective access by com.google.gerrit.common.IoUtil (file:/var/gerrit/tmp/gerrit_6153808534693658679_app/com_google_gerrit_common_libserver.jar) to method java.net.URLClassLoader.addURL(java.net.URL)

WARNING: Please consider reporting this to the maintainers of com.google.gerrit.common.IoUtil

WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations

WARNING: All illegal access operations will be denied in a future release

May 26, 2021 11:28:59 AM com.google.inject.assistedinject.FactoryProvider2 isValidForOptimizedAssistedInject

WARNING: AssistedInject factory com.google.gerrit.server.api.changes.ChangeApiImpl$Factory will be slow because class com.google.gerrit.server.api.changes.ChangeApiImpl has assisted Provider dependencies or injects the Injector. Stop injecting @Assisted Provider<T> (instead use @Assisted T) or Injector to speed things up. (It will be a ~6500% speed bump!)  The exact offending deps are: [Key[type=com.google.inject.Injector, annotation=[none]]@com.google.gerrit.server.api.changes.ChangeApiImpl.<init>()[52]]

[2021-05-26 11:29:02,312] [main] INFO  com.google.gerrit.server.cache.PersistentCacheBaseFactory : Enabling disk cache /var/gerrit/cache

[2021-05-26 11:29:02,775] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'WorkQueue' queue

[2021-05-26 11:29:02,836] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'Index-Interactive' queue

[2021-05-26 11:29:02,838] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'Index-Batch' queue

[2021-05-26 11:29:02,903] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'ReceiveCommits' queue

[2021-05-26 11:29:02,905] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'SendEmail' queue

[2021-05-26 11:29:06,282] [JGit-FileStoreAttributeReader-1] WARN  org.eclipse.jgit.util.FS : Thread[JGit-FileStoreAttributeReader-1,5,main]: got smaller file timestamp on / (none), /var/gerrit/.config/jgit: 2021-05-26T03:29:06Z < 2021-05-26T03:29:06.213614Z. Aborting measurement at resolution PT0.786386S.

[2021-05-26 11:29:07,143] [main] INFO  com.google.gerrit.server.rules.PrologEnvironment : reductionLimit: 100000, compileLimit: 1000000

[2021-05-26 11:29:07,168] [main] INFO  com.google.gerrit.server.config.ScheduleConfig : No schedule configuration for "changeCleanup".

May 26, 2021 11:29:07 AM com.google.inject.assistedinject.FactoryProvider2 isValidForOptimizedAssistedInject

WARNING: AssistedInject factory com.google.gerrit.sshd.DispatchCommand$Factory will be slow because class com.google.gerrit.sshd.DispatchCommand has assisted Provider dependencies or injects the Injector. Stop injecting @Assisted Provider<T> (instead use @Assisted T) or Injector to speed things up. (It will be a ~6500% speed bump!)  The exact offending deps are: [Key[type=com.google.inject.Injector, annotation=[none]]@com.google.gerrit.sshd.BaseCommand.injector]

[2021-05-26T11:29:08.060+08:00] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'SshCommandStart' queue

[2021-05-26T11:29:08.672+08:00] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'SSH-Stream-Worker' queue

[2021-05-26T11:29:08.676+08:00] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'SSH-Interactive-Worker' queue

[2021-05-26T11:29:08.678+08:00] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'SSH-Batch-Worker' queue

[2021-05-26T11:29:08.705+08:00] [main] WARN  com.google.gerrit.server.config.GitwebCgiConfig : gitweb not installed (no /usr/lib/cgi-bin/gitweb.cgi found)

[2021-05-26T11:29:10.157+08:00] [main] INFO  org.eclipse.jetty.util.log : Logging initialized @16744ms to org.eclipse.jetty.util.log.Slf4jLog

[2021-05-26T11:29:10.340+08:00] [main] INFO  com.google.gerrit.server.git.SystemReaderInstaller : Set JGit's SystemReader to read system config from /var/gerrit/etc/jgit.config

[2021-05-26T11:29:10.346+08:00] [main] INFO  com.google.gerrit.server.git.LocalDiskRepositoryManager : Defaulting core.streamFileThreshold to 768m

[2021-05-26T11:29:11.711+08:00] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loading plugins from /var/gerrit/plugins

May 26, 2021 11:29:12 AM com.google.inject.servlet.GuiceFilter setPipeline

WARNING: Multiple Servlet injectors detected. This is a warning indicating that you have more than one GuiceFilter running in your web application. If this is deliberate, you may safely ignore this message. If this is NOT deliberate however, your application may not work as expected.

[2021-05-26T11:29:12.148+08:00] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin avatars-gravatar, version 4d45f66597

[2021-05-26T11:29:12.277+08:00] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin codemirror-editor, version v3.4.0

May 26, 2021 11:29:12 AM com.google.inject.servlet.GuiceFilter setPipeline

WARNING: Multiple Servlet injectors detected. This is a warning indicating that you have more than one GuiceFilter running in your web application. If this is deliberate, you may safely ignore this message. If this is NOT deliberate however, your application may not work as expected.

[2021-05-26T11:29:12.348+08:00] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin commit-message-length-validator, version v3.4.0

May 26, 2021 11:29:12 AM com.google.inject.servlet.GuiceFilter setPipeline

WARNING: Multiple Servlet injectors detected. This is a warning indicating that you have more than one GuiceFilter running in your web application. If this is deliberate, you may safely ignore this message. If this is NOT deliberate however, your application may not work as expected.

[2021-05-26T11:29:12.491+08:00] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin delete-project, version v3.4.0

[2021-05-26T11:29:12.573+08:00] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin download-commands, version v3.4.0

May 26, 2021 11:29:12 AM com.google.inject.servlet.GuiceFilter setPipeline

WARNING: Multiple Servlet injectors detected. This is a warning indicating that you have more than one GuiceFilter running in your web application. If this is deliberate, you may safely ignore this message. If this is NOT deliberate however, your application may not work as expected.

[2021-05-26T11:29:12.774+08:00] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin gitiles, version v3.4.0

[2021-05-26T11:29:12.893+08:00] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin hooks, version v3.4.0

May 26, 2021 11:29:13 AM com.google.inject.servlet.GuiceFilter setPipeline

WARNING: Multiple Servlet injectors detected. This is a warning indicating that you have more than one GuiceFilter running in your web application. If this is deliberate, you may safely ignore this message. If this is NOT deliberate however, your application may not work as expected.

[2021-05-26T11:29:13.100+08:00] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin lfs, version 0b47095518

May 26, 2021 11:29:13 AM com.google.inject.servlet.GuiceFilter setPipeline

WARNING: Multiple Servlet injectors detected. This is a warning indicating that you have more than one GuiceFilter running in your web application. If this is deliberate, you may safely ignore this message. If this is NOT deliberate however, your application may not work as expected.

[2021-05-26T11:29:13.192+08:00] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin plugin-manager, version v3.4.0

[2021-05-26T11:29:13.192+08:00] [plugin-manager-preloader] INFO  com.googlesource.gerrit.plugins.manager.OnStartStop : Start-up: pre-loading list of plugins from registry

May 26, 2021 11:29:13 AM com.google.inject.assistedinject.FactoryProvider2 isValidForOptimizedAssistedInject

WARNING: AssistedInject factory com.googlesource.gerrit.plugins.replication.Destination$Factory will be slow because class com.googlesource.gerrit.plugins.replication.Destination has assisted Provider dependencies or injects the Injector. Stop injecting @Assisted Provider<T> (instead use @Assisted T) or Injector to speed things up. (It will be a ~6500% speed bump!)  The exact offending deps are: [Key[type=com.google.inject.Injector, annotation=[none]]@com.googlesource.gerrit.plugins.replication.Destination.<init>()[0]]

[2021-05-26T11:29:13.491+08:00] [WorkQueue-2[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@430ffea2[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@49f795d2[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@4730729c]]]] INFO  com.googlesource.gerrit.plugins.deleteproject.fs.RepositoryCleanupTask : Cleaning up expired git repositories...

[2021-05-26T11:29:13.498+08:00] [WorkQueue-2[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@430ffea2[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@49f795d2[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@4730729c]]]] INFO  com.googlesource.gerrit.plugins.deleteproject.fs.RepositoryCleanupTask : Cleaning up expired git repositories... Done

[2021-05-26T11:29:13.587+08:00] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin replication, version v3.4.0

[2021-05-26T11:29:13.658+08:00] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin reviewnotes, version v3.4.0

[2021-05-26T11:29:13.717+08:00] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin singleusergroup, version v3.4.0

[2021-05-26T11:29:13.894+08:00] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin uploadvalidator, version v3.0.0-rc1-284-gefecf8cdf4

[2021-05-26T11:29:14.024+08:00] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin webhooks, version v3.4.0

[2021-05-26T11:29:14.027+08:00] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin rus_logo, version

[2021-05-26T11:29:14.032+08:00] [Reindex changes v60-v61] INFO  com.google.gerrit.server.index.OnlineReindexer : Starting online reindex of changes from schema version 60 to 61

[2021-05-26T11:29:14.035+08:00] [main] INFO  com.google.gerrit.server.config.ScheduleConfig : No schedule configuration for "accountDeactivation".

Collecting projects:    1[2021-05-26T11:29:14.127+08:00] [main] INFO  com.google.gerrit.sshd.SshDaemon : Started Gerrit APACHE-SSHD-2.6.0 on *:29418

[2021-05-26T11:29:14.132+08:00] [main] INFO  org.eclipse.jetty.server.Server : jetty-9.4.36.v20210114; built: 2021-01-14T16:44:28.689Z; git: 238ec6997c7806b055319a6d11f8ae7564adc0de; jvm 11.0.11+9-Ubuntu-0ubuntu2.20.04

[2021-05-26T11:29:14.202+08:00] [main] INFO  org.eclipse.jetty.server.session : DefaultSessionIdManager workerName=node0

[2021-05-26T11:29:14.202+08:00] [main] INFO  org.eclipse.jetty.server.session : No SessionScavenger set, using defaults

[2021-05-26T11:29:14.205+08:00] [main] INFO  org.eclipse.jetty.server.session : node0 Scavenging every 600000ms

Collecting projects:    44

[2021-05-26T11:29:16.609+08:00] [main] INFO  org.eclipse.jetty.server.handler.ContextHandler : Started o.e.j.s.ServletContextHandler@3f2cc8ab{/r,null,AVAILABLE}

[2021-05-26T11:29:16.628+08:00] [main] INFO  org.eclipse.jetty.server.AbstractConnector : Started ServerConnector@20425fab{HTTP/1.1, (http/1.1)}{0.0.0.0:8081}

[2021-05-26T11:29:16.633+08:00] [main] INFO  org.eclipse.jetty.server.Server : Started @23220ms

[2021-05-26T11:29:16.636+08:00] [main] INFO  com.google.gerrit.pgm.Daemon : Gerrit Code Review 3.4.0 ready

```

[Peter Bruin]

BTW, I upgraded from 3.3.4 to 3.4.0

[Peter Bruin]

One more update.

The old key was of type  ssh-rsa, the new key is ecdsa-sha2-nistp521

using ssh-keyscan -t rsa does not give any results

[Peter Bruin]

Hi Luca,

The keys that you see in your docker image look like normal ssh keys, but the ssh_host_key in my system can't be read, it looks like a java serialized object. See below the first line of my ssh_host_key file

java.security.KeyPair privateKeyt Ljava/security/PrivateKey;  publicKey java/security/PublicKey;xpsr =org.bouncycastle.jcajce.provider.asymmetric.ec.BCECPrivateKey

Is there a way to extract the key from this file and convert it to a PEM key, such that when I upgrade again from 3.3.4 to 3.4.0 I don't have to send out an email asking my users to remove the old host key from their known_hosts file?

Or what would you advice moving forward? Generate a new set off host keys?

Best regards,

Peter

On Wednesday, 26 May 2021 at 06:03:48 UTC+8 lucamilanesio wrote:

[Luca Milanesio]

On 11 Jun 2021, at 03:32, Peter Bruin <peterb...@gmail.com> wrote:

Hi Luca,

The keys that you see in your docker image look like normal ssh keys, but the ssh_host_key in my system can't be read, it looks like a java serialized object. See below the first line of my ssh_host_key file

java.security.KeyPair privateKeyt Ljava/security/PrivateKey;  publicKey java/security/PublicKey;xpsr =org.bouncycastle.jcajce.provider.asymmetric.ec.BCECPrivateKey

Is there a way to extract the key from this file and convert it to a PEM key, such that when I upgrade again from 3.3.4 to 3.4.0 I don't have to send out an email asking my users to remove the old host key from their known_hosts file?

We did not have that problem after we migrated to v3.4.0 and our keys were not re-created.

I believe it would be useful to understand why your v3.3.4 keys were in in a non-PEM format: that would be the culprit of the problem.

Or what would you advice moving forward? Generate a new set off host keys?

Nope, my advise for you is to understand the root cause of the problem, how the old key was generated and when.

Do you have the date/time of the ssh_host_key file?

Luca.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/8cdeded8-5327-43c2-b6b0-7303666581c4n%40googlegroups.com.

[Peter Bruin]

Hi Luca,

I have done a bit of digging. Sadly I was not able to recover the original ssh_host_key file but the gerrit instance was created around May 2015.

I tried to get the oldest docker image from dockerhub and that was 2.14.4.

When I used that image to create a new server I got all the ssh keys in PEM format.

However, if I use that image and overwrite the gerrit.war with an older version, 2.10.1 dated 2015-05, I see the below line in the log:

     gerrit_1  | Generating SSH host key ... rsa(simple)... done

And after this the ssh_host_key found in the etc directory is indeed the same format I mentioned before.

To me it seems that this format key was supported up until 3.3.4 but when upgrading to 3.4.0 there is an issue. It looks like the file can contain just one host key, and when it doesn't meet the criteria it is replaced by a new one, rsa->ecdsa, hence the problem encounter by myself and the thread starter. Also, upon downgrading the key was replaced again and changed from ecdsa to rsa.

Maybe this should be added to release notes and even better if there could be an automatic upgrade path for sites that have been initialized using the version of gerrit before 2.14.

---

I tested some more version and from 2.10 to 2.13 would result in the rsa(simple) key. 2.14.1 would result in the below line and PEM encoded keys

    gerrit_1  | Generating SSH host key ... rsa... dsa... ed25519... ecdsa 256... ecdsa 384... ecdsa 521... done

Best regards,

Peter

[David Ostrovsky]

peterb...@gmail.com schrieb am Sonntag, 13. Juni 2021 um 08:39:27 UTC+2:

Hi Luca,

I have done a bit of digging. Sadly I was not able to recover the original ssh_host_key file but the gerrit instance was created around May 2015.

I tried to get the oldest docker image from dockerhub and that was 2.14.4.

When I used that image to create a new server I got all the ssh keys in PEM format.

However, if I use that image and overwrite the gerrit.war with an older version, 2.10.1 dated 2015-05, I see the below line in the log:

     gerrit_1  | Generating SSH host key ... rsa(simple)... done

And after this the ssh_host_key found in the etc directory is indeed the same format I mentioned before.

To me it seems that this format key was supported up until 3.3.4 but when upgrading to 3.4.0 there is an issue. It looks like the file can contain just one host key, and when it doesn't meet the criteria it is replaced by a new one, rsa->ecdsa, hence the problem encounter by myself and the thread starter. Also, upon downgrading the key was replaced again and changed from ecdsa to rsa.

There are different formats supported by gerrit:

Legacy:

  $ file ssh_host_key 

  ssh_host_key: Java serialization data, version 5

Last release that used this format is 2.13.x. Old format creation was removed in this CL: [1].

PEM:

Started from 2.14.x PEM formats are used.

However, if you upgrade from 2.13.14 up to 3.4.0 step by step, ssh_host_key should be preserved.

I did the following test:

1. Create test site using 2.13.14 => ssh_host_key is created 

2. Create new test site using 3.4.0 and replace all host keys in etc directory with ssh_host_key, from the previous step

3. It seems that the old format can be still used in 3.4.0 release

Also the code is still in place: [2].

So that the question is: how you managed to replace ssh_host_key

with host keys in PEM format during the upgrade process?

This host files were created for me if I initialize gerrit 3.4.0 site from scratch:

ssh_host_ecdsa_384_key

ssh_host_ecdsa_384_key.pub

ssh_host_ecdsa_521_key

ssh_host_ecdsa_521_key.pub

ssh_host_ecdsa_key

ssh_host_ecdsa_key.pub

ssh_host_ed25519_key

ssh_host_ed25519_key.pub

ssh_host_rsa_key

ssh_host_rsa_key.pub

[1] https://gerrit-review.googlesource.com/c/gerrit/+/106070

[2] https://gerrit.googlesource.com/gerrit/+/refs/heads/master/java/com/google/gerrit/sshd/HostKeyProvider.java#63 

[Makson Lee]

we have same issue after migrated from 3.3 to 3.4.

[Luca Milanesio]

On 15 Jun 2021, at 00:48, Makson Lee <cdle...@gmail.com> wrote:

we have same issue after migrated from 3.3 to 3.4.

Could this be related to the Apache Mina SSHD upgrade?

Luca.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/92f81436-b567-413b-8abe-a046489380b5n%40googlegroups.com.

[Makson Lee]

how to check it? we didn't change any gerrit configuration manually when migrated from 3.3 to 3.4.

[Sven Selberg]

On Tuesday, June 15, 2021 at 2:12:59 AM UTC+2 cdle...@gmail.com wrote:

how to check it? we didn't change any gerrit configuration manually when migrated from 3.3 to 3.4.

On Tuesday, June 15, 2021 at 7:56:53 AM UTC+8 lucamilanesio wrote:

On 15 Jun 2021, at 00:48, Makson Lee <cdle...@gmail.com> wrote:

we have same issue after migrated from 3.3 to 3.4.

Could this be related to the Apache Mina SSHD upgrade?

Upgraded 3.3.2 -> 3.4.0 on a staging environment on June 3:rd.

We didn't get any host-key warnings but it seems like the upgrade created an extra pub key.

"ssh_host_rsa_key.pub.pub" doesn't match ssh_host_rsa_key whereas "ssh_host_rsa_key.pub" does.

gerrit $ ls -l etc/

...

-rw-------  1 gerrit  gerrit  1675 May 25  2018 ssh_host_rsa_key

-rw-r--r--  1 gerrit  gerrit   423 Jun  3  2020 ssh_host_rsa_key.pub

-rw-r--r--  1 gerrit  gerrit   423 Jun  3  2020 ssh_host_rsa_key.pub.pub

[Peter Bruin]

Hi David,

I followed the upgrade instructions and did not perform any init, seemed like this was not required as there is no data base schema upgrade. Indexing will be done online.

As you can see from the logs the ssh host key changes from rsa to ecdsa after the upgrade, the ssh_host_key file is changed and the size is smaller.

Downgrading will change the host key back to rsa, and the file size is 2199 again but the key has changed.

Before upgrade. ssh_host_key file size 2199

$ ssh-keyscan -p 29418 optimus

# optimus:29418 SSH-2.0-GerritCodeReview_3.3.4 (APACHE-SSHD-2.4.0)

[optimus]:29418 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4wVRX6cDvUzV3suZyEM7RPVLaBUOlVtu5HlD+TRHz5MjiromjQp1LMqmj1OZOiU8QMVSDUDLdl+J935wr0W4Ajgp4h5d8UtpdhiKG6Z9Fr3jlO2q66E7GjaOT/rPAfATRzbGRw2BCGTOSSHVxzwjuFEuC4mJ1XjnIsx+FGb1OLbW2/wUzkl9a+ur08ZE4koSClXIzQ/cFfqg1mSiq5EyeE31cu6BberJ648uU8JHFFYVmk5XRYWgf1b7dF+5d2wP/sJEm1/TtCkDVXWyCrhFzj4mvzV0u/iwqWnGK+t2qGoKxGih+82QEWaMhkpa47VMmqQcq6zg4SjlpB76f+/Sf

# optimus:29418 SSH-2.0-GerritCodeReview_3.3.4 (APACHE-SSHD-2.4.0)

# optimus:29418 SSH-2.0-GerritCodeReview_3.3.4 (APACHE-SSHD-2.4.0)

# optimus:29418 SSH-2.0-GerritCodeReview_3.3.4 (APACHE-SSHD-2.4.0)

# optimus:29418 SSH-2.0-GerritCodeReview_3.3.4 (APACHE-SSHD-2.4.0)

After upgrade. ssh_host_key file size 2105

$ ssh-keyscan -p 29418 optimus

# optimus:29418 SSH-2.0-GerritCodeReview_3.4.0 (APACHE-SSHD-2.6.0)

# optimus:29418 SSH-2.0-GerritCodeReview_3.4.0 (APACHE-SSHD-2.6.0)

[optimus]:29418 ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBADrl7XHrnOJKVVJXzNHFcpPPMU7tYcp3WWV0EeNSCNwpOL6Wm2zxkHXTllptKDBtDx4VIfe3Ac+P9ynRSCYAz1sygHpV8w0061Kzh5wKagIXbWpzMreYKP8AjdOqVgHu6RxhT5zQ35AXss5HyXrz9XH5i9RJmEv9fXxIGfSCd902LXA0w==

# optimus:29418 SSH-2.0-GerritCodeReview_3.4.0 (APACHE-SSHD-2.6.0)

# optimus:29418 SSH-2.0-GerritCodeReview_3.4.0 (APACHE-SSHD-2.6.0)

# optimus:29418 SSH-2.0-GerritCodeReview_3.4.0 (APACHE-SSHD-2.6.0)

After downgrade and reindex changes. ssh_host_key file size 2199

$ ssh-keyscan -p 29418 optimus

# optimus:29418 SSH-2.0-GerritCodeReview_3.3.4 (APACHE-SSHD-2.4.0)

[optimus]:29418 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxjA6mx/ZqYSyzVLZPakSHAfhy++eeHOwS9ZGbRljvvo/H/AI2271BTebBV+AbdA8jBhUGKXBqo/IXshbMy5dopuJZtnzif8yNXVSDk41k5T8m8BPvBPXH8+2GwWLqt0fY2IUsOJYrbkt552m2R9+R06ha2xFIqJjLoenb5bd7MNicI4V4wKQElsfbSOkkqo1VvgN3cJRCzy4/JUfesYP9I3MLV/FZOO52LTiMINJmkUF8kN3WYPA92TsdSlvvTevIRHY/bZMedjfzB/JiHmNQw/OD9QaYY4iRIjFIreIlJf6CLRUjJlHxlN/ds/aJ+7RWY0sjIV8QiDrD1P57HVUN

# optimus:29418 SSH-2.0-GerritCodeReview_3.3.4 (APACHE-SSHD-2.4.0)

# optimus:29418 SSH-2.0-GerritCodeReview_3.3.4 (APACHE-SSHD-2.4.0)

# optimus:29418 SSH-2.0-GerritCodeReview_3.3.4 (APACHE-SSHD-2.4.0)

# optimus:29418 SSH-2.0-GerritCodeReview_3.3.4 (APACHE-SSHD-2.4.0)

No additional keys were generated in the etc dir.

Would it be possible that the old ssh_host_key file is converted and saved as PEM and the missing files are added automatically? At least up/down grading doesn't change the keys every time.

Best regards,

Peter

[David Ostrovsky]

lucamilanesio schrieb am Dienstag, 15. Juni 2021 um 01:56:53 UTC+2:

On 15 Jun 2021, at 00:48, Makson Lee <cdle...@gmail.com> wrote:

we have same issue after migrated from 3.3 to 3.4.

Could this be related to the Apache Mina SSHD upgrade?

It appears to be related to the same issue upstream in SSHD 2.6.0: [1]

   "Disable weak security settings"

I wrote this issue: [2] and sent this CL for review: [3].

[1] https://issues.apache.org/jira/browse/SSHD-1004

[2] https://crbug.com/gerrit/14676

[3] https://gerrit-review.googlesource.com/c/gerrit/+/309422

[Peter Bruin]

Hi David,

Thanks for the info.

But, what I am actually more interested in in not so much supporting the old key, as I am all for removing the weak settings, but a recommended way forward.

It seems very hard for me to retrieve the current ssh host key from the ssh_host_rsa_key file.

For my test environment I deleted it and ran java -jar /var/gerrit/bin/gerrit.war init --install-all-plugins -d /var/gerrit to re-generate the keys.

After this upgrading from 3.3.4 to 3.4.0 saw no change in keys.

Would you say that is the best way forward?

Best Regards,

Peter

[DmitryB]

Same issue with 2.9->3.4 upgrade
I exported old keys from ssh_host_key file with:

import java.io.*;

import java.security.KeyPair;

import org.bouncycastle.openssl.PEMWriter;

class MyClass {

    public static void main(String[] args) throws Exception {

        FileInputStream fin = new FileInputStream("ssh_host_key");

        ObjectInputStream ois = new ObjectInputStream(fin);

        KeyPair key = (KeyPair) ois.readObject();

        StringWriter privateWrite = new StringWriter();

        PEMWriter privatePemWriter = new PEMWriter(privateWrite);

        privatePemWriter.writeObject(key.getPrivate());

        privatePemWriter.close();

        System.out.println (privateWrite.toString());

        privateWrite.close();

    }

}

and put them at corresponding ssh_host_*_key files
For pub keys: ssh-keygen -y -f 
Old ssh_host_key file should be removed. Then gerrit will generate other missing (ecdsa 256,384 etc) keys

четверг, 17 июня 2021 г. в 11:59:10 UTC+4, peterb...@gmail.com:

[Peter Bruin]

Thanks Dmitry,

I managed to convert the key to PEM and the key remains unchanged. When running the  java -jar /var/gerrit/bin/gerrit.war init --install-all-plugins -d /var/gerrit  the missing keys are indeed generated.

Would be wonderful if this would be part of the upgrade script without having to go through this much trouble.

Thanks again.

Peter

[Luca Milanesio]

On 17 Jun 2021, at 16:17, Peter Bruin <peterb...@gmail.com> wrote:

Thanks Dmitry,

I managed to convert the key to PEM and the key remains unchanged. When running the  java -jar /var/gerrit/bin/gerrit.war init --install-all-plugins -d /var/gerrit  the missing keys are indeed generated.

Would be wonderful if this would be part of the upgrade script without having to go through this much trouble.

Which upgrade script are you referring to? v2.9 to v3.4?

In our release notes we assume people is upgrading from the previous version, not an arbitrary one.

We *could* of course include the scripts for any possible version to the target one, for v3.4 would be:

Migration from v2.1 to v3.4

Migration from  v2.2 to v3.4

Migration from  v2.3 to v3.4

Migration from  v2.4 to v3.4

Migration from  v2.5 to v3.4

Migration from  v2.6 to v3.4

Migration from  v2.7 to v3.4

Migration from  v2.8 to v3.4

Migration from  v2.9 to v3.4

Migration from  v2.10 to v3.4

Migration from  v2.11 to v3.4

Migration from  v2.12 to v3.4

Migration from  v2.13 to v3.4

Migration from  v2.14 to v3.4

Migration from  v2.15 to v3.4

Migration from  v2.16 to v3.4

Migration from from v3.0 to v3.4

Migration from from v3.1 to v3.4

Migration from from v3.2 to v3.4

Migration from from v3.3 to v3.4

As you may understand, that would be a huge effort though :-(

With regards to the upgrade script from v3.3 to v3.4, it is straightforward and a simple ‘java -jar gerrit.war init’ would do the job.

Here the problem is just a bug due to the SSHD upgrade.

Thanks for sharing your thoughts on that.

Luca.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/f9a1fcf3-4424-44d5-a8a2-c1422ee965ccn%40googlegroups.com.

[Peter Bruin]

Hi Luca,

My thoughts are more that it took some time to figure out where the ssh_host_key file even came from and that is was a legacy file from before 2.14.

The presence of this file also prevents newer keys to be generated and as it is binary, it also makes it harder to manage.

Now we all are aware of this key and it might be an idea to convert it to a PEM file and generate the newer keys as part if the init command the same as the host keys are generated when not present.

How about this:

1. during the   ‘java -jar gerrit.war init’   missing ssh_host_xxx_keys are generated, if the old ssh_host_key file is found it is converted to ssh_host_rsa_key and missing keys are added

2. Old ssh_host_key file is saved as ssh_host_key.old?

3. Log a warning that the ssh_host_rsa_key is still sha1 and remind the admin to update it to a more secure version?

Hope that this change would not be to huge an effort.

Ofcourse, for me, I have done this all by hand by now but it might save others now, or in the future some trouble.

Peter

[Kenyon Ralph]

On Thursday, June 17, 2021 at 6:45:20 AM UTC-7 DmitryB wrote:

Same issue with 2.9->3.4 upgrade
I exported old keys from ssh_host_key file with:

import java.io.*;

import java.security.KeyPair;

import org.bouncycastle.openssl.PEMWriter;

class MyClass {

    public static void main(String[] args) throws Exception {

        FileInputStream fin = new FileInputStream("ssh_host_key");

        ObjectInputStream ois = new ObjectInputStream(fin);

        KeyPair key = (KeyPair) ois.readObject();

        StringWriter privateWrite = new StringWriter();

        PEMWriter privatePemWriter = new PEMWriter(privateWrite);

        privatePemWriter.writeObject(key.getPrivate());

        privatePemWriter.close();

        System.out.println (privateWrite.toString());

        privateWrite.close();

    }

}

and put them at corresponding ssh_host_*_key files
For pub keys: ssh-keygen -y -f 
Old ssh_host_key file should be removed. Then gerrit will generate other missing (ecdsa 256,384 etc) keys

Thanks for posting this, very helpful. Just to provide some exact instructions in case anyone else needs this:

Download https://bouncycastle.org/download/bcprov-jdk15on-169.jar and https://bouncycastle.org/download/bcpkix-jdk15on-169.jar from <https://bouncycastle.org/latest_releases.html>. Save the above Java code to MyClass.java. With ssh_host_key in the current directory, run like this:

java --class-path bcpkix-jdk15on-169.jar:bcprov-jdk15on-169.jar MyClass.java

As suggested later in this thread in <https://groups.google.com/g/repo-discuss/c/csJIIyekh28/m/3lKNG3TrAwAJ>, it would be nice if gerrit init would automatically convert the java ssh_host_key to an OpenSSH ssh_host_rsa_key. But I imagine the return on that investment is diminishing since there are probably so few remaining instances of gerrit that are old enough to have ssh_host_key.

Related to this thread, I believe https://gerrit-review.googlesource.com/321757 should fix the problem where new types of ssh host keys are not generated on gerrit init, and https://gerrit-review.googlesource.com/321755 should allow gerrit to use both the OpenSSH-formatted keys and the old ssh_host_key at the same time. I haven't tested these patches though. For my own Gerrit upgrades, to work around these issues, I'm converting ssh_host_key to ssh_host_rsa_key as described above.