Bug#1011756: cloud.debian.org: Vagrant debian/testing64: sshd rejects default insecure ssh-rsa key, hangs waiting for SSH
Andrew Chadwick
Severity: normal
X-Debbugs-Cc: a.t.ch...@gmail.com
Dear base box maintainers:
I don’t know if this should be a documentation fix or an image fix. I’ll
just lay out the problem with as much detail as I can so that people can
search it and find better workarounds. I think the FAQ at
https://wiki.debian.org/Teams/Cloud/VagrantBaseBoxes should probably be
updated, given that it’s linked from https://app.vagrantup.com/debian, a
place where people sill be looking.
OpenSSH have addressed a weakness of SHA-1 by removing the ssh-rsa public
key signature algorithm from their list of supported key types. The
openssh-server 1:9.0p1-1 package in the debian/testing64 image incorporates
this change. However the vagrant 2.2.14+dfsg-1 package shipped with Debian
11 “bullseye” has a default “vagrant insecure public key” which requires
ssh-rsa.
See
- https://github.com/hashicorp/vagrant/issues/11783
- https://github.com/hashicorp/packer/issues/10074
## Symptoms
The current debian/testing64 image hangs during “vagrant up”. If I follow
the instructions at https://wiki.debian.org/Teams/Cloud/VagrantQuickStart,
the following happens. I already have vagrant and vagrant-libvirt installed
and configured enough for me to use them. I am using the current Debian 11
versions.
> ~ $ cd "$(mktemp -d)"
> tmp.1PuBegzk6c $ vagrant init debian/testing64
> tmp.1PuBegzk6c $ vagrant up
> ==> default: Checking if box 'debian/testing64' version '20220414.1' is up to date...
> ==> default: Creating image (snapshot of base box volume).
> ==> default: Creating domain with the following settings...
> [...]
> ==> default: Starting domain.
> ==> default: Waiting for domain to get an IP address...
> ==> default: Waiting for SSH to become available...
> [... hangs here, but leave it running ...]
However, I can SSH in to the running virtual just fine from another window.
Why this is, I don’t know.
> ~ $ cd /tmp/tmp.1PuBegzk6c
> tmp.1PuBegzk6c $ vagrant ssh
> Linux testing 5.16.0-6-amd64 #1 SMP PREEMPT Debian 5.16.18-1 (2022-03-29) x86_64
>
> The programs included with the Debian GNU/Linux system are free software;
> the exact distribution terms for each program are described in the
> individual files in /usr/share/doc/*/copyright.
>
> Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
> permitted by applicable law.
> vagrant@testing:~$ cat .ssh/authorized_keys
> ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA6NF8iallvQVp22WDkTkyrtvp9eWW6A8YVr+kz4TjGYe7gHzIw+niNltGEFHzD8+v1I2YJ6oXevct1YeS0o9HZyN1Q9qgCgzUFtdOKLv6IedplqoPkcmF0aYet2PkEDo3MlTBckFXPITAMzF8dJSIFo9D8HfdOV0IAdx4O7PtixWKn5y2hMNG0zQPyUecp4pzC6kivAIhyfHilFR61RGL+GPXQ2MWZWFYbAGjyiYJnAmCP3NOTd0jMZEnDkbUvxhMmBYSdETk1rRgm+R4LOzFUGaHqHDLKLX+FIPKcF96hrucXzcWyLbIbEgE98OHlnVYCzRdK8jlqm8tehUc9c9WhQ== vagrant insecure public key
However, connections from the still-running vagrant command on the host are
still failing, as a quick check of auth.log will confirm. What the heck!
> vagrant@testing:~$ sudo tail /var/log/auth.log
> [...]
> May 26 11:49:18 testing sshd[1644]: userauth_pubkey: signature algorithm ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]
> May 26 11:49:18 testing sshd[1644]: Connection closed by authenticating user vagrant 192.168.121.1 port 34488 [preauth]
## Workaround
One way to fix it from the user perspective is to grab @dustymabe’s fix from
https://github.com/hashicorp/vagrant/issues/11783#issuecomment-720822960,
and apply it over our mysteriously working SSH connection
> vagrant@testing:~$ sudo tee >/dev/null /etc/ssh/sshd_config.d/10-vagrant-insecure-rsa-key.conf <<EOF
> # For now the vagrant insecure key is an rsa key
> # https://github.com/hashicorp/vagrant/issues/11783
> PubkeyAcceptedKeyTypes=+ssh-rsa
> EOF
> vagrant@testing:~$ sudo systemctl restart ssh
This allows the hung vagrant to resume immediately:
> default:
> default: Vagrant insecure key detected. Vagrant will automatically replace
> default: this with a newly generated keypair for better security.
> default:
> default: Inserting generated public key within guest...
> default: Removing insecure key from the guest if it's present...
> default: Key inserted! Disconnecting and reconnecting using new SSH key...
> ==> default: Installing NFS client...
> ==> default: Exporting NFS shared folders...
> ==> default: Preparing to edit /etc/exports. Administrator privileges will be required...
> ==> default: Mounting NFS shared folders...
>
> ==> default: Machine 'default' has a post `vagrant up` message. This is a message
> ==> default: from the creator of the Vagrantfile, and not from Vagrant itself:
> ==> default:
> ==> default: Vanilla Debian box. See https://app.vagrantup.com/debian for help and bug reports
thanks for keeping the boxes fresh
- Andrew
Andrew Chadwick
Followup-For: Bug #1011756
X-Debbugs-Cc: a.t.ch...@gmail.com
I should add that the debian/bullseye64 image works perfectly using the
instructions given. No need to fiddle with the SSH config while it’s
half-up.
- Andrew
Andrew Chadwick
## Workaround 2 (neater)
Testing a bit more, I can confirm that installing vagrant 2.2.19 from
Debian Testing fixes this on Debian 11 (buster) hosts when trying to
connect to debian/testing64 box images.
I guess Vagrant uses ruby-net-ssh for the initial connection using the
insecure default key, the connection it uses to replace that key on the new
machine with the creating user’s. This is why it also ignores
config.ssh.extra_args in Vagrantfiles for that initial connection, but
respects them at “vagrant ssh” time.
Debian 11 users only need to update the vagrant package for now, iupdating
from stable’s 2.2.14+dfsg-1 to testing’s 2.2.19+dfsg-1. They will need to
employ apt pinning to stay on stable while updating vagrant from testing.
One HOWTO is here: https://coderwall.com/p/aatvta/apt-pinning-in-debian, or
see https://wiki.debian.org/AptConfiguration for a deeper dive.
Vagrant’s written in Ruby, and it is packaged to be pretty tolerant about
its dependencies’ versions in Debian.
- Andrew
Emmanuel Kasper
Thanks Andrew for the time you took investigating this.
So I understood right, this was a problem in the way Vagrant ruby
library did ssh connection, as in presence of a ssh-rsa key it would
always try to connect using a SHA-1 signature algorithm (the algo which
was dropped by newer OpenSSH) instead of trying SHA-256 or SHA-512 (see
https://github.com/hashicorp/vagrant/issues/11783#issuecomment-818951984)
This means this was actually a bug in Vagrant, not in the boxes
themselves. This is proved by your fact that the testing version of
Vagrant can perfectly connect to the testing boxes.
Not sure what to here, I am thinking of reassigning this bug report to
the vagrant package, and maybe point the FAQ to this bug report.
Andrew Chadwick
Emmanuel Kasper
https://wiki.debian.org/Vagrant#Vagrant_from_Debian_11_or_previous_versions_hang_on_.22default:_Waiting_for_SSH_to_become_available....22
--
You know an upstream is nice when they even accept m68k patches.
- John Paul Adrian Glaubitz, Debian OpenJDK maintainer
Lucas Nussbaum
> We have now that documented in the Debian Wiki:
> https://wiki.debian.org/Vagrant#Vagrant_from_Debian_11_or_previous_versions_hang_on_.22default:_Waiting_for_SSH_to_become_available....22
The bug in ruby-net-ssh is tracked as #1009155 and #1008541, and is
fixed in stable-proposed-updates and bullseye-backports. I've updated
the wiki page accordingly.
Lucas