Odd ssh issue using generic/centos7

[David Reagan]

I have an odd ssh issue I'm hoping someone can help me troubleshoot.

Using generic/centos7, updated today, I am able to start a vm and `vagrant ssh` into it.

But if I try to ssh in like `ssh -i .vagrant/machines/centos7/libvirt/private_key vagrant@aspectscentos7` (where aspectscentos7 is assigned to the ip in /etc/hosts), I get a `Too many authentication failures` message.

This is the debug output:

$ ssh -vvvv -i .vagrant/machines/centos7/libvirt/private_key vagrant@aspectscentos7
OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017
debug1: Reading configuration data /home/localuser/.ssh/config
debug1: /home/localuser/.ssh/config line 14: Applying options for aspects*
debug1: /home/localuser/.ssh/config line 28: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "aspectscentos7" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to aspectscentos7 [192.168.222.5] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file .vagrant/machines/centos7/libvirt/private_key type -1
debug1: key_load_public: No such file or directory
debug1: identity file .vagrant/machines/centos7/libvirt/private_key-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to aspectscentos7:22 as 'vagrant'
debug3: hostkeys_foreach: reading file "/home/localuser/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /home/localuser/.ssh/known_hosts:27
debug3: load_hostkeys: loaded 1 keys from aspectscentos7
debug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,blowfish-cbc,cast128-cbc,3des-cbc
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,blowfish-cbc,cast128-cbc,3des-cbc
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ssh-rsa SHA256:Tap9+V3e1giFiu7TS4IluxfB/uMGEquYfjuhaZJPj40
debug3: hostkeys_foreach: reading file "/home/localuser/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /home/localuser/.ssh/known_hosts:27
debug3: load_hostkeys: loaded 1 keys from aspectscentos7
debug3: hostkeys_foreach: reading file "/home/localuser/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /home/localuser/.ssh/known_hosts:28
debug3: load_hostkeys: loaded 1 keys from 192.168.222.5
debug1: Host 'aspectscentos7' is known and matches the RSA host key.
debug1: Found key in /home/localuser/.ssh/known_hosts:27
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug2: key: /home/localuser/.ssh/id_ecdsa (0x560372373f40), agent
debug2: key:  (0x560372377da0), agent
debug2: key:  (0x560372377e40), agent
debug2: key:  (0x560372377e90), agent
debug2: key:  (0x560372377f90), agent
debug2: key:  (0x5603723780d0), agent
debug2: key:  (0x560372378250), agent
debug2: key:  (0x5603723783d0), agent
debug2: key:  (0x560372374d10), agent
debug2: key: .vagrant/machines/centos7/libvirt/private_key ((nil)), explicit
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup gssapi-keyex
debug3: remaining preferred: gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-keyex
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug2: we did not send a packet, disable method
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_1000)

debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_1000)

debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: ECDSA SHA256:Mly9AV7EzEbjXey3lzTGHQQ8gjxcTdQ96ismgZNIkYA /home/localuser/.ssh/id_ecdsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: RSA SHA256:dXocWcazdIG05jJ/CSxuH7A6vhd1LUOLXTYCNVZSMs4
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: RSA SHA256:kJw2/ybPT3W1xOVlYuDuMD66U8ge+uJCzhYDqO6LjDQ
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: RSA SHA256:p2j3ChZWUtfMiy44IeSNTV+Am6SteI+4K0C3KoHwxiM
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: RSA SHA256:SBmhD49b+RIMDzWppP8oNoClt0+ygO37Nk8Y0ekobBs
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: RSA SHA256:Oarq8EIQIL5iynB5rMv1A25ijzFE9wTzKbkju4ODaPU
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 1
Received disconnect from 192.168.222.5 port 22:2: Too many authentication failures
Disconnected from 192.168.222.5 port 22

Relevant `~/.ssh/config`:

Host aspects*
 StrictHostKeyChecking no

Host *
  ServerAliveInterval 30
  ServerAliveCountMax 2

AddKeysToAgent  yes

The centos7 section of my Vagrantfile looks like:

 # Configure CentOS 7
  config.vm.define :centos7 do |centos7|
    centos7.vm.box = "generic/centos7"
    centos7.vm.network "private_network", ip: "192.168.222.5"
    centos7.vm.hostname = "aspectscentos7"
    # Use sata due to https://github.com/lavabit/robox/issues/45
    centos7.vm.provider "libvirt" do |domain|
      domain.disk_bus="ide"
    end
    centos7.vm.provision "shell", inline: "yum -y distribution-synchronization"
  end
  config.vm.provider "virtualbox" do |centos7|
    centos7.customize ["modifyvm", :id, "--memory", "4096"]
  end

I'm working on testing ansible roles. So being able to tell ansible to ssh into the box as vagrant is essential. I set the private key path as an ansible variable so I don't have to enter the password every single time.

I'm using Vagrant 2.2.5 with the libvirt provider on Pop OS 18.04 (basicall Ubuntu 18.04.)

Any ideas?

Thanks.

[Dennis Chang]

The VM is accessible via a specific port on address 127.0.0.1 (localhost). Try adding the port and use localhost instead and see if that works.

Dennis

[David Reagan]

Thanks for the response.

Annoyingly, the issue has cleared up since yesterday. I guess a reboot of the host did the trick. Sheesh.

- David

[David Reagan]

Actually.... It just decided to move to a different. My generic/oraclelinux7 vm is now doing the exact same thing as the centos7 vm did.

Any idea what port I can use with localhost? `vagrant port` doesn't work.

The libvirt provider does not support listing forwarded ports. This is
most likely a limitation of the provider and not a bug in Vagrant. If you
believe this is a bug in Vagrant, please search existing issues before
opening a new one.

Scrolling up to the provisioning output doesn't offer any clues to the localhost port. Just a second 192.168.n.n ip.

==> oraclelinux7: Configuring and enabling network interfaces...
    oraclelinux7: SSH address: 192.168.121.82:22
    oraclelinux7: SSH username: vagrant

Thanks.

- David

[David Reagan]

A coworker and I dug into this a lot more.

It appears that ssh is trying to use every private key in ssh-agent instead of just the single key I pass via -i. Since I have more than 6 keys, sshd on the vm blocks me. I tested by increasing MaxAuthTries to 666. That made it work.

I'm guessing that vagrant adds each key to my host's ssh-agent when the vm is started. Correct? So I end up with my personal key, then all 5 of my vagrant vm keys. Then, if I destroy a vm, I end up with even more. Currently I have around 10.

Does that sound about right?

Is there a way to configure ssh to not try every single key?

Or something in vagrant itself that might help?

[Dennis Chang]

On a standard Vagrant VM, you'll see a NAT'ed interface with IP 10.2.0.15 and SSH listening on port 22 (inside the VM) and typically forwarded to port 2222 (for the host).

Typically to access the Vagrant VM you just type 'vagrant ssh' and it's the equivalent of 'ssh -i ~/.vagrant/insecure_private_key -p 2222 vag...@127.0.0.1'.

It seems that you've defined your VM with a private network interface and decided that ssh should listen to that address instead? I.e. 192.168.121.82:22

When you do that, I believe, only connections "on the same subnet, i.e. 192.168.121.X, will be permitted  to establish an ssh connection.

You won't be able to connect from your host.