vault ssh; works for me, not for others?

[Jay Christopherson]

I'm trying to figure this out - I have the ssh backend setup and am using vault-ssh-helper.  I originally setup myself with more permissions (root policy) than others, but have since elevated a couple others while I troubleshoot this.  

I setup a single user on each host that a Vault user can ssh into.  I have no problems logging in to the user on any host.  I'm using Vault 0.5.1 on Mac OS X.  There are 4 hosts (all running Ubuntu 14.04) that no one else can log into.  All of the other hosts are Ubuntu 15.04 and work fine.  The users are running Ubuntu w/ Vault 0.5.1.  On these affected hosts, everyone else gets:

Failed to establish SSH connection:exit status 255

and in the auth.log, I only see this when they attempt to login:

Connection closed by 192.168.98.6 [preauth]

I checked out their token when they auth'd and it matches what I get when I auth:

Password (will be hidden):
Successfully authenticated!
token: 35a74f5e-dd62-5f33-ec99-5134d926c0b4
token_duration: 0
token_policies: [ssh, root]

So, I'm at a bit of a loss here - I verified that they can all login through SSH RSA keys (eliminating network/etc... issues) and that they *can* login to OTHER Vault enabled hosts. 

I also enable audit-logging and when they attempt to vault ssh, I can see that they are being checked against the vault successfully.

[Jay Christopherson]

I was able to repro their issue using Vault on a Ubuntu client, using the same credentials that work for me on Mac OS X. 

[Jay Christopherson]

Though the error I get is:

Failed to establish SSH connection:exit status 6

[Jay Christopherson]

Actually, that was just due to a SSH key needing to be accepted - after doing that, I get the same error they do:

Failed to establish SSH connection:exit status 255

[vishal nayak]

Hi Jay,

Are you using the latest version of vault-ssh-helper?

vault-ssh-helper 0.1.1 was released along with Vault 0.5.1.

To see if the config file is fine and vault-ssh-helper is actually able to talk to vault, use th -verify-only option in the helper.

If possible, can you share the "/etc/pam.d/sshd" configuration you are using?

[preauth] looks like the SSH/PAM configurations are not allowing the connection to go through.

Regards,

Vishal

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/23121a33-5f40-4e14-8960-8e9bf1cc451e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

vn

[Jay Christopherson]

Thanks Vishal - 

I did run the verify-only option and it said everything was fine.  Like I said, it WFM, but not for this particular user.  Running SSHD in debug mode, I can see that they *are* getting connected, but not authenticated on these particular hosts.  He is able to successfully log in to other Vault-ssh enabled hosts just fine.

 Here is the pam.d/sshd config:

# PAM configuration for the Secure Shell service


# Standard Un*x authentication.
#@include common-auth
auth requisite pam_exec.so quiet expose_authtok log=/tmp/vaultssh.log /usr/local/bin/vault-ssh-helper -config-file=/etc/vault-ssh-helper.d/config.hcl
auth optional pam_unix.so not_set_pass use_first_pass nodelay


# Disallow non-root logins when /etc/nologin exists.
account    required     pam_nologin.so


# Uncomment and edit /etc/security/access.conf if you need to set complex
# access limits that are hard to express in sshd_config.
# account  required     pam_access.so


# Standard Un*x authorization.
@include common-account


# SELinux needs to be the first session rule.  This ensures that any
# lingering context has been cleared.  Without this it is possible that a
# module could execute code in the wrong domain.
session [success=ok ignore=ignore module_unknown=ignore default=bad]        pam_selinux.so close


# Set the loginuid process attribute.
session    required     pam_loginuid.so


# Create a new session keyring.
session    optional     pam_keyinit.so force revoke


# Standard Un*x session setup and teardown.
@include common-session


# Print the message of the day upon successful login.
# This includes a dynamically generated part from /run/motd.dynamic
# and a static (admin-editable) part from /etc/motd.
session    optional     pam_motd.so  motd=/run/motd.dynamic
session    optional     pam_motd.so noupdate


# Print the status of the user's mailbox upon successful login.
session    optional     pam_mail.so standard noenv # [1]


# Set up user limits from /etc/security/limits.conf.
session    required     pam_limits.so


# Read environment variables from /etc/environment and
# /etc/security/pam_env.conf.
session    required     pam_env.so # [1]
# In Debian 4.0 (etch), locale-related environment variables were moved to
# /etc/default/locale, so read that as well.
session    required     pam_env.so user_readenv=1 envfile=/etc/default/locale


# SELinux needs to intervene at login time to ensure that the process starts
# in the proper default security context.  Only sessions which are intended
# to run in the user's context should be run after this.
session [success=ok ignore=ignore module_unknown=ignore default=bad]        pam_selinux.so open


# Standard Un*x password updating.
@include common-password

[Vishal Nayak]

Hi Jay,

There is a use case where the authentication would not work, but there is a way to fix this.

Lets just check if you are hitting this case.

From the client machine, the DNS name is resolved into an IP and the OTP generated in Vault will be associated to this IP (along with username).

During authentication, the "vault-ssh-helper" (using the OTP) receives the "username and IP" from the Vault server and tries to match those, in the remote host.

The IP might not match sometimes since the IP address that was resolved in the client, and the address of the network interface in the remote host, may be different.

In such cases, you can provide CIDR blocks of IPs that should be allowed during authentication.

This is specified in the config file of vault-ssh-helper.

See "allowed_cidr_list" (https://github.com/hashicorp/vault-ssh-helper).

Hope this helps!

Regards,

Vishal

[Jay Christopherson]

Thanks Vishal, I'll check that out.  In this scenario, the user was VPN'd in to our corporate network from home, and the host he was trying to access was in a further remote location (via IPSec tunnel).  If the OTP is tied to the IP of the client, then I could see the potential for a problem there.

[Vishal Nayak]

Hi Jay,

Just a small clarification.

The OTP will not be tied to the IP of the client.

Instead, it will be tied to the resolved IP address of the remote host, in the client machine.

If this IP is not tied to the network interface at the remote host, you can supply the same as part of "allowed_cidr_list" to "vault-ssh-helper".

In other words, "allowed_cidr_list" should represent the possible IP addresses that a remote host can resolve to.

Regards,

Vishal

[Jay Christopherson]

Thanks for the clarification Vishal.

So, after some more testing, with the user in front of me and using generated OTP's, I noticed something odd - everytime he ssh'd to the host, he would be asked to accept the host key.  He would type 'yes', but it would still ask him every time, which is weird.  It's been doing that to him for a while and he's just been living with it.  I had him delete the entry in his known_hosts file and try one more time....  and lo and behold, it worked.  :-)

There are a bunch of different ways he could have solved this, but that particular tidbit falls under the heading of "things it would have been great to know about YESTERDAY"... 

Thanks for the assistance and sorry for the red herring...

[Vishal Nayak]

Hi Jay,

No problem. Glad that it worked!

Regards,

Vishal