Using vault-ssh-helper

[Kevin Pauba]

I'm having troubles getting OTP ssh access to a host using vault.

When I attempt to authenticate with a OTP to the host, the password is not accepted.  I see the following in /tmp/vaultssh.log:

Put http://<vault_ip>:8200/v1/ssh/verify: dial tcp <vault_ip>:8200: permission denied

When I log into the host (using my key), I'm able to use curl to hit the service with this command:

curl -vvv -s -X PUT -H 'Content-Type: application/json' -d @otp.json http://<vault_ip>:8200/v1/ssh/verify

I get a "200 OK" response with the expected data.

Here's the contents of the vault-ssh-helper config file:

vault_addr="http://<vault_ip>:8200"

ssh_mount_point="ssh"

Any idea on what I'm doing wrong?

Thanks!

[Vishal Nayak]

Hi Kevin,

How does your vault-ssh-helper config file look like? Since you are not using TLS, you need to disable it in the config file ("tls_skip_verify = false").

Also, when you login to the host and use the verify service (as you have done using curl above), vault deletes the key after returning "200 OK" (hence an OTP).

You will have to recreate an OTP credential in the server before testing again.

Try using $ vault-ssh-helper -verify -config-file=<path-to-config-file> to see if the agent is working properly.

Regards,

Vishal

[Kevin Pauba]

Thanks for the reply Vishal!

I modified the /etc/vault-ssh-helper/config.hcl to include "tls_skip_verify = true" (I wan't it true, right?) but the same error persists in /tmp/vaultssh.log (and authentication with "vault ssh ..." still doesn't work.

For giggles, I tried "tls_skip_verify = false" to no avail.

I had previously used "vault-ssh-helper -verify -config-file=/etc/vault-ssh-helper/config.hcl" which was successful (and still is).

I also realized (like you also mentioned), that I had to recreate the OTP in vault in order for subsequent 'curl' requests to work ... I would have saved some time today if I posted (and received your hints) earlier.

So, I'm still confused as to why things aren't working.  I've also double-checked my /etc/pam.d/sshd and /etc/ssh/sshd_config file settings for correctness.

Any additional ideas are welcomed.

l8r

[Vishal Nayak]

Hi Kevin,

Yes. It has to be "tls_skip_verify = true". Which distribution are you using?

If the host is able to reach the service and if the helper is also verified, it can only mean that the request is getting blocked only during the authentication.

It could mean that either PAM config or SSH config needs to be cross checked.

Can you share those two file contents?

Regards,

Vishal

[Kevin Pauba]

I'm using Vault V0.4.1 and yesterday's clone of hashicorp/vault-ssh-help from github (along with Consul V0.6.0).

I'm not very familiar with PAM so the likelihood of me messing that up are pretty good.

Here's the /etc/pam.d/sshd file contents:

#%PAM-1.0

auth       required     pam_sepermit.so

auth       substack     password-auth

auth       include      postlogin

# Used with polkit to reauthorize users in remote sessions

-auth      optional     pam_reauthorize.so prepare

account    required     pam_nologin.so

account    include      password-auth

password   include      password-auth

# pam_selinux.so close should be the first session rule

session    required     pam_selinux.so close

session    required     pam_loginuid.so

# pam_selinux.so open should only be followed by sessions to be executed in the user context

session    required     pam_selinux.so open env_params

session    required     pam_namespace.so

session    optional     pam_keyinit.so force revoke

session    include      password-auth

session    include      postlogin

# Used with polkit to reauthorize users in remote sessions

-session   optional     pam_reauthorize.so prepare

# Used with vault-ssh-helper

auth sufficient pam_exec.so quiet expose_authtok log=/tmp/vaultssh.log /usr/local/bin/vault-ssh-helper -config-file=/etc/vault-ssh-helper/config.hcl

auth optional pam_unix.so not_set_pass use_first_pass nodelay

... and the contents of /etc/ssh/sshd_config:

HostKey /etc/ssh/ssh_host_rsa_key

HostKey /etc/ssh/ssh_host_ecdsa_key

SyslogFacility AUTHPRIV

AuthorizedKeysFile .ssh/authorized_keys

PasswordAuthentication no

ChallengeResponseAuthentication yes

GSSAPIAuthentication yes

GSSAPICleanupCredentials yes

UsePAM yes

X11Forwarding yes

UsePrivilegeSeparation sandbox # Default for new installations.

AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES

AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT

AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE

AcceptEnv XMODIFIERS

Subsystem sftp /usr/libexec/openssh/sftp-server

Thanks!

[Vishal Nayak]

Hi Kevin,

The SSH config file looks good to me.

Regarding PAM, just so you know, the configuration provided in the github page was for Ubuntu 14.04 and the configuration differs with each distribution.

It seems like you are using SELinux and I cannot say for sure what configuration works with that.

However, it looks like pam_exec.so exists. Otherwise, vault-ssh-helper would not have been invoked in the first place.

I am not sure if pam_unix.so exists though.

Another thing, I had disabled the previous authentication mechanism in Ubuntu to get vault-ssh-helper working.

So shooting in the blind (since I don't have the SELinux setup to test for you), I suggest commenting out the first three lines of authentication mechanisms.

Then, paste the last two lines just below these. PAM works sequentially, meaning if one auth instruction fails, it falls back on the next one. So, I would just not mess with sequences.

# auth required pam_sepermit.so

# auth substack password-auth

# auth include postlogin

auth sufficient pam_exec.so quiet expose_authtok log=/tmp/vaultssh.log /usr/local/bin/vault-ssh-helper -config-file=/etc/vault-ssh-helper/config.hcl

auth optional pam_unix.so not_set_pass use_first_pass nodelay

Again, I don't know if this works. Let me know.

Ideally, vault-ssh-helper should be a PAM module instead of a static binary. It is definitely in the back of my head to implement an independent PAM module.

Regards,

Vishal

[Kevin Pauba]

Thanks, Vishal.

I made the suggested changes and it still didn't work but it points me to where I need to do a bit more learning.

I'm running CentOS here which I now see is SELinux enforced (as you observed).  I just ran 'setenforce 0' on the host that I'm attempting to log in to and then tried a 'VAULT_ADDR=... vault ssh -role otp_key_role user@<ip> from another host and it worked just fine!!! 

I will be reading up a bit on PAM for SELinux to see if I can get it to work without having SELinux set to the "Permissive" mode.

Thanks so much for your help!

[Lukasz Mancewicz]

Hi Kevin,

I have the same problem, did you manage to get this to work without setting SELinux set to the "Permissive" mode?

Thanks! 

Lukasz

[Black]

Hi Kevin,

Would it be ok if you can share an instruction on how to enable SSH OTP on Centos?

Thanks,

[domingo gil]

i have same error with ubuntu 16.04, I have time out and have the authenticated message

*** Thu Aug 30 17:17:35 2018

2018/08/30 17:17:35 [INFO] using SSH mount point: ssh

2018/08/30 17:17:35 [INFO] devs...@52.179.188.151 authenticated!

but not login