PAM troubles while trying to setup ssh OTP

[francesco...@develon.com]

Hi everybody.
I followed instructions in https://github.com/hashicorp/vault-ssh-helper in order to set up OTP passwords.

# vault-ssh-helper -verify -config-file=/etc/vault-ssh-helper.conf
2015/10/27 00:16:29 [INFO] Using SSH Mount point: ssh
2015/10/27 00:16:29 [INFO] Agent verification successful!

First step I noticed I had to remove my local public key from remote's authorized_keys. Now I'm stuck here.
This are the commands I am running locally:

$ vault write ssh/roles/otp_key_role key_type=otp default_user=root cidr_list=192.168.56.107/24
$ vault ssh -role otp_key_role root@192.168.56.107

And these are the logs from sshd at any attempt to login with an OTP:

Oct 27 00:22:23 node sshd[16690]: pam_unix(sshd:auth): unrecognized option [no_set_pass]
Oct 27 00:22:23 node unix_chkpwd[16696]: password check failed for user (root)
Oct 27 00:22:23 node sshd[16690]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.56.1  user=root
Oct 27 00:22:23 node sshd[16688]: error: PAM: User account has expired for root from 192.168.56.1
Oct 27 00:22:23 node sshd[16688]: Connection closed by 192.168.56.1 [preauth]

But in the meanwhile vault-ssh-helper log says (when I enter a correct OTP):

2015/10/27 00:22:23 [INFO] Using SSH Mount point: ssh
2015/10/27 00:22:23 [INFO] root@192.168.56.107 Authenticated!

Any ideas about those PAM errors? Please note that account is not somehow expired: using password or key authentication I still can log in.
Thanks in advance for your support.

[Michael Fischer]

First -- I don't know when the "no_set_pass" option was introduced in pam_unix (and therefore what distributions it might be available on); it's not discussed in any man page I've yet seen.  Vishal?

According to the documentation, pam_unix was added to address a bug in pam_exec (was this bug reported?  does it exist in all versions/distros?).  Since it's marked "optional" in the documentation it should not affect the stack, so it's noise you can ignore.

My guess is that pam_exec or some other module in the auth stack is failing; can you post the entire PAM config for sshd?

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/09bb1c63-da55-4177-9331-306dfae095b8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Francesco Savignago]

I noticed too that the no_set_pass option is mostly undocumented. Currently I am trying with the following /etc/pam.d/sshd

#@include common-auth
auth requisite pam_exec.so quiet expose_authtok log=/tmp/vaultssh.log /usr/local/bin/vault-ssh-helper -config-file=/etc/vault-ssh-helper.conf
#auth optional pam_unix.so no_set_pass use_first_pass nodelay

So it's the same configuration specified in https://github.com/hashicorp/vault-ssh-helper , but with the optional line commented.

With that line commented now, for each login, I observe just this error lines in sshd logs:

Oct 27 00:53:00 node sshd[16861]: error: PAM: User account has expired for root from 192.168.56.1
Oct 27 00:53:00 node sshd[16861]: Connection closed by 192.168.56.1 [preauth]

Thanks again for your support.

[Michael Fischer]

The "account" stage (lines prefixed with "account") modules are determining that the account has expired.  What do the account lines in the PAM config say? 

Usually, expiration dates are set up in /etc/shadow; check shadow(5) for more details.

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/CAAR5vZUV1Y9-a0Pb6O%2BdvxrzGZyH59kqgA-i45iW-eD2A_4XZA%40mail.gmail.com.

[Vishal Nayak]

Hi Francesco,

1) "Agent verification successful"  only means that agent is able to talk to Vault server.

2) There is no need to remove public key from remote's authorized_keys file.

If you do not want to use public key authentication disable it in the sshd_config.

3) 'no_set_pass' is a typo. It should be 'not_set_pass'. Sorry for that. Try again after fixing this typo.

See, http://manpages.ubuntu.com/manpages/hardy/man8/pam_unix.8.html

Since you are seeing "ro...@192.168.56.107 Authenticated!", I assume that you are able to login using the OTP and the concern is only with sshd logs.

I did a fresh checkout and tried the steps exactly similar to you and I was able to login successfully through OTP.

I recommend keeping a separate SSH session open while you do the changes.

In case you are shut out, you can fix configurations from the established session and restart the ssh service.

Regards,

Vishal

[Jeff Mitchell]

Hi Francesco,

You mentioned removing your key from authorized_users; many Linux
distributions these days both a) do not have a valid password set on
root, and b) do not allow password-based root authentication over SSH.

a) is often handled by the password being set to something random and
then marked as expired
b) is often handled with a combination of options in
/etc/ssh/sshd_config (or the appropriate file in your distro)

If the suggestions from Michael/Vishal don't help, I'd be interested
in knowing if this is working for you for a non-root account. If so,
that would at least make it clear what the issue is (and may just be a
good idea while debugging).

--Jeff

> https://groups.google.com/d/msgid/vault-tool/939c8fd5-bb0f-4ce5-abe7-0b94d9a0b9c3%40googlegroups.com.

[francesco...@develon.com]

Hi,
tried your suggestions.
1) Now my /etc/pam/sshd is like that

auth requisite pam_exec.so quiet expose_authtok log=/tmp/vaultssh.log /usr/local/bin/vault-ssh-helper -config-file=/etc/vault-ssh-helper.conf

auth optional pam_unix.so not_set_pass use_first_pass nodelay

2) Verified /etc/shadow: root account was not expired

3) Additional: verified that time in the remote host is in sync with local time (in the case this could tamper with OTP expiration)

4) Created a fresh user account. Behavior is currently the same as before:

$ vault write ssh/roles/otp_key_role key_type=otp default_user=user cidr_list=192.168.56.107/24
Success! Data written to: ssh/roles/otp_key_role
$ vault ssh -role otp_key_role us...@192.168.56.107
OTP for the session is 84340127-673d-7fe7-6d3f-e197888915d0
[Note: Install 'sshpass' to automate typing in OTP]
Password:
Error while running ssh command:exit status 255

And the sshd logs:

Oct 27 03:14:19 node unix_chkpwd[17751]: password check failed for user (user)
Oct 27 03:14:19 node sshd[17746]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.56.1  user=user
Oct 27 03:14:19 node sshd[17744]: error: PAM: User account has expired for user from 192.168.56.1
Oct 27 03:14:19 node sshd[17744]: Connection closed by 192.168.56.1 [preauth]

I am running vault-ssh-helper on a Centos 7 machine, while my vault server is running on OS X.
Thanks again

[Jeff Mitchell]

I've done some searching around and it seems that the "user account
has expired" message tends to be overused a bit when something else is
the actual culprit. Some suggestions are at
http://serverfault.com/a/631061. I also found some posts indicating
that sometimes a PAM module that isn't compiled correctly can cause
issues. I can't help but think that this is due to some variation in
PAM setup between distributions.

Just to check -- can you SSH in normally to that user account using a
password, if you remove the OTP setup from it?

--Jeff

> https://groups.google.com/d/msgid/vault-tool/4fa5a60a-4ee5-4b18-9ae6-b43afed1599c%40googlegroups.com.

[Vishal Nayak]

As Jeff correctly pointed out, PAM configurations for helper, differs with distribution and the documented setup was tested on Ubuntu-14.04.

This may be too much to ask, but if Ubuntu is at your disposal, I recommend trying to get OTP working on that instead.

Ideally, vault-ssh-helper should be a PAM module in itself rather than a binary which is invoked by a PAM module.

Going forward, it should be worked out as an enhancement for the SSH backend. (now that Go supports building a shared object)

This way, configurations will be same for all *nix platforms.

Regards,

Vishal

[Michael Fischer]

I'd like to ask that the most common distributions be supported and tested:

• Ubuntu 14.04, 12.04
• CentOS/RHEL 6, 7
• Amazon Linux

Best regards,

--Michael

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/bcc1c4a7-23e5-4051-be4b-077e8ab6f4f1%40googlegroups.com.

[Francesco Savignago]

Yes, if I revert to the distro's PAM setup I can log to the machine, both with password and public key. The machine is a fresh Centos 7 install with no other tampering, except that I temporary disabled SELinux and firewalld.

You received this message because you are subscribed to a topic in the Google Groups "Vault" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vault-tool/TVf8Ktg2RZ0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/CAORe8GFKXwY_BA5ifts1pMUA6VTztKuOo7-EGa4nbgwdqi5s7A%40mail.gmail.com.