Simp 6.0.0 ldap not allowing SSH users until the user has logged in locally

191 views
Skip to first unread message

smithcl...@gmail.com

unread,
Sep 6, 2017, 1:44:37 PM9/6/17
to SIMP Q&A Forum
Hello,

I have been having a strange issue where I cannot login with my LDAP user to a box until after I have run su <username> on the box. Below is exactly what I see on a new machine:

ssh clayton@<ip>

---------------------------------- ATTENTION -----------------------------------

                      THIS IS A RESTRICTED COMPUTER SYSTEM


This computer system, and all related equipment, networks, and network devices

are provided for authorised use only.  All systems controlled by this

organisation will be monitored for all lawful purposes.  Monitoring includes

the totality of the operating system and connected networks.  No events on this

system are excluded from record and there are no exclusions from this policy.


Use of this system constitutes consent to full monitoring of your activities

for use by the authorised monitoring organisation.  Unauthorised use of this

system, including uninvited connections, may subject you to criminal

prosecution.


The data collected from this system may be used for any purpose by the

collecting organisation.  If you do not agree to this monitoring, discontinue

use of the system IMMEDIATELY.


                      THIS IS A RESTRICTED COMPUTER SYSTEM

---------------------------------- ATTENTION -----------------------------------

Authentication failed.


The /var/log/audit/audit.log says the following

node=<nodename> type=CRYPTO_KEY_USER msg=audit(1504699883.090:23359): pid=6959 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=e7:e3:59:f4:65:1f:53:07:fc:04:1e:e2:d1:03:66:73:28:51:79:70 direction=? spid=6959 suid=0  exe="/usr/sbin/sshd" hostname=? addr=<ip> terminal=? res=success'
node=<nodename> type=CRYPTO_KEY_USER msg=audit(1504699883.090:23360): pid=6959 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=17:5a:01:be:e3:4a:77:a5:90:a9:34:29:d3:b9:67:39:14:37:fe:b8 direction=? spid=6959 suid=0  exe="/usr/sbin/sshd" hostname=? addr=<ip> terminal=? res=success'
node=<nodename> type=CRYPTO_SESSION msg=audit(1504699883.218:23361): pid=6957 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=start direction=from-server cipher=aes128-ctr ksize=128 mac=hmac-sha2-256 pfs=ecdh-sha2-nistp256 spid=6959 suid=74 rport=53921 laddr=10.206.10.195 lport=22  exe="/usr/sbin/sshd" hostname=? addr=<ip> terminal=? res=success'
node=<nodename> type=CRYPTO_SESSION msg=audit(1504699883.218:23362): pid=6957 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=start direction=from-client cipher=aes128-ctr ksize=128 mac=hmac-sha2-256 pfs=ecdh-sha2-nistp256 spid=6959 suid=74 rport=53921 laddr=10.206.10.195 lport=22  exe="/usr/sbin/sshd" hostname=? addr=<ip> terminal=? res=success'
node=<nodename> type=USER_AUTH msg=audit(1504699884.222:23363): pid=6957 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=pubkey_auth rport=53921 acct="clayton" exe="/usr/sbin/sshd" hostname=? addr=<ip> terminal=? res=success'
node=<nodename> type=USER_AUTH msg=audit(1504699884.222:23364): pid=6957 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=key algo=ssh-rsa size=2048 fp=sha1:c2:f7:5e:fb:7f:95:a2:e8:91:b9:0a:d6:bc:dc:47:a2:d5:0b:90:d7 rport=53921 acct="clayton" exe="/usr/sbin/sshd" hostname=? addr=<ip> terminal=? res=success'
node=<nodename> type=USER_ACCT msg=audit(1504699884.224:23365): pid=6957 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:accounting grantors=? acct="clayton" exe="/usr/sbin/sshd" hostname=<ip> addr=<ip> terminal=ssh res=failed'
node=<nodename> type=USER_AUTH msg=audit(1504699884.224:23366): pid=6957 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=pubkey acct="clayton" exe="/usr/sbin/sshd" hostname=? addr=<ip> terminal=ssh res=failed'
node=<nodename> type=CRYPTO_KEY_USER msg=audit(1504699884.224:23367): pid=6957 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=session fp=? direction=both spid=6959 suid=74 rport=53921 laddr=10.206.10.195 lport=22  exe="/usr/sbin/sshd" hostname=? addr=<ip> terminal=? res=success'
node=<nodename> type=CRYPTO_KEY_USER msg=audit(1504699884.225:23368): pid=6957 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=e7:e3:59:f4:65:1f:53:07:fc:04:1e:e2:d1:03:66:73:28:51:79:70 direction=? spid=6957 suid=0  exe="/usr/sbin/sshd" hostname=? addr=<ip> terminal=? res=success'
node=<nodename> type=CRYPTO_KEY_USER msg=audit(1504699884.225:23369): pid=6957 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=17:5a:01:be:e3:4a:77:a5:90:a9:34:29:d3:b9:67:39:14:37:fe:b8 direction=? spid=6957 suid=0  exe="/usr/sbin/sshd" hostname=? addr=<ip> terminal=? res=success'
node=<nodename> type=USER_LOGIN msg=audit(1504699884.225:23370): pid=6957 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=login acct="clayton" exe="/usr/sbin/sshd" hostname=? addr=<ip> terminal=ssh res=failed'

As soon as I run `su clayton` as root on the box once I am able to login fine. 

The following is the hopefully relevant parts of the LDAP used to create the user:

dn: cn=clayton,ou=Group,dc=<removed>,dc=<removed>
objectClass: posixGroup
objectClass: top
cn: clayton
gidNumber: 1002
description: "Clayton Group"
 
dn: uid=clayton,ou=People,dc=<removed>,dc=<removed>
uid: clayton
cn: clayton
givenName: Clayton
sn: Smith
mail: smithcl...@gmail.com
objectClass: inetOrgPerson
objectClass: posixAccount
objectClass: top
objectClass: shadowAccount
objectClass: ldapPublicKey
shadowMax: 180
shadowMin: 1
shadowWarning: 7
shadowLastChange: 10701
sshPublicKey: <removed>
loginShell: /bin/bash
uidNumber: 1015
gidNumber: 1002
homeDirectory: /home/clayton
userPassword: {SSHA}<removed>
pwdReset: TRUE
 
This behavior did not happen in Simp 5.2 as far as I recall so I believe this is a regression. Do you have any insights on what would cause this issue? I can provide any other logs as needed.

Thank you,

Clayton

Nick Markowski

unread,
Sep 6, 2017, 2:09:01 PM9/6/17
to SIMP Q&A Forum
Hey,

Are you using NFS? Did you run the create home directory cron job before trying to authenticate with that user?

smithcl...@gmail.com

unread,
Sep 19, 2017, 6:32:48 PM9/19/17
to SIMP Q&A Forum
Hello Nick,

I have looked through my configuration and I do not believe I am using NFS for syncing user home directories. That being said I am not sure I understand if the NFS module is required or not for user login to work, according to the documentation it seems as if the NFS module is optional. 

Thank you,

Clayton

Trevor Vaughan

unread,
Sep 21, 2017, 4:58:49 PM9/21/17
to smithcl...@gmail.com, SIMP Q&A Forum
Hi Clayton,

The NFS module is not required, it can just cause issues similar to what you are seeing.

Is your LDAP server EL6 or EL7?

We haven't seen this issue and the pam_oddjob_mkhomedir.so library should be auto-creating your home directory if it does not already exist.

Can you check for PAM related errors in /var/log/secure right after you try to login?

Thanks,

Trevor

--
You received this message because you are subscribed to the Google Groups "SIMP Q&A Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/simp/bcbe83d0-6d06-46eb-a2d8-e409949648a5%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Trevor Vaughan
Vice President, Onyx Point, Inc

-- This account not approved for unencrypted proprietary information --

smithcl...@gmail.com

unread,
Sep 21, 2017, 7:55:31 PM9/21/17
to SIMP Q&A Forum
Hello Trevor,

The LDAP server is a SIMP 6 EL7 deploy. I have looked over and compared the secure log on the working and non-working machines. 

Working /var/log/secure
sshd[21105]: FIPS mode initialized
sshd[21105]: Accepted publickey for clayton from <ip> port <port> ssh2: RSA <rsa>

pam_tty_audit(sshd:session): changed status from 0 to 0

Broken /var/log/secure
sshd[14076]: FIPS mode initialized
sshd[14076]: pam_access(sshd:account): access denied for user `clayton' from `<ip>'
sshd[14076]: fatal: Access denied for user clayton by PAM account configuration [preauth]

On the first machine, the home directory has successfully been created from when I ran `su clayton` and no such home directory is created on the second machine (since I have yet to `su clayton` while logged into that box). Happy to provide any other logs that may be useful.

Clayton

To unsubscribe from this group and stop receiving emails from it, send an email to simp+uns...@googlegroups.com.

Trevor Vaughan

unread,
Sep 22, 2017, 1:31:17 PM9/22/17
to smithcl...@gmail.com, SIMP Q&A Forum
This is quite odd.

On the broken system, if you create your home directory prior to logging in, and nothing more, do you still have an issue?

Are both systems completely identical in terms of patches?

Thanks,

Trevor

To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/simp/01ceac05-ad4f-4b92-b135-d800c75c6311%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

smithcl...@gmail.com

unread,
Sep 22, 2017, 4:09:45 PM9/22/17
to SIMP Q&A Forum
I just attempted to create the home directory before logging in and it seems that just creating the folder does not fix the issue. The two machines are exactly the same and were deployed at the same time, the only exception being I have run `su clayton` on one of them and not the other. The /var/log/secure log looks exactly the same after the home directory is created.

Thank you,

Clayton

Trevor Vaughan

unread,
Sep 25, 2017, 9:58:24 AM9/25/17
to smithcl...@gmail.com, SIMP Q&A Forum
Well, that eliminates pam_oddjob_mkhomedir as the issue and makes me think that it might be the SSSD cache.

On the broken system, can you try running 'sss_cache -E' and then logging in as the 'clayton' user?

My guess is that the 'su' command that you're running as root is forcing a cache lookup but a normal login is not for some reason.

Thanks,

Trevor

To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/simp/a2d5ae8b-ba84-44c9-977d-2c60a358cbde%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
Message has been deleted

smithcl...@gmail.com

unread,
Sep 25, 2017, 1:33:27 PM9/25/17
to SIMP Q&A Forum
I have run `sss_cache -E` and then attempted to SSH and it seems the problem still persists with the exact same messages in the `/var/log/secure`. Does PAM or SSSD have any logging options I can turn up to get more visibility into exactly what is failing?

Thank you,

Clayton

Trevor Vaughan

unread,
Sep 25, 2017, 10:32:30 PM9/25/17
to smithcl...@gmail.com, SIMP Q&A Forum
Not really, honestly.

I have had to do the following in the past:

1) Stop sssd
2) Delete everything in /var/lib/sss/db/*
3) Start sssd

Trevor

To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/simp/dca94f87-7453-4fe7-ba5b-5ca995d59554%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

smithcl...@gmail.com

unread,
Sep 26, 2017, 9:37:09 PM9/26/17
to SIMP Q&A Forum
Hmm, it seems to still be throwing the Access denied for user error even after deleting the whole directory and restarting sssd. I also tried restarting to no avail.

Clayton

Trevor Vaughan

unread,
Sep 27, 2017, 10:16:20 AM9/27/17
to smithcl...@gmail.com, SIMP Q&A Forum
Honestly, this seems like a quirk somewhere in the SSSD stack.

Can you try enabling the upstream repos, doing a 'yum update sssd*', restarting the daemon and see if it fixes things?

Thanks,

Trevor

To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/simp/b16841ca-7063-4e78-9e9c-ffe8a5604226%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

smithcl...@gmail.com

unread,
Oct 2, 2017, 8:11:56 PM10/2/17
to SIMP Q&A Forum
I am now running sssd version 1.15.2-50.el7_4.2 on CentOS 7 and am seeing the same issue. I have stood up a full CentOS environment as well to test using the exact same steps and am seeing the errors there as well. I noticed there is a debug option in sssd called debug_level, should I try setting that higher to see if the sssd logs return any useful results?

Thank you,

Clayton

Trevor Vaughan

unread,
Oct 3, 2017, 9:05:59 AM10/3/17
to smithcl...@gmail.com, SIMP Q&A Forum
Yes, please crank up the debugging and see if you can get any information.

Trevor

To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/simp/e72ef0fa-952f-4bf5-a897-c28fb1804037%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

smithcl...@gmail.com

unread,
Oct 18, 2017, 2:56:14 PM10/18/17
to SIMP Q&A Forum
Hello Trevor,

I have bumped the debug level up to 9 and attempted an SSH login when the user has not yet been established on the box, and then again when the user has been established. I have attached the diff of those two files (- are only in the failed login and + are only in the successful login). I have looked through a few times and unfortunately do not see anything that jumps out to me as an obvious reason why it is failing. Does anything stand out as important to you?

Thank you,

Clayton 
diff.log

Trevor Vaughan

unread,
Oct 18, 2017, 3:47:09 PM10/18/17
to smithcl...@gmail.com, SIMP Q&A Forum
Hi Clayton,

This is interesting:

[sssd[be[LDAP]]] [dp_get_account_info_handler] (): Got request for [][BE_REQ_GROUP][1][name=security@ldap]
-[sssd[be[LDAP]]] [dp_attach_req] (): DP Request [Account #9]: New request. Flags [].
+[sssd[be[LDAP]]] [dp_get_account_info_handler] (): Got request for [][BE_REQ_INITGROUPS][1][name=clayton@ldap]

Are you sure that the 'security' GID is allowed to login to your system?

It seems like something is slightly different in your accounts between the two systems.

Trevor

To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/simp/ea879402-77e1-4a9b-adc6-e2068c9a2075%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Trevor Vaughan

unread,
Oct 19, 2017, 5:36:19 PM10/19/17
to smithcl...@gmail.com, SIMP Q&A Forum
Clayton,

We've finally managed to isolate the issue and it turns out that the man page for the `min_id` option in `/etc/sssd/sssd.conf` is a bit misleading.

The section says "For non-primary group memberships, those that are in range will be reported as expected." Apparently, this means that those that are not 'in range' are non-deterministic and probably why you're seeing the behavior that you are.

The patch will be coming out in 6.1.0-0 but the solution is quite simple. If you set simp::sssd::min_id to 500, this should alleviate the issue from this point forward. https://github.com/simp/pupmod-simp-simp/pull/130/files

Apologies for this taking so long. As you discovered, if you do anything with the user, then things start to work which made this difficult to pin down concretely.

Hopefully this solves your issue.

Thanks,

Trevor

Reply all
Reply to author
Forward
0 new messages