LDAP stuck threads

[Will Humphries]

Has anyone seen stuck threads that look like this in a thread dump?

"QuartzScheduler_Worker-1" prio=10 tid=0x00007ff588a88000 nid=0xb2fb in
Object.wait() [0x00007ff5c46a9000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007207e5148> (a java.lang.Object)
at java.lang.Object.wait(Object.java:503)
at
com.novell.ldap.Connection.acquireWriteSemaphore(Connection.java:286)
- locked <0x00000007207e5148> (a java.lang.Object)
at
com.novell.ldap.Connection.isConnectionAlive(Connection.java:862)
at
com.novell.ldap.LDAPConnection.isConnectionAlive(LDAPConnection.java:609)
at
edu.amc.sakai.user.NativeLdapConnectionLivenessValidator.isConnectionAlive(NativeLdapConnectionLivenessValidator.java:39)
at
edu.amc.sakai.user.PooledLDAPConnectionFactory.validateObject(PooledLDAPConnectionFactory.java:200)

More examples at the bottom of this email.

They don't show up until after at least one exception like this is
written to catalina.out

Exception in thread "Thread-6099" java.lang.NullPointerException
at com.novell.ldap.Message.putReply(Message.java:340)
at
com.novell.ldap.Connection$ReaderThread.run(Connection.java:1303)
at java.lang.Thread.run(Thread.java:745)


It seems like the stuck threads can originate from pretty much anything
that will attempt to look up a user - we've seen login, roster, the
announcements tool, etc. The threads remain stuck until the application
is restarted. This is particularly troublesome when it sticks our
nightly sis integration quartz job.

I found this that looks pretty similar:
https://forums.novell.com/showthread.php/491011-acquireWriteSemaphore-hang-Null-Pointer-exception-in-Reader-thread
. Just wondering if anyone has run into this with Sakai in particular.

Thanks,
Will

"ajp-bio-8009-exec-2158" daemon prio=10 tid=0x00007f4f6c100000
nid=0x6b56 in Object.wait() [0x00007f4f1415d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000006a407a040> (a java.lang.Object)
at java.lang.Object.wait(Object.java:503)
at
com.novell.ldap.Connection.acquireWriteSemaphore(Connection.java:286)
- locked <0x00000006a407a040> (a java.lang.Object)
at com.novell.ldap.Connection.writeMessage(Connection.java:765)
at com.novell.ldap.Connection.writeMessage(Connection.java:730)
at com.novell.ldap.Message.sendMessage(Message.java:101)
at com.novell.ldap.MessageAgent.sendMessage(MessageAgent.java:286)
at com.novell.ldap.LDAPConnection.search(LDAPConnection.java:3531)
at com.novell.ldap.LDAPConnection.search(LDAPConnection.java:3412)
at
com.rsmart.ldap.JLDAPDirectoryProvider.getResultsFromDirectory(JLDAPDirectoryProvider.java:655)
at
com.rsmart.ldap.JLDAPDirectoryProvider.getUsers(JLDAPDirectoryProvider.java:738)
at
org.sakaiproject.user.impl.DbUserService$DbStorage.getUsersByIds(DbUserService.java:702)
at
org.sakaiproject.user.impl.BaseUserDirectoryService.getUsers(BaseUserDirectoryService.java:867)
at
org.sakaiproject.profile2.logic.SakaiProxyImpl.getUsers(SakaiProxyImpl.java:292)
at
org.sakaiproject.profile2.logic.ProfileConnectionsLogicImpl.getConnectedUsers(ProfileConnectionsLogicImpl.java:490)
at
org.sakaiproject.profile2.logic.ProfileConnectionsLogicImpl.getConnectionsForUser(ProfileConnectionsLogicImpl.java:65)
at
org.sakaiproject.profile2.service.ProfileServiceImpl.getConnectionsForUser(ProfileServiceImpl.java:39)



"ajp-bio-8009-exec-2872" daemon prio=10 tid=0x00007f4f6c183000
nid=0x3069 in Object.wait() [0x00007f4f13c58000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000006c0f802b8> (a java.lang.Object)
at java.lang.Object.wait(Object.java:503)
at
com.novell.ldap.Connection.acquireWriteSemaphore(Connection.java:286)
- locked <0x00000006c0f802b8> (a java.lang.Object)
at
com.novell.ldap.Connection.isConnectionAlive(Connection.java:862)
at
com.novell.ldap.LDAPConnection.isConnectionAlive(LDAPConnection.java:609)
at
edu.amc.sakai.user.NativeLdapConnectionLivenessValidator.isConnectionAlive(NativeLdapConnectionLivenessValidator.java:39)
at
edu.amc.sakai.user.PooledLDAPConnectionFactory.validateObject(PooledLDAPConnectionFactory.java:200)
at
org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1198)
at
edu.amc.sakai.user.PoolingLdapConnectionManager.getBoundConnection(PoolingLdapConnectionManager.java:102)
at
com.rsmart.ldap.JLDAPDirectoryProvider.getConnection(JLDAPDirectoryProvider.java:333)
at
com.rsmart.ldap.JLDAPDirectoryProvider.authenticateUserHelper(JLDAPDirectoryProvider.java:235)
at
com.rsmart.ldap.JLDAPDirectoryProvider.authenticateUser(JLDAPDirectoryProvider.java:267)
at
org.sakaiproject.user.impl.BaseUserDirectoryService.getProviderAuthenticatedUser(BaseUserDirectoryService.java:1680)
at
org.sakaiproject.user.impl.BaseUserDirectoryService.authenticate(BaseUserDirectoryService.java:1614)
at
org.sakaiproject.user.impl.UserAuthnComponent.authenticate(UserAuthnComponent.java:108)
at
org.sakaiproject.login.impl.LoginServiceComponent.authenticate(LoginServiceComponent.java:90)
at
org.sakaiproject.login.tool.SkinnableLogin.doPost(SkinnableLogin.java:333)

[Sam Ottenhoff]

Have you tried disabling pooling? Is the underlying operating system possibly limiting the number of open sockets/processes/threads?

--
You received this message because you are subscribed to the Google Groups "Sakai Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sakai-dev+...@apereo.org.
To post to this group, send email to saka...@apereo.org.
Visit this group at https://groups.google.com/a/apereo.org/group/sakai-dev/.

[Will Humphries]

We did not try disabling pooling, and we did not see evidence of an
issue with sockets, processes, or threads. I think something slowed down
the average response we get from our LDAP server enough that we
occassionally ran into an existing timing issue in the com.novell.ldap
library. This postdescribes it pretty well.
https://forums.novell.com/showthread.php/491011-acquireWriteSemaphore-hang-Null-Pointer-exception-in-Reader-thread

I got the com.novell.ldap source from
git://git.openldap.org/openldap-jldap.git , made the 1 line change to
avoid setting conn to null in Message.java, and pointed the providers
project at the new version of the ldap jar I built. We haven't seen the
nullpointerexception or any stuck threads since then. It's only been a
week, I'll update here if anything changes.

-Will

> <mailto:sakai-dev%2Bunsu...@apereo.org>.

> To post to this group, send email to saka...@apereo.org

> <mailto:saka...@apereo.org>.

[Earle Nietzel]

I have seen this stack before where the finalizer thread is stuck in the acquireWriteSemaphore method and basically VM comes to a halt because the finalizer thread is locked up and nothing can be GC'd!

This brings to question that it maybe be time to get off the jldap library as it is old and hasn’t had any activity since 2006.

A possible replacement could be https://www.ldap.com/unboundid-ldap-sdk-for-java, one nice thing about this library is you can easily start an in memory ldap server for unit testing, it is well maintained and is active (source https://github.com/UnboundID/ldapsdk).

To unsubscribe from this group and stop receiving emails from it, send an email to sakai-dev+...@apereo.org.
To post to this group, send email to saka...@apereo.org.

[Matthew Buckett]

On 22 February 2016 at 14:36, Earle Nietzel <ea...@longsight.com> wrote:
> I have seen this stack before where the finalizer thread is stuck in the
> acquireWriteSemaphore method and basically VM comes to a halt because the
> finalizer thread is locked up and nothing can be GC'd!
>
> This brings to question that it maybe be time to get off the jldap library
> as it is old and hasn’t had any activity since 2006.

I'm not sure it's quite that bad, but it's hadn't been touched in 5+
years which is not good.

> A possible replacement could be
> https://www.ldap.com/unboundid-ldap-sdk-for-java, one nice thing about this
> library is you can easily start an in memory ldap server for unit testing,
> it is well maintained and is active (source
> https://github.com/UnboundID/ldapsdk).

Yep, I think there's allot to be said for moving to a newer LDAP library.

--
Matthew Buckett, VLE Developer, IT Services, University of Oxford

[Sam Ottenhoff]

> > A possible replacement could be
> > https://www.ldap.com/unboundid-ldap-sdk-for-java, one nice thing about this
> > library is you can easily start an in memory ldap server for unit testing,
> > it is well maintained and is active (source
> > https://github.com/UnboundID/ldapsdk).
>
> Yep, I think there's allot to be said for moving to a newer LDAP library.

I created a new provider called "unboundid" and confirmed it's working inside Sakai master. I have not touched the tests:

  https://github.com/sakaiproject/sakai/pull/1924

Any interest out there?

[Jeff Pasch]

Hi Sam,

Circling back on this. We're definitely interested and can help out with testing. We'll get this set up in one of our test instances. 

Thanks!

Jeff

----

Jeff Pasch

LMS Product Director

NYU IT .edu Services

New York University

212.992.7555

----

--

You received this message because you are subscribed to the Google Groups "Sakai Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sakai-dev+...@apereo.org.
To post to this group, send email to saka...@apereo.org.