[cas-user] FastBindLdapAuthenticationHandler thread limits
Kirk, Matt
During performance testing of our CAS cluster the testers have been seeing "SocketException: Too many open files" as described at the end of this email.
CAS is configured with the FastBindLdapAuthenticationHandler. Is it possible to use a pooled context source with this configuration? When I wire the pooledContextSource (as described here: https://wiki.jasig.org/display/CASUM/LDAP) into the fast bind handler I see the following exception:
org.springframework.webflow.execution.ActionExecutionException: Exception thrown executing [AnnotatedAction@6bd1f967 targetAction = [EvaluateAction@264d107e expression = authenticationViaFormAction.submit(flowRequestContext, flowScope.credentials, messageContext), resultExpression = [null]], attributes = map[[empty]]] in state 'realSubmit' of flow 'login' -- action execution attributes were 'map[[empty]]'....
at org.springframework.webflow.execution.ActionExecutor.execute(ActionExecutor.java:60)
at org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:101)
at org.springframework.webflow.engine.State.enter(State.java:194)
at org.springframework.webflow.engine.Transition.execute(Transition.java:227)
Caused by:
java.lang.UnsupportedOperationException: Not supported for this implementation
at org.springframework.ldap.pool.factory.PoolingContextSource.getContext(PoolingContextSource.java:436)
at org.jasig.cas.adaptors.ldap.FastBindLdapAuthenticationHandler.authenticateUsernamePasswordInternal(FastBindLdapAuthenticationHandler.java:32)
at org.jasig.cas.authentication.handler.support.AbstractUsernamePasswordAuthenticationHandler.doAuthentication(AbstractUsernamePasswordAuthenticationHandler.java:58)
at org.jasig.cas.authentication.handler.support.AbstractPreAndPostProcessingAuthenticationHandler.authenticate_aroundBody2(AbstractPreAndPostProcessingAuthenticationHandler.java:72)
at org.jasig.cas.authentication.handler.support.AbstractPreAndPostProcessingAuthenticationHandler.authenticate_aroundBody3$advice(AbstractPreAndPostProcessingAuthenticationHandler.java:44)
at org.jasig.cas.authentication.handler.support.AbstractPreAndPostProcessingAuthenticationHandler.authenticate(AbstractPreAndPostProcessingAuthenticationHandler.java:1)
at org.jasig.cas.authentication.AuthenticationManagerImpl.authenticateAndObtainPrincipal(AuthenticationManagerImpl.java:78)
at org.jasig.cas.authentication.AbstractAuthenticationManager.authenticate_aroundBody0(AbstractAuthenticationManager.java:41)
at org.jasig.cas.authentication.AbstractAuthenticationManager.authenticate_aroundBody1$advice(AbstractAuthenticationManager.java:44)
...
at org.jasig.cas.authentication.AbstractAuthenticationManager.authenticate(AbstractAuthenticationManager.java:1)
Is the pooled context source only suitable for the Bind handler and not the Fast Bind handler?
Thanks,
Matt
From tester:
When we introduce a large number of users (3500 ramped in over around 30 minutes) to our 4 server CAS estate we are observing very high thread usage in each of the 4 Tomcat containers.
Information in this email including any attachments may be privileged, confidential and is intended exclusively for the addressee. The views expressed may not be official policy, but the personal views of the originator. If you have received it in error, please notify the sender by return e-mail and delete it from your system. You should not reproduce, distribute, store, retransmit, use or disclose its contents to anyone. Please note we reserve the right to monitor all e-mail communication through our internal and external networks. SKY and the SKY marks are trade marks of British Sky Broadcasting Group plc and are used under licence. British Sky Broadcasting Limited (Registration No. 2906991), Sky Interactive Limited (Registration No. 3554332), Sky-In-Home Service Limited (Registration No. 2067075) and Sky Subscribers Services Limited (Registration No. 2340150) are direct or indirect subsidiaries of British Sky Broadcasting Group plc (Registration No. 2247735). All of the companies mentioned in this paragraph are incorporated in England and Wales and share the same registered office at Grant Way, Isleworth, Middlesex TW7 5QD.
--
You are currently subscribed to cas-...@lists.jasig.org as: jasig-cas-user...@googlegroups.com
To unsubscribe, change settings or access archives, see http://www.ja-sig.org/wiki/display/JSG/cas-user
Marvin Addison
> possible to use a pooled context source with this configuration?
No, although arguably it should be able to to do this. Unfortunately
that's a spring-ldap enhancement. The good news is that the author is
someone in the Jasig community we could work with to make it happen.
I've been thinking about submitting an enhancement over there. Anyone
else interested?
The problem reported by your tester is only remotely related to
pooling connections. You're starting long-lived connections that are
not closed in a timely manner.
> Once the number of threads reaches around 950 in each of the Tomcat
> containers the process runs out of open files as the current Linux limit on
> open files per process is 1024 (see below)
Why aren't these threads dying naturally on connection close? We do
the right thing in CAS:
} finally {
if (dirContext != null) {
LdapUtils.closeContext(dirContext);
}
}
You want closing the LDAP connection to send the FIN to initiate tear
down immediately. I'd expect this to be the default behavior, but
I've been surprised by socket behavior in Java in the past.
> without pooling we are surprised to see that the LDAP threads remain in
> process waiting on a socket read to LDAP, although we periodically observe
> (after approx 15 minutes) a large number of threads being released although
> this could be down to some form of application housekeeping or as a
> consequence of TCP Keepalives.
I'm surprised, but proper management of sockets in Java is somewhat of
an art and I do agree that TCP keepalives may be the problem. Can you
provide some more information about your environment:
- Java version
- Operating system
- LdapContextSource configuration
The good news is that there is a way to get at most any socket
property needed, but we'll dig into that after I get some more
information and try to reproduce here.
Thanks,
M
Kirk, Matt
As requested:
$ uname -a
Linux uncsg07a 2.6.18-238.12.1.el5 #1 SMP Sat May 7 20:18:50 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
$ /apps/java/jdk-1.6/bin/java -version
java version "1.6.0_10"
Java(TM) SE Runtime Environment (build 1.6.0_10-b33)
Java HotSpot(TM) 64-Bit Server VM (build 11.0-b15, mixed mode)
<bean id="contextSource" class="org.springframework.ldap.core.support.LdapContextSource">
<property name="url" value="${ldap.context.url}" />
<property name="base" value="${ldap.context.base}" />
<property name="pooled" value="true"/>
</bean>
Thanks,
Matt
________________________________________
From: Marvin Addison [marvin....@gmail.com]
Sent: 07 October 2011 16:16
To: cas-...@lists.jasig.org
Subject: Re: [cas-user] FastBindLdapAuthenticationHandler thread limits
Thanks,
M
--
You are currently subscribed to cas-...@lists.jasig.org as: Matt...@bskyb.com
To unsubscribe, change settings or access archives, see http://www.ja-sig.org/wiki/display/JSG/cas-user
Information in this email including any attachments may be privileged, confidential and is intended exclusively for the addressee. The views expressed may not be official policy, but the personal views of the originator. If you have received it in error, please notify the sender by return e-mail and delete it from your system. You should not reproduce, distribute, store, retransmit, use or disclose its contents to anyone. Please note we reserve the right to monitor all e-mail communication through our internal and external networks. SKY and the SKY marks are trade marks of British Sky Broadcasting Group plc and are used under licence. British Sky Broadcasting Limited (Registration No. 2906991), Sky Interactive Limited (Registration No. 3554332), Sky-In-Home Service Limited (Registration No. 2067075) and Sky Subscribers Services Limited (Registration No. 2340150) are direct or indirect subsidiaries of British Sky Broadcasting Group plc (Registration No. 2247735). All of the companies mentioned in this paragraph are incorporated in England and Wales and share the same registered office at Grant Way, Isleworth, Middlesex TW7 5QD.
--
Kirk, Matt
Just noticed the bean configuration for the contextSource has pooled set to true which on looking at the com.sun.jndi.ldap.LdapClient.close method instructs the pool to release the connection and not close it. I've stepped through in a debugger and I can see this happening with pooled = true and see it call conn.cleanup when instead when pooled = false. So I think I've found the cause of the log running connections.
Regards,
Matt
________________________________________
From: Kirk, Matt [Matt...@bskyb.com]
Sent: 07 October 2011 17:16
To: cas-...@lists.jasig.org
Subject: RE: [cas-user] FastBindLdapAuthenticationHandler thread limits
Marvin Addison
Thanks for sharing this. I wasn't aware of this behavior per se, but
it makes sense upon consideration. This is yet another reason to
avoid JNDI-based connection pooling. It has a number of undesirable
behaviors; object-based connection pooling as described on
https://wiki.jasig.org/display/CASUM/LDAP is the way to go.
M
--