CAS memory leak issue in Production : CAS 6.3.2

[Morning Star]

Hi Team,

We are facing a peculiar issue in production. After CAS deployment we observed that every 6 day once we are getting the below error/exception happening in CAS upgraded environment. 

CAS Server version : CAS 6.3.2

Java version : 11

Apache Tomcat version : 9

2021-04-29 06:10:01 [ERROR] org.apereo.cas.authentication.PolicyBasedAuthenticationManager  [com.ex.sso.authentication.ExLdapHandlerAuthentication]: [failed to create a child event loop / timerfd_create() failed: Too many open files] FatalLog /logs/ex-cas.tomcat/applicationLogs/cas.log

2021-04-29 06:10:01,738 ERROR [org.apereo.cas.authentication.PolicyBasedAuthenticationManager] - [com.ex.sso.authentication.ExLdapHandlerAuthentication]: [failed to create a child event loop / timerfd_create() failed: Too many open files] FatalLog /logs/ex-cas.tomcat/catalina.out

SEVERE [Thread-2759] org.apache.tomcat.util.net.Nio2Endpoint$Nio2Acceptor.failed Socket accept failed java.io.IOException: Too many open files FatalLog /logs/ex-cas.tomcat/catalina.out

CAS servers in production : server1, server2, server3

LDAP connection count from each server : 

server1 : 256, 

server2 : 265, 

server3 : 238

Open file count:

server1 : CNT=1769 PCT=43, 

server2 : CNT=1769 PCT=43, 

server3 : CNT=1677 PCT=40

We are closing LDAP connections manually via code after each authentication.

Java Code:

import org.apereo.cas.util.LdapUtils; //cas-server-support-ldap-core-6.3.2 jar

import org.ldaptive.ConnectionFactory; //ldaptive-2.0.1 jar

private ConnectionFactory searchFactory; 

private AbstractLdapSearchProperties ldapProperties;

@Autowired

private CasConfigurationProperties casProperties;

try{

val ldap = casProperties.getAuthn().getLdap();

this.ldapProperties = ldap.get(0);

this.searchFactory = LdapUtils.newLdaptiveConnectionFactory(ldap.get(0));

val filter = LdapUtils.newLdaptiveSearchFilter(ldapProperties.getSearchFilter(), LdapUtils.LDAP_SEARCH_FILTER_DEFAULT_PARAM_NAME, Collections.singletonList(upc.getUsername()));

searchResponse = LdapUtils.executeSearchOperation(searchFactory, ldapProperties.getBaseDn(), filter, this.ldapProperties.getPageSize());

}

finally

{

if(searchFactory != null)

{

searchFactory.close();

log.info(CLASS_NAME + METHOD_NAME + "Connection Factory Closed Successfully");

}

}

LDAP properties:

# LDAP configurations

cas.authn.ldap[0].order=0

cas.authn.ldap[0].ldapUrl=

cas.authn.ldap[0].type=AUTHENTICATED

cas.authn.ldap[0].useSsl=true

cas.authn.ldap[0].useStartTls=false

cas.authn.ldap[0].connectTimeout=5000

cas.authn.ldap[0].baseDn=

cas.authn.ldap[0].searchFilter=email={user}

cas.authn.ldap[0].subtreeSearch=true

cas.authn.ldap[0].bindDn=

cas.authn.ldap[0].bindCredential=

cas.authn.ldap[0].principalAttributeId=ui

cas.authn.ldap[0].principalAttributePassword=

cas.authn.ldap[0].principalAttributeList=userStatus,tryCount,uid,CN,mail

cas.authn.ldap[0].minPoolSize=3

cas.authn.ldap[0].maxPoolSize=10

cas.authn.ldap[0].validateOnCheckout=true

cas.authn.ldap[0].validatePeriodically=true

cas.authn.ldap[0].validatePeriod=600

cas.authn.ldap[0].failFast=false

cas.authn.ldap[0].idleTime=5000

cas.authn.ldap[0].prunePeriod=5000

cas.authn.ldap[0].blockWaitTime=5000

Can someone help me how to find the memory leak place & how to address this? 

As per the below link, 

Too many LDAP connections with 6.0.5 (google.com)

They asked to add the below property:

cas.authn.ldap[0].providerClass=org.ldaptive.provider.unboundid.UnboundIDProvider

But this property is not available in 6.3.2

Do we have any alternative for this property?

Thanks in advance!

Regards,

Morning Star.

[Daniel Fisher]

On Wed, May 5, 2021 at 1:03 PM Morning Star <anusu...@gmail.com> wrote:

private ConnectionFactory searchFactory; 

private AbstractLdapSearchProperties ldapProperties;

@Autowired

private CasConfigurationProperties casProperties;

try{

val ldap = casProperties.getAuthn().getLdap();

this.ldapProperties = ldap.get(0);

this.searchFactory = LdapUtils.newLdaptiveConnectionFactory(ldap.get(0));

val filter = LdapUtils.newLdaptiveSearchFilter(ldapProperties.getSearchFilter(), LdapUtils.LDAP_SEARCH_FILTER_DEFAULT_PARAM_NAME, Collections.singletonList(upc.getUsername()));

searchResponse = LdapUtils.executeSearchOperation(searchFactory, ldapProperties.getBaseDn(), filter, this.ldapProperties.getPageSize());

}

finally

{

if(searchFactory != null)

{

searchFactory.close();

log.info(CLASS_NAME + METHOD_NAME + "Connection Factory Closed Successfully");

}

}

Without seeing the entire class it's hard to say, but there are probably some thread safety issues here. It's likely new connection factories are getting created before close is getting invoked. I'd recommend initializing searchFactory as part of the bean's initialization. Otherwise you should change searchFactory to be a local variable. (It will be fairly inefficient to create a pooled connection factory for each search operation.)

--Daniel Fisher

[Morning Star]

Hi Daniel,

Thanks for your response.

Please find the attached source code.

[Morning Star]

Can someone help me?