Large number of "ESTABLISHED" LDAP connections initialted by jenkins

182 views
Skip to first unread message

Rumesh Bandara

unread,
Nov 10, 2016, 4:48:10 AM11/10/16
to Jenkins Users
Hi All,

Our Jenkins instance is making a large number of "ESTABLISHED" connections to ldap server which cause higher cpu usage of ldap instance. Do you have any clue about what could be the issue from Jenkins?

Jenkins ver. 2.7.2

Thanks,
Rumesh

Paxton, Darren

unread,
Nov 10, 2016, 5:16:54 AM11/10/16
to jenkins...@googlegroups.com

Suspect you need to provide a lot more information such as what are you using LDAP for,purely authentication? Do the LDAP logs indicate what the connections could be?

 

More info about what you’ve done to troubleshoot yourself.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/d1c81074-c421-4b69-81b8-0d66dbd7c4dc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Rumesh Bandara

unread,
Nov 11, 2016, 1:34:24 AM11/11/16
to Jenkins Users, Darren...@mercer.com
I use LDAP for authentication. I ran ldap in debug mode and could see Jenkins continuously search for all the users in an unexpected manner which causing server to consume almost 100% cpu.
Meanwhile, Jenkins logs indicate LDAP response timeouts as attached below.

When I stop the Jenkins server, LDAP does not consume considerable amount of CPU.

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
 2825 openldap  20   0  917256  94672   7044 S 99.5  9.3   0:12.48 slapd

openldap sample debug logs

582558ca => access_allowed: search access to "uid=testuser,ou=user,dc=ldap,dc=domain,dc=org" "mail" requested

582558ca => dn: [2] ou=group,dc=ldap,dc=domain,dc=org

582558ca => dn: [8]

582558ca => acl_get: [9] attr mail

582558ca => acl_mask: access to entry "uid=testuser,ou=user,dc=ldap,dc=domain,dc=org", attr "mail" requested

582558ca => acl_mask: to value by "uid=ldapbinduser,ou=user,dc=ldap,dc=domain,dc=org", (=0)

582558ca <= check a_dn_pat: uid=ldapbinduser,ou=user,dc=ldap,dc=domain,dc=org

582558ca <= acl_mask: [2] applying read(=rscxd) (stop)

582558ca <= acl_mask: [2] mask: read(=rscxd)

582558ca => slap_access_allowed: search access granted by read(=rscxd)

582558ca => access_allowed: search access granted by read(=rscxd)

582558ca <= test_filter 5

582558ca bdb_search: 648 does not match filter

582558ca => test_filter

582558ca     EQUALITY


Jenkins logs

Nov 11, 2016 11:07:04 AM hudson.security.LDAPSecurityRealm$LDAPUserDetailsService loadUserByUsername

WARNING: Failed to search LDAP for username=someone

org.acegisecurity.ldap.LdapDataAccessException: LdapCallback;LDAP response read timed out, timeout used:60000ms.; nested exception is javax.naming.NamingException: LDAP response read timed out, timeout used:60000ms.; remaining name 'ou=user'

at org.acegisecurity.ldap.LdapTemplate$LdapExceptionTranslator.translate(LdapTemplate.java:295)

at org.acegisecurity.ldap.LdapTemplate.execute(LdapTemplate.java:128)

at org.acegisecurity.ldap.LdapTemplate.searchForSingleEntry(LdapTemplate.java:246)

at org.acegisecurity.ldap.search.FilterBasedLdapUserSearch.searchForUser(FilterBasedLdapUserSearch.java:119)

at hudson.security.LDAPSecurityRealm$LDAPUserDetailsService.loadUserByUsername(LDAPSecurityRealm.java:708)

at hudson.security.LDAPSecurityRealm$LDAPUserDetailsService.loadUserByUsername(LDAPSecurityRealm.java:670)

at hudson.security.LDAPSecurityRealm.loadUserByUsername(LDAPSecurityRealm.java:572)

at hudson.model.User$UserIDCanonicalIdResolver.resolveCanonicalId(User.java:1050)

at hudson.model.User.get(User.java:395)

at hudson.model.User.get(User.java:364)

at hudson.plugins.git.GitChangeSet.findOrCreateUser(GitChangeSet.java:288)

at hudson.plugins.git.GitChangeSet.getAuthor(GitChangeSet.java:349)

at hudson.model.AbstractBuild.hasParticipant(AbstractBuild.java:392)

at hudson.model.User.relatedTo(User.java:626)

at hudson.model.User.doRssLatest(User.java:820)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:324)

at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:167)

at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:100)

at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:124)

at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)

at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)

at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)

at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:233)

at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)

at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)

at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)

at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)

at org.kohsuke.stapler.Stapler.service(Stapler.java:238)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)

at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:89)

at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)

at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:198)

at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:176)

at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:85)

at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:99)

at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)

at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:58)

at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)

at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)

at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)

at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)

at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)

at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135)

at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)

at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)

at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)

at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)

at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)

at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)

at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:611)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)

at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:409)

at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1044)

at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)

at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:744)

Caused by: javax.naming.NamingException: LDAP response read timed out, timeout used:60000ms.; remaining name 'ou=user'

at com.sun.jndi.ldap.Connection.readReply(Connection.java:483)

at com.sun.jndi.ldap.LdapClient.getSearchReply(LdapClient.java:639)

at com.sun.jndi.ldap.LdapClient.search(LdapClient.java:562)

at com.sun.jndi.ldap.LdapCtx.doSearch(LdapCtx.java:1985)

at com.sun.jndi.ldap.LdapCtx.searchAux(LdapCtx.java:1847)

at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1772)

at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1789)

at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(ComponentDirContext.java:412)

at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:394)

at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:376)

at javax.naming.directory.InitialDirContext.search(InitialDirContext.java:286)

at org.acegisecurity.ldap.LdapTemplate$3.doInDirContext(LdapTemplate.java:249)

at org.acegisecurity.ldap.LdapTemplate.execute(LdapTemplate.java:126)

... 95 more

Reply all
Reply to author
Forward
0 new messages