[JIRA] (JENKINS-52374) Issue with unclosed LDAP connections

14 views
Skip to first unread message

thrash.nation@gmail.com (JIRA)

unread,
Jul 4, 2018, 2:16:03 PM7/4/18
to jenkinsc...@googlegroups.com
Felipe Nascimento created an issue
 
Jenkins / Bug JENKINS-52374
Issue with unclosed LDAP connections
Issue Type: Bug Bug
Assignee: Félix Belzunce Arcos
Components: active-directory-plugin
Created: 2018-07-04 18:15
Environment: - Jenkins 2.130
- active-directory-plugin 2.8
- Ubuntu 18.04
- MS Active Directory
Priority: Minor Minor
Reporter: Felipe Nascimento

On our environment we are observing a high number of threads waiting with the following stack:

Thread-169"Thread-169" Id=277 Group=main WAITING on java.lang.Object@64e59be7
	at java...@10.0.1/java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@64e59be7
	at java...@10.0.1/java.lang.Object.wait(Object.java:328)
	at java....@10.0.1/com.sun.jndi.ldap.Connection.pauseReader(Connection.java:771)
	at java....@10.0.1/com.sun.jndi.ldap.Connection.run(Connection.java:911)
	at java...@10.0.1/java.lang.Thread.run(Thread.java:844)

Thread-175"Thread-175" Id=283 Group=main WAITING on java.lang.Object@156aef27
	at java...@10.0.1/java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@156aef27
	at java...@10.0.1/java.lang.Object.wait(Object.java:328)
	at java....@10.0.1/com.sun.jndi.ldap.Connection.pauseReader(Connection.java:771)
	at java....@10.0.1/com.sun.jndi.ldap.Connection.run(Connection.java:911)
	at java...@10.0.1/java.lang.Thread.run(Thread.java:844)

Thread-177"Thread-177" Id=285 Group=main WAITING on java.lang.Object@3df5e68a
	at java...@10.0.1/java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@3df5e68a
	at java...@10.0.1/java.lang.Object.wait(Object.java:328)
	at java....@10.0.1/com.sun.jndi.ldap.Connection.pauseReader(Connection.java:771)
	at java....@10.0.1/com.sun.jndi.ldap.Connection.run(Connection.java:911)
	at java...@10.0.1/java.lang.Thread.run(Thread.java:844)

(dump from http://jenkinsurl/threadDump )

The amount of similar waiting threads increases by 2 on every login and are not being closed leading to a "Too many open files" after a couple days, at that point 378 were waiting with the stack above.

Please let me know if any further information is required to help solve/reproduce this issue.

 
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396)

thrash.nation@gmail.com (JIRA)

unread,
Jul 4, 2018, 2:37:02 PM7/4/18
to jenkinsc...@googlegroups.com
Felipe Nascimento updated an issue
Change By: Felipe Nascimento
Environment: - Jenkins 2.130
- active-directory-plugin 2.8 / STARTTLS enabled
- Ubuntu 18.04
- MS Active Directory

thrash.nation@gmail.com (JIRA)

unread,
Jul 4, 2018, 2:45:01 PM7/4/18
to jenkinsc...@googlegroups.com
Felipe Nascimento commented on Bug JENKINS-52374
 
Re: Issue with unclosed LDAP connections

Important hint: disabling the START TLS option makes the problem go away

jeremyc2010@gmail.com (JIRA)

unread,
Jul 1, 2019, 1:00:02 PM7/1/19
to jenkinsc...@googlegroups.com
Jeremy Cornett updated an issue
 
Change By: Jeremy Cornett
Labels: java11-compatibility
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

jeremyc2010@gmail.com (JIRA)

unread,
Jul 1, 2019, 1:03:02 PM7/1/19
to jenkinsc...@googlegroups.com
Jeremy Cornett commented on Bug JENKINS-52374
 
Re: Issue with unclosed LDAP connections

I am able to duplicate this issue running Jenkins via docker 2.176.1-jdk11 with active-directory:2.16. 

jeremyc2010@gmail.com (JIRA)

unread,
Jul 1, 2019, 1:06:02 PM7/1/19
to jenkinsc...@googlegroups.com
Jeremy Cornett edited a comment on Bug JENKINS-52374
I am able to duplicate this issue running Jenkins via docker 2.176.1-jdk11 with active-directory:2.16.   This issue brings my Jenkins instance to a standstill with 8 to 48 hours, causing the UI to become unresponsive, and eventually give an out of memory exception due to the number of threads and open files. I am exploring three options unless the plugin can be fixed...
# Switch to LDAPS.


# Just live with having insecure LDAP connections for AD authentication on our Jenkins servers.


# Downgrade the Jenkins master and build nodes to use Java 8 instead of Java 11.

jeremyc2010@gmail.com (JIRA)

unread,
Jul 1, 2019, 1:07:02 PM7/1/19
to jenkinsc...@googlegroups.com
Jeremy Cornett edited a comment on Bug JENKINS-52374
I am able to duplicate this issue running Jenkins via docker 2.176.1-jdk11 with active-directory:2.16. This issue brings my Jenkins instance to a standstill with 8 to 48 hours, causing the UI to become unresponsive, and eventually give an out of memory exception due to the number of threads and open files. I am exploring three options unless the plugin can be fixed...
# Switch to LDAPS.
# Just live with having insecure LDAP connections for AD authentication on our Jenkins servers (i . e. disable StartTLS permanently).


# Downgrade the Jenkins master and build nodes to use Java 8 instead of Java 11.

bmathus+ossjira@cloudbees.com (JIRA)

unread,
Jul 3, 2019, 4:25:02 AM7/3/19
to jenkinsc...@googlegroups.com

Jeremy Cornett are you saying you confirmed this issue does not happen on Java 8?
Could you please provide the memory settings you are using (or the image in use, if the defaults). Thanks!

jeremyc2010@gmail.com (JIRA)

unread,
Jul 3, 2019, 10:58:03 AM7/3/19
to jenkinsc...@googlegroups.com

Yes, I can confirm this was working on Java 8. Specifically, on 6/4/2019, I upgrade our Jenkins instance from docker jenkins/jenkins:2.164.1 to jenkins/jenkins:2.164.1-jdk11 with active-directory:2.8 to active-directory:2.13. Immediately thereafter, our Jenkins instance started crashing. It took a number of weeks for me diagnose this problem properly, and I subsequently tried upgrading to newer versions of Jenkins and the active-directory plugin. We are now using docker jenkins/jenkins:2.176.1-jdk11 and active-directory:2.16. I finally resolved the issue in our instance by abandoning StartTLS and using LDAPS, as outlined in the plugin documentation.

Memory settings, the VM I was using had 2 cores and 12 GB of RAM initially. I thought the problem was a memory issue, so on 6/11/2019, I changed the VM to 4 cores and 16 GB of RAM, but that didn't make a discernible difference. I found that when I monitored top on the VM (CentOS 7.6), memory usage would never go above 4 GB of RAM, but the virtual memory would grow and grow over time. The highest I saw VIRT was about 28 GB.

Eventually, I installed the plugin monitoring:1.77.0. This allowed me to see the number of open files and threads, which also allowed me to see threads were Waiting and had similar information as what is on this ticket. When I then disabled StartTLS, the symptom went away completely.

egutierrez@cloudbees.com (JIRA)

unread,
Jul 8, 2019, 4:18:02 AM7/8/19
to jenkinsc...@googlegroups.com
Evaristo Gutierrez updated an issue
 
Change By: Evaristo Gutierrez
Labels: java11-compatibility triaged

nick@ridgworld.com (JIRA)

unread,
Mar 11, 2020, 10:17:03 AM3/11/20
to jenkinsc...@googlegroups.com
Nick Ridgway commented on Bug JENKINS-52374
 
Re: Issue with unclosed LDAP connections

I am also seeing this issue.  Switching STARTTLS off isn't an option for me, is there any information I can provide to help diagnose/fix the issue? 
This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38)
Atlassian logo

oliver.grad@gmx.de (JIRA)

unread,
Mar 18, 2020, 5:19:07 AM3/18/20
to jenkinsc...@googlegroups.com
Oliver Grad updated an issue
 
Change By: Oliver Grad
Attachment: jenkins_file_handles.jpg

oliver.grad@gmx.de (JIRA)

unread,
Mar 18, 2020, 5:34:03 AM3/18/20
to jenkinsc...@googlegroups.com
Oliver Grad commented on Bug JENKINS-52374
 
Re: Issue with unclosed LDAP connections

We discovered the same issue. On 3/10/2020 we upgraded our productive Jenkins (2.204.5) from Java 8 to Java 11 and the missbehaviour started immediatly.

We use AD-Plugin version 2.16.
As soon as we disabled StartTLS the problem disappeared (on the right side of the graph).
Reply all
Reply to author
Forward
0 new messages