[JIRA] [core] (JENKINS-24155) Jenkins Slaves Go Offline In Large Quantities and Don't Reconnect Until Reboot

[vladimir.lazarev@intel.com (JIRA)]

Vladimir Lazarev commented on JENKINS-24155

Re: Jenkins Slaves Go Offline In Large Quantities and Don't Reconnect Until Reboot We have "peer reconnecting" issue ones per 2 weeks. After applying the WA proposed by Amal connections with crashing during heavy jobs (high CPU load and long duration) on regular basis. So be aware... Add Comment

This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)

[vladimir.lazarev@intel.com (JIRA)]

Vladimir Lazarev edited a comment on JENKINS-24155

Re: Jenkins Slaves Go Offline In Large Quantities and Don't Reconnect Until Reboot

We have "peer reconnecting" issue  ones per 2 weeks. After applying the WA proposed by Amal connections  with  are  crashing during heavy jobs (high CPU load and long duration) on regular basis. So be aware...

Add Comment

[stephenconnolly@java.net (JIRA)]

stephenconnolly commented on JENKINS-24155

Re: Jenkins Slaves Go Offline In Large Quantities and Don't Reconnect Until Reboot

This is almost certainly the same issue as JENKINS-25218

Add Comment

[sgannon200@gmail.com (JIRA)]

Shane Gannon commented on JENKINS-24155

Re: Jenkins Slaves Go Offline In Large Quantities and Don't Reconnect Until Reboot

I've been told that this issue is the same as JENKINS-28844 and has been resolved in the 1.609.3 LTS.

Add Comment

[mohittater@bluejeansnet.com (JIRA)]

mohit tater commented on JENKINS-24155

Re: Jenkins Slaves Go Offline In Large Quantities and Don't Reconnect Until Reboot

We are facing this issue on Jenkins ver. 1.605. On most of the offline slaves I am seeing: "JNLP agent connected from /x.y.z.a" in the node log. Here is the threadDump link of the affected Jenkins instance. http://pastebin.com/9hUR1Awf Please provide a temporary workaround for this so that it can be avoided in future. Note: We are using 50+ nodes on a single master.

Add Comment

[o.v.nenashev@gmail.com (JIRA)]

Oleg Nenashev updated an issue

Jenkins / JENKINS-24155

Jenkins Slaves Go Offline In Large Quantities and Don't Reconnect Until Reboot

Change By: Oleg Nenashev Component/s: remoting

Add Comment

This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)

[aubert.aa@gmail.com (JIRA)]

Alexandre Aubert updated an issue

Jenkins / JENKINS-24155 Jenkins Slaves Go Offline In Large Quantities and Don't Reconnect Until Reboot

Change By: Alexandre Aubert Attachment: log.txt

Add Comment

[aubert.aa@gmail.com (JIRA)]

Alexandre Aubert commented on JENKINS-24155

Re: Jenkins Slaves Go Offline In Large Quantities and Don't Reconnect Until Reboot same problem since several days with jenkins 2.23, here is the extract of log with : first 'outofmemory' error then all 'java.lang.OutOfMemoryError: unable to create new native thread' then 'disconnection of all slaves' log.txt Hope this could help.

Add Comment

[aubert.aa@gmail.com (JIRA)]

Alexandre Aubert edited a comment on JENKINS-24155

Re: Jenkins Slaves Go Offline In Large Quantities and Don't Reconnect Until Reboot

same problem since several days with jenkins 2.23, here is the extract of log with :

- first 'outofmemory' error then - all 'java.lang.OutOfMemoryError: unable to create new native thread' - then 'disconnection of all slaves' [^log.txt] 2 slaves are not disconnected : slave.jar is more recent on those. I will update slave.jar on all and check if it happens again.... (waiting also the autoupdate of slave.jar files which is pending in another ticket....) Hope this could help.

Add Comment

[aubert.aa@gmail.com (JIRA)]

Alexandre Aubert commented on JENKINS-24155

Re: Jenkins Slaves Go Offline In Large Quantities and Don't Reconnect Until Reboot

In my case this was a outofmemory problem : i fixed it by increasing the -Xmx in jenkins args and all seems to be ok since.

Add Comment

[trushar.scm@gmail.com (JIRA)]

Trushar Patel commented on JENKINS-24155

Re: Jenkins Slaves Go Offline In Large Quantities and Don't Reconnect Until Reboot

We are also facing the same issue on Jenkins 1.624. I had to reboot it. Please someone help. This looks like its been going on for while.

Add Comment

[nelu.vasilica@cirrus.com (JIRA)]

Nelu Vasilica commented on JENKINS-24155

Re: Jenkins Slaves Go Offline In Large Quantities and Don't Reconnect Until Reboot

Just seen the same issue on Jenkins 1.642.1 Linux master. the fix was to restart tomcat and the windows slaves reconnected automatically. Found several instances of: Ping started at xxxxxx hasn't completed by xxxxxxx in the logs. Is setting jenkins.slaves.NioChannelSelector.disabled property to true a viable workaround?

Add Comment

[hailcesos@gmail.com (JIRA)]

Cesos Barbarino commented on JENKINS-24155

Re: Jenkins Slaves Go Offline In Large Quantities and Don't Reconnect Until Reboot

Same issue here. Only this time, my tests never get done. The slaves area always dropping during the tests please halp!!

Add Comment

[l.heche@hotmail.fr (JIRA)]

Louis Heche updated an issue

Jenkins / JENKINS-24155

Jenkins Slaves Go Offline In Large Quantities and Don't Reconnect Until Reboot

Change By: Louis Heche Attachment: masterJenkins.log Attachment: jenkins-slave.0.err.log

Add Comment

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

[l.heche@hotmail.fr (JIRA)]

Louis Heche commented on JENKINS-24155

Re: Jenkins Slaves Go Offline In Large Quantities and Don't Reconnect Until Reboot I'm having what seems to be this issue with Jenkins 2.138.3. Every 3-4 days all the slaves node go offline although it seems to have no network problem. They return online once the master has been restarted.  In attachment you'll find the logs jenkins-slave.0.err.logmasterJenkins.log

Add Comment

[jenkins2jrw@nym.hush.com (JIRA)]

Jeremy Whiting commented on JENKINS-24155

Re: Jenkins Slaves Go Offline In Large Quantities and Don't Reconnect Until Reboot

Louis Heche Oleg Nenashev Cesos Barbarino  Can one of you do the following. To help narrow down the possible leak areas it will be useful to capture process memory usage and JVM heap usage. Start your master process as normal. Then start 2 tools on the system and redirect the output to separate files. Both tools have low system resource usage.  Memory stats can be captured using pidstat. Specifically to capture resident set size. $ pidstat -r -p <pid> 8 > /tmp/pidstat-capture.txt  JVM heap size and GC behavior. Specifically the percentage of reclaimed heap space after a full collection. $ jstat -gcutil -t -h12 <pid> 8s > /tmp/jstat-capture.txt Please attach the generated files to this issue.

Add Comment