Disabling the ping thread in Jenkins

[Jan Lutenko]

Hi,

I'm not really familiar how this works internally, but looking at thread dump in Jenkins I can not interpret this
in any other way than the ping thread is terribly slowing down Jenkins by causing other threads to wait for it:

Short background: Jenkins 1580.3, running on x64 RHEL with 4 cores and 24 G RAM. Not the only instance on master host, other seem to be feeling better, I suspect this pinging thread to be the cause of long time page loading.

Not listing any further info not to overload someone who's reading this with useless information.. At the moment.

Ping thread for channel hudson.remoting.Channel@32a6cf5b:<host>


"Ping thread for channel hudson.remoting.Channel@32a6cf5b:<host>" Id=108 Group=main TIMED_WAITING

at java.lang.Thread.sleep(Native Method)

at hudson.remoting.PingThread.run(PingThread.java:91)




The rest of thread dump (specifically jobs) looks more or less like this:

Executor #6 for <host> : executing <job> #<build_number> / waiting for hudson.remoting.Channel@32a6cf5b:<host>


"Executor #6 for eselivm2v759l : executing mct_up_check #2004 / waiting for hudson.remoting.Channel@32a6cf5b:<host>" Id=625627 Group=main TIMED_WAITING


on hudson.remoting.UserRequest@7654c0aa


at java.lang.Object.wait(Native Method) - waiting on hudson.remoting.UserRequest@7654c0aa


at hudson.remoting.Request.call(Request.java:146)


at hudson.remoting.Channel.call(Channel.java:751)


at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:173)


at com.sun.proxy.$Proxy56.join(Unknown Source)

at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:979)

at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:137)


at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:97)


at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)


at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)


at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:770)


at hudson.model.Build$BuildExecution.build(Build.java:199)


at hudson.model.Build$BuildExecution.doRun(Build.java:160)


at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:533)


at hudson.model.Run.execute(Run.java:1745)


at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)


at hudson.model.ResourceController.execute(ResourceController.java:89)


at hudson.model.Executor.run(Executor.java:240)

Can someone please confirm that my logic here is right/wrong? Also, how can I disable the ping thread?
I know that adding -Dhudson.remoting.Launcher.pingIntervalSec=-1 to slave call and a similar
option to master start up can stop the thread from pinging, but will it be turned off then?

Any help on this would be very much appreciated as I am stumbling on this issue not for the first time and hitting a wall as well again.

Thank you and Best Regards,
Jan

[milki milk]

Have you tried the suggestions here: https://wiki.jenkins-ci.org/display/JENKINS/Ping+Thread

[James Nord]

in any other way than the ping thread is terribly slowing down Jenkins by causing other threads to wait for it:

 No the ping thread is sleeping.  The executor thread is waiting on the channel to give it more data.

- you got a little confused due to the fact that the ping thread has the name of the channel Object in its threads name (which is the same Object that the executor is wating for...)

/James

[Jan Lutenko]

Hi,

James, thank you for explanation! I fugured that too...
Won't make this misinterpretation in the future now.
Also, that'd be very bad desing and I doubt that community would just let this in.

So the issue is some place else. I wonder if the instance has too many Jenkins masters running at once...
Other do not experience this slowness though

Anyway, thanks for help, guys!

Cheers,

Jan

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/eQTbvl1UPCw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/25dfbc23-8313-48a3-8616-a9b4033917f3%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Z poważaniem,

Jan Lutenko

[Stephen Connolly]

If you are running 1.580.x then I suspect it is lock contention with
the Queue that is causing the slow page loads.

1.609.x should be much better in that regard... but you may want to
hold back until 1.625.1 as there are some deadlocks that might affect
you in 1.609.1 and 1.609.2 (1.609.3 has most of them fixed, but one
with the old data monitor missed the boat and is in 1.625.1)

> You received this message because you are subscribed to the Google Groups
> "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to jenkinsci-use...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/jenkinsci-users/CADzPo267w7xERZD38UZzReT%3DcL-KBOWET3%3DvUDRT%2BHF7EycmBA%40mail.gmail.com.

[Jan Lutenko]

HI Stephen,

That might be useful. I'll see if it's possible to upgrade to 1609.3 for me.

Is Issue 27566 the cause for this in 1580 + ?

Thanks and BR,

Jan

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CA%2BnPnMy28c4BDZH5F_QwsBKOB_CW12YEr%3DJ2F2Gr8h1%2BszKD4g%40mail.gmail.com.

[Stephen Connolly]

It's one of the issues. Another is the big fat single lock in OldDataMonitor (you need to wait for 1.625.1 for that fix though)

We are really in chase out the bottlenecks mode as we improve the multithreading. Eventually we will make Queue lock free but it's not the current bottleneck so doesn't warrent the extra complexity yet

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CADzPo27UJ_RgAVU7svpdrTk9w6H%3DoV180LfSyQ_sBgd4ubEVWw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
Sent from my phone