[JIRA] (JENKINS-38815) ForemanSharedNodeCloud.canProvision(...) has to use a caching mechanism

0 views
Skip to first unread message

pjanouse@redhat.com (JIRA)

unread,
Oct 7, 2016, 4:58:01 AM10/7/16
to jenkinsc...@googlegroups.com
Pavel Janoušek updated an issue
 
Jenkins / Improvement JENKINS-38815
ForemanSharedNodeCloud.canProvision(...) has to use a caching mechanism
Change By: Pavel Janoušek
Summary: ForemanSharedNodeCloud.canProvision(...) has to use a caching mechanism
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo

pjanouse@redhat.com (JIRA)

unread,
Oct 7, 2016, 9:27:20 AM10/7/16
to jenkinsc...@googlegroups.com
Pavel Janoušek updated an issue
{{ForemanSharedNodeCloud.canProvision(Label l)}} has to use some caching mechanism. It is utilized very often by Jenkins core, AJAX etc. not only if Jenkins master is looking for where it can schedule a run from the queue but for many other reasons.

Typical time spent in this method in our testing environment is 5-30 seconds, so if you get threaddump you can be pretty sure you can see at least one thread ("jenkins.util.Timer") in it every time.

One simple approach can be to obtain all possible labels from the Foreman, cache it and use it for let say couple of seconds/minutes etc. and then drop it and obtain it again. Caching time can be configurable (lower value for development, higher value for production) and it can improve current state much.

In addition there are several path to {{ForemanSharedNodeCloud.canProvision()}} from the core where {{hudson.model.Hudson}} or {{hudson.model.Queue}} is locked before, so Jenkins is effectively blocked by this for couple of seconds which can be a big issue in a production deployment.

You can monitor that behavior simply by using Dumpling plugin like:{code}
import com.github.olivergondza.dumpling.query.*;
Dumpling.runtime.threads.grep {it.thread.stackTrace.toString().contains("foreman")}
{code}

scoheb@gmail.com (JIRA)

unread,
Oct 7, 2016, 9:53:01 AM10/7/16
to jenkinsc...@googlegroups.com
Scott Hebert started work on Improvement JENKINS-38815
 
Change By: Scott Hebert
Status: Open In Progress

scoheb@gmail.com (JIRA)

unread,
Oct 7, 2016, 10:17:06 AM10/7/16
to jenkinsc...@googlegroups.com
Scott Hebert commented on Improvement JENKINS-38815
 
Re: ForemanSharedNodeCloud.canProvision(...) has to use a caching mechanism

I was able to reproduce this very easily just by creating a simple Node...

"Handling POST /computer/createItem from 10.3.116.222 : RequestHandlerThread[#19]" #96 daemon prio=5
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
	- locked <0x29cede97> (a java.io.BufferedInputStream)
	at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704)
	at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1536)
	- locked <0x5a26222c> (a sun.net.www.protocol.http.HttpURLConnection)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
	- locked <0x5a26222c> (a sun.net.www.protocol.http.HttpURLConnection)
	at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
	at org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:394)
	at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:285)
	at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:255)
	at org.glassfish.jersey.client.JerseyInvocation$1.call(JerseyInvocation.java:684)
	at org.glassfish.jersey.client.JerseyInvocation$1.call(JerseyInvocation.java:681)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:444)
	at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:681)
	at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:411)
	at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:311)
	at com.redhat.foreman.ForemanAPI.getForemanResponse(ForemanAPI.java:153)
	at com.redhat.foreman.ForemanAPI.getHostParameterValue(ForemanAPI.java:195)
	at com.redhat.foreman.ForemanAPI.getHostForQuery(ForemanAPI.java:309)
	at com.redhat.foreman.ForemanAPI.getCompatibleHosts(ForemanAPI.java:327)
	at com.redhat.foreman.ForemanSharedNodeCloud.canProvision(ForemanSharedNodeCloud.java:158)
	at hudson.model.Label.getClouds(Label.java:227)
	at hudson.model.Label.isEmpty(Label.java:435)
	at jenkins.model.Jenkins.trimLabels(Jenkins.java:1899)
	at jenkins.model.Nodes$2.run(Nodes.java:137)
	at hudson.model.Queue._withLock(Queue.java:1320)
	at hudson.model.Queue.withLock(Queue.java:1197)
	at jenkins.model.Nodes.addNode(Nodes.java:132)
	at jenkins.model.Jenkins.addNode(Jenkins.java:1853)
	at hudson.model.ComputerSet.doCreateItem(ComputerSet.java:262)
	- locked <0x6054fcd8> (a hudson.model.ComputerSet)

The Foreman host does seem slow today...luckily so...it uncovered this problem...

scm_issue_link@java.net (JIRA)

unread,
Oct 14, 2016, 8:44:01 AM10/14/16
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Scott Hebert
Path:
pom.xml
src/main/java/com/redhat/foreman/ForemanAPI.java
src/main/java/com/redhat/foreman/ForemanSharedNodeCloud.java
src/test/java/com/redhat/foreman/ForemanSharedNodeCloudTest.java
http://jenkins-ci.org/commit/foreman-node-sharing-plugin/bad8752d42d2d1f152f2459fe5ac4ed7b0b4d8b8
Log:
Add Hosts caching for canProvision()

[FIXES JENKINS-38815]

scm_issue_link@java.net (JIRA)

unread,
Oct 14, 2016, 8:44:02 AM10/14/16
to jenkinsc...@googlegroups.com
SCM/JIRA link daemon resolved as Fixed
 
Change By: SCM/JIRA link daemon
Status: In Progress Resolved
Resolution: Fixed
Reply all
Reply to author
Forward
0 new messages