[JIRA] [ec2-plugin] (JENKINS-25832) Launch multiple slaves in parallel for jobs with same node label

[leonid.beder@gmail.com (JIRA)]

Leonid Beder commented on JENKINS-25832

Re: Launch multiple slaves in parallel for jobs with same node label Hi, I'm experiencing exactly the same issue, which prevents me from fully parallelizing my builds. Is there an expected fix or a work around to this issue? I've tried the above, but nothing worked: 1. Adding another label restriction to the job and adding another AMI with this label - still, the every consequent job was queued. 2. Adding another label restriction to the job and adding the label as another label to the same AMI. I'd really appreciate your help, Leonid Add Comment

This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)

[me@joshma.com (JIRA)]

Joshua Ma commented on JENKINS-25832

Re: Launch multiple slaves in parallel for jobs with same node label

Agreed! Right now I'm running a few permanent slaves to help with this, but it'd be great if the EC2 plugin's response scaled with the # of pending builds. Otherwise, it seems like it'll wait a few minutes to determine a new machine is needed, wait a few minutes to determine a new machine is needed, and so on.

Add Comment

[krismassey@gmail.com (JIRA)]

Kris Massey commented on JENKINS-25832

Re: Launch multiple slaves in parallel for jobs with same node label

We're being hit by the same issue. Could this be made a configuration option? I can see why someone people may want to start nodes sequentially to fully ensure the node limit is not exceeded...however when you've got 10+ slaves, having to wait for each slave to start sequentially is extreamly slow

Add Comment

[vthakur@splunk.com (JIRA)]

Vikas Thakur commented on JENKINS-25832

Re: Launch multiple slaves in parallel for jobs with same node label

We were facing the same issue earlier and then we tuned our jam with these 2 settings -Dhudson.model.LoadStatistics.decay=0.1 -Dhudson.model.LoadStatistics.clock=1000 and then this issue was gone

Add Comment

[maxdrib@gmail.com (JIRA)]

Maxwell Dribinsky commented on JENKINS-25832

Re: Launch multiple slaves in parallel for jobs with same node label

Vikas, How did you go about changing the Dhudson.model.LoadStatistics.decay and Dhudson.model.LoadStatistics.clock parameters? Does this require a change in the source code, or is there an easier way to set it?

Add Comment

This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)

[vthakur@splunk.com (JIRA)]

Vikas Thakur commented on JENKINS-25832

Re: Launch multiple slaves in parallel for jobs with same node label

These are JVM settings. This doesnt need code change

Add Comment

[cheecheeo@gmail.com (JIRA)]

J C commented on JENKINS-25832

Re: Launch multiple slaves in parallel for jobs with same node label

To add my experience: modifying the LoadStatistics parameters did not help for bursty queues. Looking at the (quite complicated) LoadStatistics code, maybe these parameters help over a longer period of time. I ended up writing a shell script that uses Jenkins' REST api to look at the queue length, the number of offline agents, and then provisions nodes immediately.

Add Comment

[maxdrib@gmail.com (JIRA)]

Maxwell Dribinsky commented on JENKINS-25832

Re: Launch multiple slaves in parallel for jobs with same node label

Thank you all. Changing the JVM parameters worked for me. For future reference, instructions on changing JVM's for Jenkins can be found here http://stackoverflow.com/questions/14762162/how-do-i-give-jenkins-more-heap-space-when-its-running-as-a-daemon-on-ubuntu. Now if there are multiple jobs in the queue, Jenkins will boot up a new instance, and when it boots, it will re-evaluate the queue. If more jobs remain, it will instantly boot up another instance.

Add Comment

[vthakur@splunk.com (JIRA)]

Vikas Thakur resolved as Fixed

Jenkins / JENKINS-25832

Launch multiple slaves in parallel for jobs with same node label

Change By: Vikas Thakur Status: Open Resolved Resolution: Fixed

Add Comment

[maxdrib@gmail.com (JIRA)]

Maxwell Dribinsky edited a comment on JENKINS-25832

Re: Launch multiple slaves in parallel for jobs with same node label

Thank you all. Changing the JVM parameters worked for me. For future reference, instructions on changing JVM's for Jenkins can be found here http://stackoverflow.com/questions/14762162/how-do-i-give-jenkins-more-heap-space-when-its-running-as-a-daemon-on-ubuntu. Now if there are multiple jobs in the queue, Jenkins will boot up a new instance, and when it boots, it will re-evaluate the queue. If more jobs remain, it will instantly boot up another instance.

I would also like to include that the link below suggests NOT using LoadStatistics.clock values of less than 3000. I was able to launch new slaves within ~15 seconds using a clock value of 3000 https://cloudbees.zendesk.com/hc/en-us/articles/204690520-Why-do-slaves-show-as-suspended-while-jobs-wait-in-the-queue-

Add Comment

[maxdrib@gmail.com (JIRA)]

Maxwell Dribinsky reopened an issue

Workaround doesn't launch slaves for each job in queue at once. It still launches one at a time, but slightly faster. The original issue persists.

Jenkins / JENKINS-25832

Launch multiple slaves in parallel for jobs with same node label

Change By: Maxwell Dribinsky Resolution: Fixed Status: Resolved Reopened

Add Comment

[vintrojan@gmail.com (JIRA)]

Vinay Sharma commented on JENKINS-25832

Re: Launch multiple slaves in parallel for jobs with same node label Vikas Thakur Hi, Regarding the JVM settings : -Dhudson.model.LoadStatistics.decay=0.1 -Dhudson.model.LoadStatistics.clock=3000 Did u add these JVM settings in the jenkins.xml file or the EC2 plugin (textbox) in Jenkins ?

Add Comment

[raihaan.shouhell@autodesk.com (JIRA)]

Raihaan Shouhell resolved as Fixed

This has already been fixed in 1.40

Jenkins / JENKINS-25832

Launch multiple slaves in parallel for jobs with same node label

Change By: Raihaan Shouhell Status: Reopened Resolved Resolution: Fixed

Add Comment

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)