How can I add builds to the queue a little at a time?

[Chris Overend]

I am beginning the process of changing to pipelines.

We currently use flows to run thousands of jobs(builds).

Say we have 6000 builds to run on 100 Windows virtual machines and 6000 builds to run on 100 Linux virtual machines.

To reduce available virtual machines with no jobs to run we try to keep the queue as full as possible.

Each job takes between 1 - 20 min to execute. 

I need a way to not over load Jenkins. 

With our current workflow we have to restart Jenkins every couple of days as it becomes sluggish (we just purchased and are using a new server). 

We have not determined if Flows with Flows is a contributing factor but are beginning testing with Pipelines next week.

We believe the number of jobs in the queue is the cause of our problems, but we have not found a better solution than having that many jobs.

My thought was to try and trickle jobs to the queue. I would like to add a new job to the queue only if it drops below say 1000.

I am open to other suggestions.

[Baptiste Mathus]

Hello,

The JVM has cool standard features that help you at least know what the hell it's doing. IMO that's the first thing to do before acting. 

You also for instance want to know if really your system is overloaded or not before possibly growing it, and risking a stab in the dark.

Basically start with vmstat. Especially look for system consistently swapping. Having a swapping/thrashing jvm/system is indeed going to be very sluggish.

Then you may want to look at thread dumps and possibly heap dump depending on some factors.

If you don't feel like you can do that, with my corp cap on, be aware that some companies like mine do offer professional support for Jenkins. With your apparent load, it would probably not seem wasteful to be backed by Jenkins experts.

HTH

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/ca93b1a3-bf6c-4a0d-b41e-60ebf1d2d36d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chris Overend]

Thank you for your insights.

This reinforces my thoughts after reading https://jenkins.io/blog/2016/11/21/gc-tuning/