Maximum number of executors?

[Jakov Sosic]

Hi guys,

I'm wondering what is the highest number of executors on a single
jenkins master instance that you have seen?

What number of executors is some kind of theoretical maximum for
Jenkins, and are there any information at which number Master gets
overwhelmed?


I'm currently running single master, with 360 executors on 3 slaves, but
am getting executors starvations from time to time.

I was thinking of battling it with dynamic slaves, but I've heard that
it's not very good idea to run more then 100 executors per a single
Jenkins master instance.

What are your thoughts / experiences with this?


Thanks,

Jakov

[Stephen Connolly]

Please take a look at my Jenkins world talk.

All the answers you want.

--

You received this message because you are subscribed to the Google Groups "Jenkins Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/ab321ca7-e682-fc43-53c3-20292194ba7b%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Sent from my phone

[Baptiste Mathus]

Le 10 janv. 2017 12:45 AM, "Jakov Sosic" <jso...@gmail.com> a écrit :

Hi guys,

I'm wondering what is the highest number of executors on a single jenkins master instance that you have seen?

What number of executors is some kind of theoretical maximum for Jenkins, and are there any information at which number Master gets overwhelmed?


I'm currently running single master, with 360 executors on 3 slaves, but am getting executors starvations from time to time.

That, I think, is not good. But is not a Jenkins issue. More engineering and isolation issue: what tasks are you running? 

By experience you should try and target 1 agent=1 executor. But it really depends on tasks you're running.

Bottom line being: not worth the engineering time to diagnose issues when many big builds together at the wrong time will basically kill the whole machine.

Better isolate with dedicated VMs per agent.

I was thinking of battling it with dynamic slaves, but I've heard that it's not very good idea to run more then 100 executors per a single Jenkins master instance.

No. That is wrong. Curious where you heard that. I regularly see masters with hundreds of agents.

One thing that can arise on big configs is the GC misconfiguration (the JVM defaults to parallel GC, which is basically a very unfortunate choice for user facing apps like Jenkins masters, or any webapp really. Only suited for batches...). For example resulting in long freezes from a user standpoint.

We cannot unfortunately improve that where ppl are running the war directly but there's work ongoing to choose better defaults for other packagings (thanks Sam!)

See Stephen's talk, where basically he played with many definitions of "the biggest cluster in the world" :)

What are your thoughts / experiences with this?


Thanks,

Jakov

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscribe@googlegroups.com.

[Jakov Sosic]

On 01/14/2017 11:53 AM, Baptiste Mathus wrote:

> Bottom line being: not worth the engineering time to diagnose issues
> when many big builds together at the wrong time will basically kill the
> whole machine.
> Better isolate with dedicated VMs per agent.

Each executor basically spins new VM via Vagrant/cloud provider - so
slaves are basically offloaded to dynamic VMs.

*BUT* - what's hurting me - biggest issue currently - is that at some
point Jenkins just gets "stuck".

What do I mean by stuck:
- jobs pile up in queue
- queue grows bigger and bigger
- there are 50-100 available executors but jobs from queue
either don't get started or get started after couple of hours...

I haven't pinpointed yet when or why exactly happens, but only solution
is to restart Jenkins master.

> No. That is wrong. Curious where you heard that. I regularly see masters
> with hundreds of agents.

I think Cloudbees, but it's information from third hand so I can't vouch
for it.

> One thing that can arise on big configs is the GC misconfiguration (the
> JVM defaults to parallel GC, which is basically a very unfortunate
> choice for user facing apps like Jenkins masters, or any webapp really.
> Only suited for batches...). For example resulting in long freezes from
> a user standpoint.
> We cannot unfortunately improve that where ppl are running the war
> directly but there's work ongoing to choose better defaults for other
> packagings (thanks Sam!)
>
> See Stephen's talk, where basically he played with many definitions of
> "the biggest cluster in the world" :)

Actually there weren't any major GC shitstorms that we had to endure...

[Chris Overend]

Did you find any useful information I am facing a very similar issue we have to restart jenkins every few days.

We are not sure if it is the number of executors, flows within flows, or large number of builds.