What are reasons for high load on Jenkins master?

[Dominik]

Hi,

We are running Jenkins with about 20 build agents on separate VMs (with up to 4 executors each). The master has 4 CPUs, so I would expect a load of about 4 when there's "a lot of work" to be done.

However, no job is ever scheduled on the master itself, and still we can observe very high load levels on the master when many jobs are running concurrently on the agents. For example, one of our test suites launches about 30 jobs (lasting 5-20min each) in a 1h time slot and at that moment the master reaches a load of up to 40, with all CPUs fully occupied! However, looking at the UI, all jobs clearly run on the agents, and the master simply shows as 'Idle'.

When I look at Jenkins' own monitoring, I see a huge amount of Computer.threadPoolForRemoting and similar threads with a summary stating Threads on master: Number = 415, Maximum = 511, Total started = 7,731,717.

Can anyone give some insight on what could be going on? What can be the reasons for high load on the master when all the jobs are running on agents?

Regards,
Dominik

[Kamil Dzierzanowski]

Hi Dominik,

High load on the master might be caused by incorrectly implemented Jenkinsfiles which contain complex logic, thus inducing load on the master instead of the nodes. Furthermore, this load will not be visible in the node overview. I think this is the best point to start investigating this issue from, see this piece of documentation for explanation: https://www.jenkins.io/doc/book/pipeline/pipeline-best-practices/#making-sure-to-use-groovy-code-in-pipelines-as-glue

Best regards,

Kamil

[Dominik]

Hi Kamil,

Thanks for the link. Although we do use pipelines for many projects, the set of test jobs I mentioned are plain old 'freestyle' jobs.
They include one or multiple Maven build steps which run some more or less heavy unit and integration tests (which all happens on agents anyway).

When looking at top/htop on the server during those tests, it's really just the Jenkins main process rotating at 380-400% CPU usage (of 4 CPUs).

Regards,
Dominik

[Dominik]

We upgraded our Jenkins instance yesterday (from version 2.190.1 to 2.249.1) and the heavy load on the master "magically disappeared".
The exact same test suite now runs (and 'spams' all the agents), but the load on master stays below a value of 3 (with 4 CPUs).

[Ivan Fernandez Calvo]

remoting has tons of improvements between those two versions 3.33 vs 4.5, if you agents ar JNLP the new default protocols is JNLP 4 that is it's better than JNLP 3 IIRC it has an important performance issue