Is there a maximum number of parallel builds a pipeline job can run?

[Chris Overend]

I am doing testing of our Jenkins setup.

I have a build job that sleeps for 30 sec.

I ran a pipeline that runs 12,000 of these in parallel on 100 agents.

Best possible time is 1 hr.

The builds seem to finish in a reasonable amount of time.

So after build queue is emptied and the pipeline continues to run for a lengthy amount of time.

It has been running for 1 hr 30 min and is processing FINISHED job 8880.

So 3120 more FINISHED jobs to process.

It is unclear what the pipeline is doing?

Is there documentation on the basic operations the pipeline is conducting after all the builds finish?

Are there setting that affect this behavior?

Could my Groovy script be causing this long delay?

Thanks,

Chris

[Chris Overend]

So 3 hr 30 min later the pipeline is locked up

[Chris Overend]

I need to retry it may be that the jobs never all started

Branch: job_0 - (1ms in block)

Branch: job_9830 - (3min 16sec in block)

Branch: job_9831 - (No timing in block)

[Chris Overend]

So not sure if this is a Jenkins limitation or pipeline.

The jobs never exceeded available resources.

The garbage collection was stable.

So why did it lock-up?

It did say I used 

• 2950 million active threads
• 350 threads

[Bill Dennis]

Hey there -

I think it might be worth posting the pipeline of your orchestration job that runs the 12000 builds for anyone to comment in more detail.

From what I understand, if your pipeline is not orchestrating each build in a node section it will use something called a 'flyweight executor' on the Jenkins master.

If this is happening, even though you have 100 agents your master may be heavily loaded running the pipeline - I guess each build allocates some Java objects that have to be GCed.

If I can't find any documentation, I tend to go onto Github to look at the source code to understand what the Jenkins pipeline is doing behind the scenes: https://github.com/jenkinsci/workflow-cps-plugin.

I ran a test on a system where I had an orchestrator kicking off 1000 builds every 4 minutes on a schedule. After a few days the Jenkins service stopped responding (Java memory issues). One thing I have found I needed to pay attention to was file and process limits as documented here:

https://support.cloudbees.com/hc/en-us/articles/204231510-Memory-problem-unable-to-create-new-native-thread-

All the best,

--Bill

[Chris Overend]

So I was not running in a node block I will do more testing thank you.

Here is a sample of my code: 

#!groovy

import hudson.model.*

stage('many_jobs') {

    parallel (get_many_jobs(6000))

}

def get_many_jobs(num_builds) {

    // The map we'll store the parallel steps in before executing them.

    def stepsForParallel = [:]

    for (int i = 0; i < num_builds; i++) {

        def sub_run_config = 'job_' + i

            stepsForParallel[sub_run_config] = {

                build job: 'sleep_30', parameters: [string(name: 'job_name', value: sub_run_config)]

         }

    }

    return stepsForParallel

}