Throttling parallel step

[Mirek S]

Hi!

I'm using something like this to run tests in parallel:

stage('Test') {
  steps {
    script {
      testing_closures = [one: { print("staring one"); sleep 10; print("finishing one") },
                          two: { print("staring two"); sleep 10; print("finishing two") },
                          three: { print("staring three"); sleep 10; print("finishing three") },
                          four: { print("staring four"); sleep 10; print("finishing four") },
                          five: { print("staring five"); sleep 10; print("finishing five") },
                          six: { print("staring six"); sleep 10; print("finishing six") }]
      parallel(testing_closures)
    }
  }
}

The main goal is to throttle those closures - I don't want for all six of them to run concurrently - only 3 at a time.

I was thinking about using nodes for this - i.e. wrapping each closure in node{} block:

one: { node { print("staring one"); sleep 10; print("finishing one") } }

Works OK as long as I use master node and limit the executors to 4 (1 for main job, 3 for the node{} concurrent steps). 

Unfortunately I need master node executors to be available for other jobs, so I cannot limit them.

So - how do I achieve my goal? Is there a way to create local node similar to master, but not master?

Or is there another way to throttle parallel step? 

The only solution I came up with is to create 3 lockable resources of the same label and lock them in each

of the closures. But in reality my closures are already locking resources (for test-machines, there's 7 of them, 

but I want one build to use only up to 3 of them), so I would have to have:

  lock(label: 'test-host') {
    lock (label: 'test-throttler') {
      ...code...
    }
  }

Looks dirty, increases maintenance nightmare and it's adding unnecessary resources (which we already have too much) that do

not reflect the "real, physical" resource.

[Mirek S]

The only solution I came up with is to create 3 lockable resources of the same label and lock them in each

of the closures. 

Now that I think of it - it wouldn't work anyway - another build of this job would not be able to reach the test-throttler resources, so it would wait for them anyway...