Simultaneous pipelines deadlocked "waiting for next available executor"

[John Girvin]

I'm trying to understand a problem with our Jenkins, where it seems piplines get deadlocked waiting for an executor if we run many of them simultaneously.

Jenkins 2.271 on Linux x64. We have one master node configured with 4 executors. Running a job singly shows it taking 2 slots in the Build Executor Status panel and works fine. However, if we run a number of jobs simultaneously they all get stuck "waiting for next available executor" and not progressing, seemingly indefinitely.

The Console Output in each shows the pipeline library being checked out, then:

[Pipeline] Start of Pipeline

[Pipeline] node
Still waiting to schedule task
Waiting for next available executor

It's as if the script flyweights are consuming all the executors, leaving none available to run the steps, but that doesn't really make sense to me given what I've read about flyweights (they are unlimited?).

It's not a machine capacity issue either. I wondered if it was how the pipelines are structured?

The pipelines are all scripted not declarative, and look like this:

@Library('jenkins-pipelines@master') _
node {
    buildDeployMaven([

...parameters...

    ])
}

buildDeployMaven is a groovy method structured like so, with each stage being a script step:

def call(params) {
    agent any
    tools {
    ...
    }
    pipeline {
        stages {
            stage('Checkout') {
                steps {
                    script {
                    ...
                    }
                }
            }

            [further stages omitted but similar]
        }
    }
}

Any ideas appreciated!

[John Girvin]

I've since found that removing the node{} wrapping the custom buildDeployMaven step seems to solve the deadlock.

I'm guessing node{} was allocating an executor for the step, which then immediately requested another executor for it's internal pipeline{} ?