Jenkins Complex Pipeline gets constant: "Queued Waiting for run to start"

[Dan zaWhite]

Hi All,

We have a pretty big pipeline and we are running multiple iterations of it constantly.

The pipeline uses multiple executors and parallel stages for the automation.

We started getting some issues that manifest as follows:

- UI gets very slugish

- The pipeline jobs get stuck with message: Queued   Waiting for run to start

- Memory spikes to max alocated: 36 GB

- We are using the pipeline wipe workspace utility to wipe almost 50 GB of space when the job finishes

Do any of you out there encountered this type of issues?

Using latest blue ocean plugins and latest LTS version of Jenkins: 2.222.1

Thank you,

Dan Albu

[James Nord]

> We started getting some issues that manifest as follows:
- UI gets very slugish
- The pipeline jobs get stuck with message: Queued Waiting for run to start

the queue performs badly when there are lots (n hundreds) of tasks stuck waiting to be scheduled. it may be you are starting to hit this limit, but it's. a guess. Hard to say without more information and knowing what your pipeline is doing. it may be your pipeline is performing too much work in the master (actual pipeline) rather than on agents.

[Dan zaWhite]

The pipeline checks out multiple git repos and builds the product based on our architecture, then runs the automation in parallel: 20 lanes (linux/windows with selenium).

I have more than enough executors free but for some reason the pipeline gets stuck. From my investigations over the weekend it seems thatif a slave disconects mid-way because of network issues this problem manifest itself.

I use the extra durable option for pipelines.

The only thing that runs on master from my investigations is allure reporting generation and the workspace cleanup.

Does anyone know a way to log what is going on in the pipelines and on master when this types of issues happen.

The usual monitoring memory with javamelody does not work for me.

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/YI2AMQpKqts/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/5d1c018f-b7cc-4bc0-9ea6-599bc9ba2c79%40googlegroups.com.