How to parallelize builds across many agents that reside on different machines?

[xcjvhv]

Hi all,

So I have the following requirements for the CI of our big monolith and I'd like to know if and how this is best achieved with Jenkins.

- We have many different machines that we want to utilize as workers

- Each machine will host many workers (as many as its cores, actually)

- A single run of our test suite should be parallelized among different different workers (our project supports this). The command each worker will use is something like this: `./tests.sh --build foo --worker bar` and this will be spawned via a docker-compose.yml

- I want to have many "build queues" for my project, so that when a user executes a build, another one is not blocked waiting for the first one. So each worker must somehow be assigned to a queue (labels?)

- I want to be able to use Pipelines declared inside the repo

How would you go about this? Should I have a "Jenkins agent per worker"? Do I have to have a "node per machine (worker host)"? What about # of executors? How about the pipeline itself: can I tell it somehow to run 50 instances of that same command (1 in each worker)?

Thanks in advance.