Small follow-up to close the case.
The limit I was experiencing in the build queue was due to the test I was using to stress Jenkins, which was a simple echo. Since the tests were quick to finish and I had lots of requests coming in Jenkins was a bit slow to answer, enough to look like it was waiting for a build to finish...
Now with a sleep of 10 seconds in the test I can stress Jenkins as I wanted with thousands of builds and it still works pretty good (congrats!).
Sorry for the trouble!
Ronan