Schedule large numbers of building jobs - perfromance issue
Tomasz Majchrowski
The challenge that I recognize will be a number of parallel tasks that needs to be handle by Jenkins. Basically I create project which based on dependency triggers the downstream project with using of groovy system scripts and
schedule2(Task, int, List). Since the number of downstream builds of the same project (with different parameter) may achieve range (200-1000) I recognize performance issue. It's really hard to say if the bottle neck is coming from:
* the bad design of one of the plugin installed in the chain or
* basically the Jenkins is not the right tool for such a scale of parallel task due to its' overhead.
I debug a bit with visualvm to found a bottle neck, however couldn't found any interesting. Anyone is having experience with such a large build queue. Could you share hints/best practice?
Best regards, Tomasz Majchrowski.
Rob Mandeville
Are you talking about 200-1000 jobs configured, or running simultaneously?
Clients feed log data and artifacts to the Jenkins server in real time. If you’re running hundreds of these simultaneously, you are using tons of threads, tons of RAM for buffers, and tons of disk revolutions. At this point, you can start thinking of your Jenkins server as a database server and determine your hardware requirements accordingly. It’s fairly easy to become I/O bound, and you might become network bound if you’re using older, slower networking.
Also, how is your polling? If you have hundreds of jobs polling, think of those as hundreds of “mini-jobs” querying source control on a regular basis. They don’t show up in the regular build logs, but they have similar resource needs. If there’s a way to avoid polling (we use GitLab and have a plugin which lets GitLab push updates to Jenkins rather than have Jenkins poll), consider it.
The guys at Cloudbees would be willing to sell you a way to do this using multiple Jenkins servers and have them all centrally managed; you may want to talk to them if this is that big of a project.
--Rob
--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/bb6d5973-f2e1-4326-843e-5a20af42443a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Click here to report this email as spam.
This e-mail and the information, including any attachments it contains, are intended to be a confidential communication only to the person or entity to whom it is addressed and may contain information that is privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the sender and destroy the original message.
Thank you.
Please consider the environment before printing this email.