Build Queue Sorting

22 views

Skip to first unread message

Volker Gimple

unread,

Jan 10, 2017, 12:14:16 PM1/10/17

to jenkins...@googlegroups.com

Hi,

I am looking after a Jenkins server with roughly 300 projects. Those projects reflect different aspects of a software product, ranging from static libraries that are only used internally, shared libraries that are part of the product, several executables, documentation, tutorial programs and finally installers. The dependencies between the different projects were set up fairly thoroughly and after I installed the Dependency Queue Plugin everything was working reasonably well – the sequence in which the build queue was processed usually made sense.

Last year new requirements came up that were easiest to model with the new pipeline-as-code feature of Jenkins 2.x. But Dependency Queue does not seem to harmonize with the newer version of Jenkins anymore – ever since the upgrade, Jenkins no longer heeds the dependencies between projects when determining what project should run next. The sequence seems mostly based on first-come-first-serve now and the build nodes spend most of their time building projects numerous times because they have been built before one or several of their upstream dependencies were done. The overhead is significant: If a project is triggered that is near the root of the dependency graph we’re looking at build nodes going building for two days in a row compared to about six or seven hours with the previous version of Jenkins in which the plugin still worked.

I have experimented with the Priority Sorter Plugin but it only slightly improves the situation and according to the logs it sometimes makes decisions that I can’t get to match with its configuration. Nor have I found a suitable replacement for Dependency Queue. Dependency Queue itself has not been modified any more since 2012 so I guess there is little chance that it’ll be adapted to Jenkins 2.x and my Java skills are not sufficient to do it myself.

Has anyone else run into similar problems? (and if so: what have you done about it?)

Thanks for any input on this – it’s driving me nuts to the point where I need to consider ditching pipeline as code and going back to the original installation :’-(