| I think the idea of a declarative (and not imperative) pipeline is a lot easier on the writers end (and I assume the implementers end as well), but I think the implementation of several serial stages each fanning out to a number of parallel stages is limiting to the overall description of the dependencies between different steps/nodes. In my case (simplifying, please excuse pseudo-code), I'd like to do something like this:
stage("checkout") { node () { git XXX } }
stage("build") {
parallel ("ubuntu") { node("ubuntu") { make XXX }},
("centos") { node("centos") { make YYY }},
// tons more distributions here
}
stage ("test") {
node ("ubuntu") { test-ubuntu ZZZ }
}
stage ("publish") { node() { publish.sh }
Since I know the "test" stage only depends on the "ubuntu" part of the build stage, I'd like the test stage to fire without waiting on the whole "build" stage completes. From what I see, the parallel block waits for everything inside to complete, but I know the dependencies and can speed it up (in my case, there are a lot more nodes who match the testing than who don't). If I were to write it in bash, I would do something like:
git checkout XXX
build ubuntu &
TEST_PID=$!
build centos &
wait $TEST_PID
test-ubuntu
wait # wait for all testing and bulding to complete
publish
This is a rough way to explain the dependency graph to bash by saying "wait on 'build ubuntu' before starting the testing, but centos can take as long as it wants" and "before going to the publish step, make sure everything else is done". In my case, all the build steps for different distributions take forever because there's a finite number of VMs that can be spun up, but there's an abundance of "ubuntu" nodes that can be used for testing and compiling. It slows things down a ton to have to wait for the entire parallel node to finish before moving on, when there are free executors in latter steps that can start running. |
