Pipeline: Resource Locking Plugin hangs

300 views
Skip to first unread message

Sverre Moe

unread,
Mar 8, 2018, 5:38:50 AM3/8/18
to Jenkins Users
Lockable Resource Plugin
Lockable Resources Manager
  Lockable Resources
     resource: testing1-repository, labels:testing-repository
     resource: testing2-repository, labels:testing-repository
     resource: testing3-repository, labels:testing-repository


Configured these 3 resources in Jenkins. Different resource name and same label on all three resources.

Pipeline script:
def stepsForParallel = [:]
stepsForParallel["nodeName1"] = transformIntoStep("nodeName1")
stepsForParallel["nodeName2"] = transformIntoStep("nodeName2")
stepsForParallel["nodeName3"] = transformIntoStep("nodeName3")
stepsForParallel["nodeName4"] = transformIntoStep("nodeName4")

parallel stepsForParallel

def transformIntoStep(nodeName) {
   return {
       node("master") {
           if (nodeName.equals("nodeName4")) {
               lock(label: "testing-repository") {
                   println "Testing lock label"
                   sleep 20
               }
           } else {
               sleep 5 /* to make sure nodeName4 aquires its lock first. */
               if (nodeName.equals("nodeName1")) {
                   lock(resource: 'testing1-repository') {
                       println "Testing lock resource"
                       sleep 10
                   }  
                }

                if (nodeName.equals("nodeName2")) {
                   lock(resource: 'testing2-repository') {
                       println "Testing lock resource"
                       sleep 10
                   }  
                }

                if (nodeName.equals("nodeName3")) {
                   lock(resource: 'testing3-repository') {
                       println "Testing lock resource"
                       sleep 10
                   }  
                }
           }
       }
   }
}


[Pipeline] parallel
[Pipeline] [nodeName1] { (Branch: nodeName1)
[Pipeline] [nodeName2] { (Branch: nodeName2)
[Pipeline] [nodeName3] { (Branch: nodeName3)
[Pipeline] [nodeName4] { (Branch: nodeName4)
[Pipeline] [nodeName1] node
[nodeName1] Running on Jenkins in /var/lib/jenkins/workspace/pipeline-test
[Pipeline] [nodeName2] node
[nodeName2] Running on Jenkins in /var/lib/jenkins/workspace/pipeline-test@2
[Pipeline] [nodeName3] node
[nodeName3] Running on Jenkins in /var/lib/jenkins/workspace/pipeline-test@3
[Pipeline] [nodeName4] node
[Pipeline] [nodeName1] {
[Pipeline] [nodeName2] {
[Pipeline] [nodeName3] {
[nodeName4] Running on Jenkins in /var/lib/jenkins/workspace/pipeline-test@4
[Pipeline] [nodeName1] sleep
[nodeName1] Sleeping for 5 sec
[Pipeline] [nodeName2] sleep
[nodeName2] Sleeping for 5 sec
[Pipeline] [nodeName3] sleep
[nodeName3] Sleeping for 5 sec
[Pipeline] [nodeName4] {
[Pipeline] [nodeName4] lock
[nodeName4] Trying to acquire lock on [Label: testing-repository]
[nodeName4] Lock acquired on [Label: testing-repository]
[Pipeline] [nodeName4] {
[Pipeline] [nodeName4] echo
[nodeName4] Testing lock label
[Pipeline] [nodeName4] sleep
[nodeName4] Sleeping for 20 sec
[Pipeline] [nodeName1] lock
[nodeName1] Trying to acquire lock on [testing1-repository]
[nodeName1] Found 0 available resource(s). Waiting for correct amount: 1.
[nodeName1] [testing1-repository] is locked, waiting...
[Pipeline] [nodeName2] lock
[nodeName2] Trying to acquire lock on [testing2-repository]
[nodeName2] Found 0 available resource(s). Waiting for correct amount: 1.
[nodeName2] [testing2-repository] is locked, waiting...
[Pipeline] [nodeName3] lock
[nodeName3] Trying to acquire lock on [testing3-repository]
[nodeName3] Found 0 available resource(s). Waiting for correct amount: 1.
[nodeName3] [testing3-repository] is locked, waiting...
[nodeName1] Lock acquired on [testing1-repository]
[Pipeline] [nodeName4] }
[nodeName4] Lock released on resource [Label: testing-repository]
[Pipeline] [nodeName1] {
[Pipeline] [nodeName4] // lock
[Pipeline] [nodeName4] }
[Pipeline] [nodeName1] echo
[nodeName1] Testing lock resource
[Pipeline] [nodeName1] sleep
[nodeName1] Sleeping for 10 sec
[Pipeline] [nodeName4] // node
[Pipeline] [nodeName4] }
[Pipeline] [nodeName1] }
[nodeName1] Lock released on resource [testing1-repository]
[Pipeline] [nodeName1] // lock
[Pipeline] [nodeName1] }
[Pipeline] [nodeName1] // node
[Pipeline] [nodeName1] }


Does not work as I expected it to.
When nodeName4 releases the testing-repository which contains 3 resources, then the other can acquire the lock on their resources. However only one of them does so, the others do not. The entire build hangs and goes no further. 

[nodeName1] Lock acquired on [testing1-repository]
[nodeName4] Lock released on resource [Label: testing-repository]
[nodeName1] Lock released on resource [testing1-repository]

Do not know why nodeName1 acquired the lock before nodeName released.

This is the output I expected:
[nodeName4] Trying to acquire lock on [Label: testing-repository]
[nodeName4] Lock acquired on [Label: testing-repository]
[nodeName4] Lock released on resource [Label: testing-repository]
[nodeName1] Lock acquired on [testing1-repository]
[nodeName2] Lock acquired on [testing2-repository]
[nodeName3] Lock acquired on [testing3-repository]
[nodeName1] Lock released on resource [testing1-repository]
[nodeName2] Lock released on resource [testing2-repository]
[nodeName3] Lock released on resource [testing3-repository]

Reinhold Füreder

unread,
Mar 8, 2018, 6:37:47 AM3/8/18
to jenkins...@googlegroups.com

Very naïve thought(s):

 

While “Do not know why nodeName1 acquired the lock before nodeName released.“ might be just a matter of output flushing/buffering, “The entire build hangs and goes no further.” sounds really frightening to me.

 

Maybe it is the usage of lock inside parallel step and/or some CPS transformation reason, or some little resource misconfiguration issue!?

 

Ignoring CPS transformations (due to parallel step) I assume/hope the lock plugin does unit testing the “normal” parallelization usage of locking resources?

Sverre Moe

unread,
Mar 8, 2018, 11:06:12 AM3/8/18
to Jenkins Users


torsdag 8. mars 2018 12.37.47 UTC+1 skrev Reinhold Füreder følgende:

Very naïve thought(s):

 

While “Do not know why nodeName1 acquired the lock before nodeName released.“ might be just a matter of output flushing/buffering,

I thought so also, probably because of parallell execution.
 

“The entire build hangs and goes no further.” sounds really frightening to me.

 

Maybe it is the usage of lock inside parallel step and/or some CPS transformation reason, or some little resource misconfiguration issue!?

 

Ignoring CPS transformations (due to parallel step) I assume/hope the lock plugin does unit testing the “normal” parallelization usage of locking resources?


Perhaps a bug in Lockable Resource Plugin. 
There is no output of CPS transformation exception, nor any other error output. It just hangs after the first acquired resource lock is finished.

Sverre Moe

unread,
Mar 9, 2018, 5:46:25 AM3/9/18
to Jenkins Users
We are using the Lockable Resource Plugin, but currently are locking around the parallell step.
This causes a severe bottleneck when building. If several projects are building simultaneous it increases the build time by a lot.

stage("build") {
    def stepsForParallel = [:]
    stepsForParallel[buildNode1] = transformIntoStep(buildNode1)
    stepsForParallel[buildNode2] = transformIntoStep(buildNode2)

    lock(resource: 'master-repository') {
        parallel stepsForParallel
    }
}

stage("publish") {
    lock(resource: 'master-repository') {
        publishYumArtifacts()
    }
}

def transformIntoStep() {
    return {
        node(nodeName) {
            preInstall()
            compileAndBuild()
            postInstall()
        }
    }
}


This is what I would like to do instead with the lock:
stage("build") {
    def stepsForParallel = [:]
    stepsForParallel[buildNode1] = transformIntoStep(buildNode1)
    stepsForParallel[buildNode2] = transformIntoStep(buildNode2)
    parallel stepsForParallel
}

stage("publish") {
    lock(label: 'master-repository') {
        publishYumArtifacts()
    }
}

def transformIntoStep(nodeName) {
    return {
        node(nodeName) {
            lock(resource: nodeName) {
                preInstall()
            }

            compileAndBuild()

            lock(resource: nodeName) {
                postInstall()
            }
        }
    }
}


When publishing yum artifacts we need to lock all the resources with label.
When building we lock only the build node resource, and if any other build is going to publish it waits until all resources are free.

Sverre Moe

unread,
Mar 13, 2018, 3:44:21 AM3/13/18
to Jenkins Users
I created two seperate pipeline jobs to test this with.
Got the same behaviour.

pipeline-test1
def stepsForParallel = [:]

stepsForParallel["nodeName4"] = transformIntoStep("nodeName4")

parallel stepsForParallel

def transformIntoStep(nodeName) {
    return {
        node("master") {

            lock(label: "testing-repository") {
                println "Testing lock label"

                sleep 60
            }
        }
    }
}

OUTPUT
[Pipeline] parallel
[Pipeline] [nodeName4] { (Branch: nodeName4)
[Pipeline] [nodeName4] node
[nodeName4] Running on Jenkins in /var/lib/jenkins/workspace/pipeline-test
[Pipeline] [nodeName4] {
[Pipeline] [nodeName4] lock
[nodeName4] Trying to acquire lock on [Label: testing-repository]
[nodeName4] Lock acquired on [Label: testing-repository]
[Pipeline] [nodeName4] {
[Pipeline] [nodeName4] echo
[nodeName4] Testing lock label
[Pipeline] [nodeName4] sleep
[nodeName4] Sleeping for 1 min 0 sec
[Pipeline] [nodeName4] }
[nodeName4] Lock released on resource [Label: testing-repository]
[Pipeline] [nodeName4] // lock
[Pipeline] [nodeName4] }
[Pipeline] [nodeName4] // node
[Pipeline] [nodeName4] }
[Pipeline] // parallel
[Pipeline] End of Pipeline
Finished: SUCCESS


pipeline-test2
def stepsForParallel = [:]
stepsForParallel["nodeName1"] = transformIntoStep("nodeName1")
stepsForParallel["nodeName2"] = transformIntoStep("nodeName2")
stepsForParallel["nodeName3"] = transformIntoStep("nodeName3")


parallel stepsForParallel

def transformIntoStep(nodeName) {
    return {
        node("master") {

            if (nodeName.equals("nodeName1")) {
                lock(resource: 'testing1-repository') {
                    println "Testing lock resource"

                    sleep 30

                }  
            }
               
            if (nodeName.equals("nodeName2")) {
                lock(resource: 'testing2-repository') {
                    println "Testing lock resource"

                    sleep 30

                }  
            }
               
            if (nodeName.equals("nodeName3")) {
                lock(resource: 'testing3-repository') {
                    println "Testing lock resource"

                    sleep 30
                }  
            }
        }
    }
}

OUTPUT
[Pipeline] parallel
[Pipeline] [nodeName1] { (Branch: nodeName1)
[Pipeline] [nodeName2] { (Branch: nodeName2)
[Pipeline] [nodeName3] { (Branch: nodeName3)
[Pipeline] [nodeName1] node
[nodeName1] Running on Jenkins in /var/lib/jenkins/workspace/pipeline-test3
[Pipeline] [nodeName2] node
[Pipeline] [nodeName3] node
[nodeName2] Running on Jenkins in /var/lib/jenkins/workspace/pipeline-test3@2
[nodeName3] Running on Jenkins in /var/lib/jenkins/workspace/pipeline-test3@3
[Pipeline] [nodeName1] {
[Pipeline] [nodeName2] {
[Pipeline] [nodeName3] {
[Pipeline] [nodeName1] lock
[nodeName1] Trying to acquire lock on [testing1-repository]
[nodeName1] Found 0 available resource(s). Waiting for correct amount: 1.
[nodeName1] [testing1-repository] is locked, waiting...
[Pipeline] [nodeName2] lock
[nodeName2] Trying to acquire lock on [testing2-repository]
[nodeName2] Found 0 available resource(s). Waiting for correct amount: 1.
[nodeName2] [testing2-repository] is locked, waiting...
[Pipeline] [nodeName3] lock
[nodeName3] Trying to acquire lock on [testing3-repository]
[nodeName3] Found 0 available resource(s). Waiting for correct amount: 1.
[nodeName3] [testing3-repository] is locked, waiting...
[nodeName1] Lock acquired on [testing1-repository]
[Pipeline] [nodeName1] {
[Pipeline] [nodeName1] echo
[nodeName1] Testing lock resource
[Pipeline] [nodeName1] sleep
[nodeName1] Sleeping for 30 sec
[Pipeline] [nodeName1] }
[nodeName1] Lock released on resource [testing1-repository]
[Pipeline] [nodeName1] // lock
[Pipeline] [nodeName1] }
[Pipeline] [nodeName1] // node
[Pipeline] [nodeName1] }
Aborted by Sverre Moe
[Pipeline] [nodeName3] // lock
[Pipeline] [nodeName2] // lock
[Pipeline] [nodeName2] }
[Pipeline] [nodeName3] }
[Pipeline] [nodeName2] // node
[Pipeline] [nodeName3] // node
[Pipeline] [nodeName2] }
[nodeName2] Failed in branch nodeName2
[Pipeline] [nodeName3] }
[nodeName3] Failed in branch nodeName3
[Pipeline] // parallel
[Pipeline] End of Pipeline
Exception: null
org.jenkinsci.plugins.workflow.steps.FlowInterruptedException
at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution.cancel(CpsBodyExecution.java:245)
at org.jenkinsci.plugins.workflow.steps.BodyExecution.cancel(BodyExecution.java:76)
at org.jenkinsci.plugins.workflow.cps.steps.ParallelStepExecution.stop(ParallelStepExecution.java:67)
at org.jenkinsci.plugins.workflow.cps.CpsThread.stop(CpsThread.java:296)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$6.onSuccess(CpsFlowExecution.java:1083)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$6.onSuccess(CpsFlowExecution.java:1072)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4$1.run(CpsFlowExecution.java:861)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Finished: ABORTED

I had to abort this build because it had stalled.

All resources where free, but two of the parallell executions never started.
Resources
testing1-repository FREE
testing2-repository FREE
testing3-repository FREE
Labels
testing-repository 3 free resources

Sverre Moe

unread,
Mar 13, 2018, 8:04:12 AM3/13/18
to Jenkins Users
Running two builds of pipeline-test2 works fine. The second build gets all three resources when the first build has released them.
So seems the problem is when locking on label. The build will only get the first release lock.

Sverre Moe

unread,
Mar 13, 2018, 9:02:02 AM3/13/18
to Jenkins Users
Looks like there is a fix for this in the works.

Steven Foster

unread,
Mar 13, 2018, 10:52:32 AM3/13/18
to Jenkins Users
On Tuesday, March 13, 2018 at 1:02:02 PM UTC, Sverre Moe wrote:
Looks like there is a fix for this in the works.

I *think* this fix is just for manual reservation/unreservation of a resource by a user (there's a button in the list of resources)

Sverre Moe

unread,
Mar 13, 2018, 5:48:42 PM3/13/18
to Jenkins Users
That's too bad. I had my hopes up that fix would solve my problem. I have asked that question on the GitHub issue.
For the time being we are locking around the parallel step with one resource on the Build stage and the same resource on Publish stage.
This adds a severe bottleneck to the build time. Would be much better if we could use one resource lock for each parallel stage.
Reply all
Reply to author
Forward
0 new messages