[JIRA] (JENKINS-50744) locks in parallel phases are incorrectly handled

12 views
Skip to first unread message

premgangana@gmail.com (JIRA)

unread,
Apr 11, 2018, 12:42:02 PM4/11/18
to jenkinsc...@googlegroups.com
Prem Gangana created an issue
 
Jenkins / Bug JENKINS-50744
locks in parallel phases are incorrectly handled
Issue Type: Bug Bug
Assignee: Unassigned
Components: lockable-resources-plugin
Created: 2018-04-11 16:41
Environment: Jenkins : 2.89.4
Lockable Resource Plugin : 2.1
Labels: lockable pipeline parallel
Priority: Major Major
Reporter: Prem Gangana

Hi,

We have noticed, that the locks were not handled / cleaned up properly during parallel phase executions. 

Sample Pipeline Script: 

node {
milestone()
lock(resource: "my_bld_lock", inversePrecedence: true) {
milestone()
stage("Bld") {
sleep 5
}
}

milestone()
parallel([
"Testing": {
lock(resource: "my_test_lock", inversePrecedence: true) {
stage("Test") {
// error "error" 
sleep 10
}
}
},
"Second Level Testing": {
lock(resource: "my_second_test_lock", inversePrecedence: true) {
stage("Second Level Test") {
sleep 10
}
}
},
"Deployment": {
lock(resource: "my_deploy_lock", inversePrecedence: true) {
stage("Deploy") {
sleep 90
}
}
},
failFast: true
])
}

Scenario :

Trigger a pipeline build with with above pipeline script first and then trigger a second one with the line error "error" un-commented.  You will see that, the first pipeline build gets killed (ABORTED or NOT_BUILT) as soon as the second pipeline build errors out. The first pipeline build console shows that it is Superseded by the second build, which obviously should not be the case, the second build  was waiting for the lock "my_deploy_lock"  and it should not have killed the build that possessed the lock.

Note: failFast is required in this case, the problem is that the errored build is killing another build

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo

premgangana@gmail.com (JIRA)

unread,
Apr 11, 2018, 1:04:02 PM4/11/18
to jenkinsc...@googlegroups.com
Prem Gangana updated an issue
Change By: Prem Gangana
Hi,

We have noticed, that the locks were not handled / cleaned up properly during parallel phase executions. 

Sample Pipeline Script: 
{code:java}
{code}
Scenario :

Trigger a pipeline build with with above pipeline script first and then trigger a second one with the line _error "error"_ un-commented.  You will see that, the first pipeline build gets killed (ABORTED or NOT_BUILT) as soon as the second pipeline build errors out. The first pipeline build console shows that it is Superseded by the second build, which obviously should not be the case, the second build  was waiting for the lock "my_deploy_lock"  and it should not have killed the build that possessed the lock.


Note: failFast is required in this case, the problem is that the errored build is killing another build

premgangana@gmail.com (JIRA)

unread,
May 8, 2018, 4:29:02 PM5/8/18
to jenkinsc...@googlegroups.com

andrew.bayer@gmail.com (JIRA)

unread,
Aug 8, 2018, 2:04:02 PM8/8/18
to jenkinsc...@googlegroups.com
Andrew Bayer updated an issue
Change By: Andrew Bayer
Component/s: pipeline
This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396)

premgangana@gmail.com (JIRA)

unread,
May 23, 2019, 4:31:03 PM5/23/19
to jenkinsc...@googlegroups.com
Prem Gangana commented on Bug JENKINS-50744
 
Re: locks in parallel phases are incorrectly handled

Could some one look into this ? This has major side effects even with out failFast:true. the Errored builds kill earlier builds making them think they were superseeded by the errored build. I believe this is happening because the errored build is releasing all the locks even the ones that are not yet acquired by it and then the earlier builds were getting superseeded due to this and are getting into ABORTED / NOT_BUILT state.

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

premgangana@gmail.com (JIRA)

unread,
Jun 4, 2019, 12:14:03 AM6/4/19
to jenkinsc...@googlegroups.com
Prem Gangana updated an issue
 
Change By: Prem Gangana
Labels: lockable milestone parallel pipeline

premgangana@gmail.com (JIRA)

unread,
Jun 4, 2019, 12:14:03 AM6/4/19
to jenkinsc...@googlegroups.com
Prem Gangana updated an issue
Change By: Prem Gangana
Component/s: pipeline-milestone-step-plugin
Component/s: lockable-resources-plugin

premgangana@gmail.com (JIRA)

unread,
Jun 4, 2019, 12:17:02 AM6/4/19
to jenkinsc...@googlegroups.com
 
Re: locks in parallel phases are incorrectly handled

After few experiments, noticed that the milestone steps are the culprits, without the milestone steps, the earlier builds does not get superseded and aborted if the current one fails

premgangana@gmail.com (JIRA)

unread,
Jun 4, 2019, 12:17:02 AM6/4/19
to jenkinsc...@googlegroups.com
Prem Gangana updated an issue
Change By: Prem Gangana
Labels: lockable milestone parallel pipeline

premgangana@gmail.com (JIRA)

unread,
Jun 4, 2019, 12:20:02 AM6/4/19
to jenkinsc...@googlegroups.com
Prem Gangana updated an issue
Hi,

We have noticed, I would like to know if there is something wrong with below pipeline script.

Step 1 : Trigger a pipeline build with the {{error error}} commented out Step 2 : Immediately trigger another pipeline build with {{error error}} un-commented Step 3 : You will see
that the locks were build 1 gets superseded and aborted as soon as the build 2 errors out.

I would expect build 1 to
not handled / cleaned up properly during parallel phase executions get superseded and finish, since build 2 {color:#FF0000}*failed*{color}, not *passed* .  

I have removed the milestone steps from the script and then repeated Steps 1 & 2. This time, build 1 does not get superseded and aborted.

I wonder if the usage of milestones is flawed here ? If so, how can i change the script to achieve expected behavior

Sample Pipeline Script: 
{code:java}
node {
  timestamps{
  milestone()
  lock(resource: "my_bld_lock", inversePrecedence: true) {
    milestone()
    stage("Bld") {
      sleep 5 10
    }
  }

  milestone()
    lock(resource: "my_pkg_lock", inversePrecedence: true) {
      milestone()
      stage("Pkg") {
        sleep 15
      }
    }
    milestone()
  parallel([
          "Testing"           : {
            lock(resource: "my_test_lock", inversePrecedence: true) {
              stage("Test") {
                // error "error"
                sleep 10 6
              }
            }
          },
          "Second Level Testing": {
            lock(resource: "my_second_test_lock", inversePrecedence: true) {
              stage("Second Level Test") {
                sleep 10 4
              }
            }
          },
          "
Deployment Create & Deploy "          : {
            lock(resource: "
my_deploy_lock my_create_lock ", inversePrecedence: true) {
              stage("
Deploy Create ") {
                sleep
90 5
              }                   // error "error"
                  sleep 30
            }
          }
,
          failFast
              lock(resource
              : "my_deploy_lock", inversePrecedence: true ) {
  ]                 stage("Deploy" ) {
                  sleep 70
}
{code }
Scenario :             },
    ])
Trigger a pipeline build with with above pipeline script first and then trigger a second one with the line _error "error"_ un-commented.  You will see that, the first pipeline build gets killed     milestone ( ABORTED or NOT_BUILT ) as soon as the second pipeline build errors out. The first pipeline build console shows that it is Superseded by the second build, which obviously should not be the case, the second build  was waiting for the lock "my_deploy_lock"  and it should not have killed the build that possessed the lock.
  }
Note: failFast is required in this case, the problem is that the errored build is killing another build
}
{code}
 

premgangana@gmail.com (JIRA)

unread,
Jun 4, 2019, 12:20:03 AM6/4/19
to jenkinsc...@googlegroups.com
Prem Gangana updated an issue
Hi,

I would like to know if there is something wrong with below pipeline script.

Step 1 : Trigger a pipeline build with the {{error error}} commented out
Step 2 : Immediately trigger another pipeline build with {{error error}} un-commented

Step 3 : You will see that the build 1 gets superseded and aborted as soon as the build 2 errors out.

I would expect build 1 to not get superseded and finish, since build 2 {color:#
FF0000 ff0000 }*failed*{color}, not *passed*.


I have removed the milestone steps from the script and then repeated Steps 1 & 2. This time, build 1 does not get superseded and aborted.

I wonder if the usage of milestones is flawed here ? If so, how can i change the script to achieve expected behavior

Sample Pipeline Script: 
{code:java}
node {
  timestamps{
    milestone()
    lock(resource: "my_bld_lock", inversePrecedence: true) {
      milestone()
      stage("Bld") {
        sleep 10

      }
    }

    milestone()
    lock(resource: "my_pkg_lock", inversePrecedence: true) {
      milestone()
      stage("Pkg") {
        sleep 15
      }
    }
    milestone()
    parallel([
            "Testing"           : {
              lock(resource: "my_test_lock", inversePrecedence: true) {
                stage("Test") {
                  sleep 6

                }
              }
            },
            "Second Level Testing": {
              lock(resource: "my_second_test_lock", inversePrecedence: true) {
                stage("Second Level Test") {
                  sleep 4
                }
              }
            },
            "Create & Deploy"          : {
              lock(resource: "my_create_lock", inversePrecedence: true) {
                stage("Create") {
                  sleep 5

                  // error "error"
                  sleep 30
                }
              }

              lock(resource: "my_deploy_lock", inversePrecedence: true) {
                stage("Deploy") {
                  sleep 70
                }
              }
            },
    ])
    milestone()
  }
}
{code}
 

premgangana@gmail.com (JIRA)

unread,
Jun 4, 2019, 12:21:01 AM6/4/19
to jenkinsc...@googlegroups.com
Prem Gangana updated an issue
Hi,

I would like to know if there is something wrong with below pipeline script.

Step 1 : Trigger a pipeline build with the {{error error}} commented out

Step 2 : Immediately trigger another pipeline build with {{error error}} un-commented

Step 3 : You will see that the build 1 gets superseded and aborted as soon as the build 2 errors out.

I would expect build 1 to not get superseded and finish, since build 2 {color:#ff0000}*failed*{color}, not *passed*.

I have removed the milestone steps from the script and then repeated Steps 1 & 2. This time, build 1 does not get superseded and
do not get aborted.

premgangana@gmail.com (JIRA)

unread,
Jun 4, 2019, 12:22:01 AM6/4/19
to jenkinsc...@googlegroups.com
Prem Gangana assigned an issue to Antonio Muñiz
 

Hi Antonio,

Could you check this issue ?

-Prem

Change By: Prem Gangana
Assignee: Antonio Muñiz
Reply all
Reply to author
Forward
0 new messages