[JIRA] (JENKINS-46697) Pipeline hangs on allocate node step

4 views
Skip to first unread message

joakim.wallin@gmail.com (JIRA)

unread,
May 18, 2018, 3:14:02 AM5/18/18
to jenkinsc...@googlegroups.com
Joakim Wallin commented on Bug JENKINS-46697
 
Re: Pipeline hangs on allocate node step

Exactly the same problem seen on jenkins version 2.107.2.

 
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo

jglick@cloudbees.com (JIRA)

unread,
Jun 6, 2018, 5:04:03 PM6/6/18
to jenkinsc...@googlegroups.com
Jesse Glick updated an issue
 
Jenkins / Bug JENKINS-46697
Change By: Jesse Glick
Labels: Groovy pipeline

jglick@cloudbees.com (JIRA)

unread,
Jun 6, 2018, 5:04:03 PM6/6/18
to jenkinsc...@googlegroups.com
Jesse Glick commented on Bug JENKINS-46697
 
Re: Pipeline hangs on allocate node step

Jenkins core version probably irrelevant.

rui.teng@foxmail.com (JIRA)

unread,
Jun 19, 2018, 7:00:05 AM6/19/18
to jenkinsc...@googlegroups.com
Rui Teng commented on Bug JENKINS-46697

I can reproduce the hang issue by following code. Just run this same task on different nodes concurrently. (LOOP = 10000)

Any workaround can be made to fix the hang issue before the bug has been fixed, how about timeout and retry?

// Declarative Pipeline
pipeline {
    agent {
        label 'master'
    }

    stages {
        stage('Stress allocate node') {
            steps {
                script {
                    def nodeName = "${NODE}"
                    def loop = "${LOOP}" as Integer

                    lock(nodeName) {
                        for (int i = 1; i <= loop; i++) {
                            node(nodeName) {
                                echo "run test case. Loop: $i"
                            }
                        }
                    }
                }
            }
        }
    }
}

v.smirnov@movavi.com (JIRA)

unread,
Sep 7, 2018, 5:51:03 AM9/7/18
to jenkinsc...@googlegroups.com

We also experiencing this problem on our Jenkins server. It happens almost every day. We have mac and Win build nodes, it not depends on build type.

Please help, which logs we could provide to localize problem?
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

v.smirnov@movavi.com (JIRA)

unread,
Sep 7, 2018, 5:52:03 AM9/7/18
to jenkinsc...@googlegroups.com
Vladimir Smirnov edited a comment on Bug JENKINS-46697
We also experiencing this problem on our Jenkins server (Jenkins ver . 2.107.3). It happens almost every day. We have mac and Win build nodes, it not depends on build type.


Please help, which logs we could provide to localize problem?

nikita.stupnikov@gmail.com (JIRA)

unread,
Dec 20, 2018, 6:58:02 AM12/20/18
to jenkinsc...@googlegroups.com
Nikita Stupnikov updated an issue
 
Change By: Nikita Stupnikov
Attachment: agent02-executors.jpg
Attachment: declarative-issue.jpg
Attachment: scripted-issue.jpg

nikita.stupnikov@gmail.com (JIRA)

unread,
Dec 20, 2018, 7:04:04 AM12/20/18
to jenkinsc...@googlegroups.com
Nikita Stupnikov commented on Bug JENKINS-46697
 
Re: Pipeline hangs on allocate node step

We've noticed the same problem with our instances. Initially, we thought it is an issue with declarative pipelines only but we are able to reproduce it with scripted pipeline too (sometimes we have to make more attempts to catch it though):

def nodeName = 'agent02'
def iterations = 1000
node(nodeName) {
    stage('Stress allocate node') {
        for (int i = 1; i <= iterations; i++) {
            node(nodeName) {
                echo "run test case. Loop: $i" 
            }
        }
    }
}

For the declarative version of the same pipeline the issue is 100% reproducible in our instances:

pipeline {
    agent {
        label 'agent02'
    }
    stages {
        stage('Stress allocate node'
) {
            steps {
                script {
                    def iterations = 1000
                    for (int i = 1; i <= iterations; i++) {
                        node('agent02') {
                            echo "run test case. Loop: $i" 
                        }
                    }
                }
            }
        }
    }
}

Jesse Glick, could you please propose ideas how to troubleshoot this problem further?

jglick@cloudbees.com (JIRA)

unread,
Jan 11, 2019, 5:34:01 PM1/11/19
to jenkinsc...@googlegroups.com

A Pipeline maintainer would need to check whether this is indeed reproducible, and then take it from there.

bgiftge@gmail.com (JIRA)

unread,
Apr 30, 2019, 4:39:02 PM4/30/19
to jenkinsc...@googlegroups.com

Any update on this? It happens quite frequently for us. 

bgiftge@gmail.com (JIRA)

unread,
Apr 30, 2019, 4:41:02 PM4/30/19
to jenkinsc...@googlegroups.com
Boris Giftge edited a comment on Bug JENKINS-46697
Any update on this? It happens quite frequently for us. 


(Version 2.164.2)

kamil.kozieradzki@gmail.com (JIRA)

unread,
Jul 10, 2019, 7:47:02 AM7/10/19
to jenkinsc...@googlegroups.com

Jenkins ver. 2.176.1
It destroyed my week. (It worked long time, and after one restart this happened)
In my case workarround was:
nodeName = 'agent02'iterations = 1000

@NonCPS
def workarround() {


for (int i = 1; i <= iterations; i++) { node(nodeName)

{ echo "run test case. Loop: $i" }

}
}
node(nodeName)

{ stage('Stress allocate node') \{ workarround() }

}

kamil.kozieradzki@gmail.com (JIRA)

unread,
Jul 10, 2019, 7:48:02 AM7/10/19
to jenkinsc...@googlegroups.com
Kamil Kozieradzki edited a comment on Bug JENKINS-46697

kamil.kozieradzki@gmail.com (JIRA)

unread,
Jul 10, 2019, 7:50:03 AM7/10/19
to jenkinsc...@googlegroups.com
Kamil Kozieradzki edited a comment on Bug JENKINS-46697
 

 Jenkins ver. 2.176.1
It destroyed my week. (It worked long time, and after one restart this happened)
In my case workarround was:
``` {code:java}
nodeName = 'agent02'
iterations = 1000
@NonCPS
def workarround() {
for (int i = 1; i <= iterations; i++) {
node(nodeName)

{
echo "run test case. Loop: $i"
}
}
}
node(nodeName)

{
stage('Stress allocate node') \\ {
workarround()
}
 }
{code }
 ```



 

jglick@cloudbees.com (JIRA)

unread,
Jul 10, 2019, 8:52:05 AM7/10/19
to jenkinsc...@googlegroups.com

Kamil Kozieradzki You cannot run node from inside a @NonCPS function. Current versions of workflow-cps will warn you to this effect.
Reply all
Reply to author
Forward
0 new messages