[JIRA] (JENKINS-56673) Better handling of ChannelClosedException in Declarative pipeline

2 views
Skip to first unread message

collin@vimeo.com (JIRA)

unread,
Mar 21, 2019, 10:29:02 PM3/21/19
to jenkinsc...@googlegroups.com
Collin Lefeber created an issue
 
Jenkins / Improvement JENKINS-56673
Better handling of ChannelClosedException in Declarative pipeline
Issue Type: Improvement Improvement
Assignee: Carlos Sanchez
Components: kubernetes-plugin
Created: 2019-03-22 02:28
Environment: Jenkins: 2.150.2, k8s plugin version: 1.14.3
Priority: Minor Minor
Reporter: Collin Lefeber

When pods get deleted for any reason,  there is a log/exception like so:

hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on JNLP4-connect connection from .... failed. The channel is closing down or has closed down 

The job then appears to hang indefinitely until a timeout is reached or it's stopped manually.

In our use case (k8s using preemptible vms) we actually expect pods to be deleted mid build and want to be able to handle pod deletion with a retry.

I have not been able to find a way to handle this in declarative syntax.

For testing, using a very simple declarative example:

    stages {
        stage('Try test') {
            steps {
                container('jnlp') {
                    sh """
                    echo Kill the pod now
                    sleep 5m
                    """
                }
            }
            post {
                failure {
                    echo "Failuuure"
                }
            }
        }

But the exception does not actually trigger the failure block when the pod is killed.

Is there currently any best practice to handle the deletion of a pod? Are there any timeout parameters that would be useful in this case?

I'm happy to add a PR to the Readme after learning

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Apr 17, 2019, 7:30:02 AM4/17/19
to jenkinsc...@googlegroups.com
Carlos Sanchez commented on Improvement JENKINS-56673
 
Re: Better handling of ChannelClosedException in Declarative pipeline

I think this is just another version of JENKINS-55392
You can't catch these exceptions are they are underlying infra issues

jenkins-ci.org@meneguello.com (JIRA)

unread,
Jun 4, 2019, 7:31:02 AM6/4/19
to jenkinsc...@googlegroups.com

Carlos Sanchez
I don't think this is the same case.
I've opened a ticket with the same problem (sorry).
What I've tracked is when my pods are killed (by OOMKiller, usually) the job doesn't get aborted, it hangs indefinitely. The node label gets an "offline" and node log displays the OP message.
If I click the abort button, the job is aborted immediately. So, why this don't occur when the node is detected to be offline?

jglick@cloudbees.com (JIRA)

unread,
Jul 8, 2019, 2:26:03 PM7/8/19
to jenkinsc...@googlegroups.com
Jesse Glick resolved as Duplicate
 

I believe the patch for JENKINS-49707 addresses this.

Change By: Jesse Glick
Status: Open Resolved
Assignee: Carlos Sanchez
Resolution: Duplicate
Reply all
Reply to author
Forward
0 new messages