[JIRA] (JENKINS-55228) Kubernetes Pod/Container disappears after timeout or delay in build

1 view
Skip to first unread message

goff.caley@gmail.com (JIRA)

unread,
Dec 17, 2018, 1:51:37 PM12/17/18
to jenkinsc...@googlegroups.com
Caley Goff created an issue
 
Jenkins / Bug JENKINS-55228
Kubernetes Pod/Container disappears after timeout or delay in build
Issue Type: Bug Bug
Assignee: Carlos Sanchez
Components: kubernetes-plugin
Created: 2018-12-17 18:50
Priority: Minor Minor
Reporter: Caley Goff

Consistently the container will disappear in the Jenkins Pipeline with both DSL and scripted job types. 

TL;DR – When the job is delayed for whatever reason, in our case when using the input parameter, and wait around for greater than 5 or so minutes, the containers are just gone. 

 

This is a major problem for us because we need to be able to control the deployment into various environments for change control.

 

Thanks in advance for any input..

        kubernetes {
          label 'multi-image-' + env.BUILD_NUMBER
          defaultContainer 'jnlp'
          yaml """
    apiVersion: v1
    kind: Pod
    metadata:
      labels:
        some-label: some-label-value
    spec:
      containers:
      - name: notary
        slaveConnectTimeout: 500
        image: some.private.dtr/admin/jenkinsnotary:v3
        command:
        - cat
        tty: true
        livenessProbe:
          exec:
            command:
            - cat
          initialDelaySeconds: 5
          periodSeconds: 5
        volumeMounts:
        - name: dockersock
          mountPath: "/var/run/docker.sock"
      volumes:
      - name: dockersock
        hostPath:
          path: /var/run/docker.sock
    """
        }
      }

environment {
      JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true -Dhudson.remoting.Launcher.pingIntervalSec=172800"
} 
podTemplate(label: 'multi-image-' + env.BUILD_NUMBER, 
   containers: [containerTemplate(image: 'some.private.dtr/admin/jenkinsnotary:v3', alwaysPullImage: true, name: 'notary', command: 'cat', ttyEnabled: true, slaveConnectTimeout: 300, activeDeadlineSeconds: 172800 )],
   volumes: [hostPathVolume(name: 'dockersock', hostPath: '/var/run/docker.sock', mountPath: '/var/run/docker.sock')]) { }
hudson.remoting.ProxyException: java.nio.channels.ClosedChannelException
	at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
	at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:142)
	at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:795)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
Caused: hudson.remoting.ProxyException: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on JNLP4-connect connection from 10.255.1.161/10.255.1.161:1896 failed. The channel is closing down or has closed down
	at hudson.remoting.Channel.call(Channel.java:948)
	at hudson.FilePath.act(FilePath.java:1070)
	at hudson.FilePath.act(FilePath.java:1059)
	at hudson.FilePath.mkdirs(FilePath.java:1244)
	at org.jenkinsci.lib.configprovider.model.ConfigFileManager.provisionConfigFile(ConfigFileManager.java:86)
	at org.jenkinsci.plugins.configfiles.buildwrapper.ManagedFileUtil.provisionConfigFiles(ManagedFileUtil.java:57)
	at org.jenkinsci.plugins.configfiles.buildwrapper.ConfigFileBuildWrapper.setUp(ConfigFileBuildWrapper.java:66)
	at org.jenkinsci.plugins.workflow.steps.CoreWrapperStep$Execution.start(CoreWrapperStep.java:80)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:270)
Caused: hudson.remoting.ProxyException: org.codehaus.groovy.runtime.InvokerInvocationException: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on JNLP4-connect connection from 10.255.1.161/10.255.1.161:1896 failed. The channel is closing down or has closed down 




Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

goff.caley@gmail.com (JIRA)

unread,
Dec 17, 2018, 1:52:01 PM12/17/18
to jenkinsc...@googlegroups.com

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Apr 17, 2019, 7:32:03 AM4/17/19
to jenkinsc...@googlegroups.com
Carlos Sanchez commented on Bug JENKINS-55228
 
Re: Kubernetes Pod/Container disappears after timeout or delay in build

are the pods deleted in the middle of the pipeline ? or they are still running?
can you paste the debug logs from the master ?

you could check the system properties mentioned in JENKINS-55392 and increase some timeouts there

jglick@cloudbees.com (JIRA)

unread,
Jul 16, 2019, 3:43:44 PM7/16/19
to jenkinsc...@googlegroups.com
Jesse Glick assigned an issue to Unassigned
 
Change By: Jesse Glick
Assignee: Carlos Sanchez
Reply all
Reply to author
Forward
0 new messages