[JIRA] (JENKINS-56888) Unexpected cgroup syntax when using docker-in-docker slave launched via SSH

3 views
Skip to first unread message

evan.bluhm@ibm.com (JIRA)

unread,
Apr 4, 2019, 3:37:02 PM4/4/19
to jenkinsc...@googlegroups.com
Evan Bluhm created an issue
 
Jenkins / Bug JENKINS-56888
Unexpected cgroup syntax when using docker-in-docker slave launched via SSH
Issue Type: Bug Bug
Assignee: Unassigned
Components: docker-workflow-plugin
Created: 2019-04-04 19:36
Environment: Operating System: Docker 17.09.0-ce on Ubuntu 16.04.4 LTS (GNU/Linux 4.4.0-127-generic x86_64)
Java Runtime: OpenJDK Runtime Environment 1.8.0_181-8u181-b13-2~deb9u1-b13
How Jenkins was installed: docker run -v /jenkins-data-mount:/var/jenkins_home jenkins/jenkins:lts
Jenkins + Plugin Versions:
    Jenkins: 2.164.1
    docker-workflow: 1.18
    docker-plugin: 1.1.6
    docker-java-api: 3.0.14
    docker-commons: 1.14
    ssh-slaves: 1.29.4
Reverse proxy: None
Slave nodes: Launching nodes using the Docker plugin via the SSH connection method (https://wiki.jenkins.io/display/JENKINS/Docker+Plugin). More info in the description
Priority: Minor Minor
Reporter: Evan Bluhm

In our configuration, we use the Docker plugin as a build cloud to dynamically spawn Jenkins slaves (https://wiki.jenkins.io/display/JENKINS/Docker+Plugin. In some cases, we need a sandboxed docker environment for building images, so the slaves start a docker-in-docker daemon when they're launched. The slaves are connected via SSH using the Docker plugin's normal "Connect with SSH" launch method. The result is that the default `agent` environment is a remote slave connected via SSH that's running in a docker container. The slave has a docker client installed, and the default docker host is the docker-in-docker daemon running within the slave container.

When using the dockerfile type of agent parameter (https://jenkins.io/doc/book/pipeline/syntax/#agent-parameters), the withDockerContainer step fails with error message:

java.io.IOException: Unexpected cgroup syntax /docker/a913fc1766a972d99ae545d6ba2508fe797c2df9b6371ff0be629436a30502a4/docker/a913fc1766a972d99ae545d6ba2508fe797c2df9b6371ff0be629436a30502a4/user/jenkins/0
{{ at org.jenkinsci.plugins.docker.workflow.client.ControlGroup.getContainerId(ControlGroup.java:60)}}
{{ at org.jenkinsci.plugins.docker.workflow.client.ControlGroup.getContainerId(ControlGroup.java:45)}}
{{ at org.jenkinsci.plugins.docker.workflow.client.ControlGroup.getContainerId(ControlGroup.java:37)}}
{{ at org.jenkinsci.plugins.docker.workflow.client.DockerClient.getContainerIdIfContainerized(DockerClient.java:336)}}
{{ at org.jenkinsci.plugins.docker.workflow.WithContainerStep$Execution.start(WithContainerStep.java:159)}}
{{ at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:268)}}
{{ at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:176)}}
{{ at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)}}
{{ at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:48)}}
{{ at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)}}
{{ at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)}}
{{ at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20)}}
{{ at org.jenkinsci.plugins.docker.workflow.Docker$Image.inside(jar:file:/var/jenkins_home/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:134)}}
{{ at org.jenkinsci.plugins.docker.workflow.Docker.node(jar:file:/var/jenkins_home/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:66)}}
{{ at org.jenkinsci.plugins.docker.workflow.Docker$Image.inside(jar:file:/var/jenkins_home/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:122)}}
{{ at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.DockerPipelineFromDockerfileScript.runImage(jar:file:/var/jenkins_home/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/DockerPipelineFromDockerfileScript.groovy:56)}}
{{ at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.AbstractDockerPipelineScript.configureRegistry(jar:file:/var/jenkins_home/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/AbstractDockerPipelineScript.groovy:73)}}
{{ at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.AbstractDockerPipelineScript.run(jar:file:/var/jenkins_home/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/AbstractDockerPipelineScript.groovy:52)}}
{{ at org.jenkinsci.plugins.pipeline.modeldefinition.agent.CheckoutScript.checkoutAndRun(jar:file:/var/jenkins_home/plugins/pipeline-model-extensions/WEB-INF/lib/pipeline-model-extensions.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/CheckoutScript.groovy:63)}}
{{ at __cps.transform__(Native Method)}}
{{ at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)}}
{{ at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)}}
{{ at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)}}
{{ at sun.reflect.GeneratedMethodAccessor110.invoke(Unknown Source)}}
{{ at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
{{ at java.lang.reflect.Method.invoke(Method.java:498)}}
{{ at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)}}
{{ at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46)}}
{{ at com.cloudbees.groovy.cps.Next.step(Next.java:83)}}
{{ at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)}}
{{ at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)}}
{{ at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)}}
{{ at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)}}
{{ at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)}}
{{ at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$101(SandboxContinuable.java:34)}}
{{ at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.lambda$run0$0(SandboxContinuable.java:59)}}
{{ at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:136)}}
{{ at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:58)}}
{{ at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:182)}}
{{ at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332)}}
{{ at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83)}}
{{ at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244)}}
{{ at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232)}}
{{ at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)}}
{{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}}
{{ at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)}}
{{ at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)}}
{{ at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)}}
{{ at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)}}
{{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}}
{{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}
{{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}
{{ at java.lang.Thread.run(Thread.java:748)}}

 

This happens with a very simple pipeline configuration with just two files in the git repo:

 

Jenkinsfile:

pipeline {
{{  agent { dockerfile true }}}
  stages {
    stage('Test 1') {
      steps {
        sh '''#!/bin/sh -x
        echo "If we got to this point, the DockerClient was correctly initialized"
        cat /proc/self/cgroup
        '''
{{      }}}
{{    }}}
{{  }}}
}

Dockerfile:

FROM busybox
RUN echo 'Add a dummy layer'

 

The unexpected "/user/jenkins/0" at the end of the cgroup path comes from the fact that we're logging in to a docker-based slave via SSH. It first started appearing when we upgraded our slave container's base OS from ubuntu:trusty to ubuntu:xenial. I've set up a basic demonstration of the difference in cgroup naming convention here: https://github.com/PeppyHare/demo-docker-workflow-issue

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

evan.bluhm@ibm.com (JIRA)

unread,
Apr 4, 2019, 3:43:04 PM4/4/19
to jenkinsc...@googlegroups.com
Evan Bluhm updated an issue
Change By: Evan Bluhm
In our configuration, we use the Docker plugin as a build cloud to dynamically spawn Jenkins slaves ( [ https://wiki.jenkins.io/display/JENKINS/Docker+Plugin ] . In some cases, we need a sandboxed docker environment for building images, so the slaves start a docker-in-docker daemon when they're launched. The slaves are connected via SSH using the Docker plugin's normal "Connect with SSH" launch method. The result is that the default `agent` environment is a remote slave connected via SSH that's running in a docker container. The slave has a docker client installed, and the default docker host is the docker-in-docker daemon running within the slave container.


When using the dockerfile type of agent parameter ( [ https://jenkins.io/doc/book/pipeline/syntax/#agent-parameters ] ), the withDockerContainer step fails with error message:


{ { noformat}
java.io.IOException: Unexpected cgroup syntax /docker/a913fc1766a972d99ae545d6ba2508fe797c2df9b6371ff0be629436a30502a4/docker/a913fc1766a972d99ae545d6ba2508fe797c2df9b6371ff0be629436a30502a4/user/jenkins/0 }}
{{ at org.jenkinsci.plugins.docker.workflow.client.ControlGroup.getContainerId(ControlGroup.java:60) }}
{{ at org.jenkinsci.plugins.docker.workflow.client.ControlGroup.getContainerId(ControlGroup.java:45) }}
{{ at org.jenkinsci.plugins.docker.workflow.client.ControlGroup.getContainerId(ControlGroup.java:37) }}
{{ at org.jenkinsci.plugins.docker.workflow.client.DockerClient.getContainerIdIfContainerized(DockerClient.java:336) }}
{{ at org.jenkinsci.plugins.docker.workflow.WithContainerStep$Execution.start(WithContainerStep.java:159) }}
{{ at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:268) }}
{{ at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:176) }}
{{ at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122) }}
{{ at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:48) }}
{{ at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) }}
{{ at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) }}
{{ at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20) }}
{{ at org.jenkinsci.plugins.docker.workflow.Docker$Image.inside(jar:file:/var/jenkins_home/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:134) }}
{{ at org.jenkinsci.plugins.docker.workflow.Docker.node(jar:file:/var/jenkins_home/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:66) }}
{{ at org.jenkinsci.plugins.docker.workflow.Docker$Image.inside(jar:file:/var/jenkins_home/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:122) }}
{{ at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.DockerPipelineFromDockerfileScript.runImage(jar:file:/var/jenkins_home/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/DockerPipelineFromDockerfileScript.groovy:56) }}
{{ at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.AbstractDockerPipelineScript.configureRegistry(jar:file:/var/jenkins_home/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/AbstractDockerPipelineScript.groovy:73) }}
{{ at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.AbstractDockerPipelineScript.run(jar:file:/var/jenkins_home/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/AbstractDockerPipelineScript.groovy:52) }}
{{ at org.jenkinsci.plugins.pipeline.modeldefinition.agent.CheckoutScript.checkoutAndRun(jar:file:/var/jenkins_home/plugins/pipeline-model-extensions/WEB-INF/lib/pipeline-model-extensions.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/CheckoutScript.groovy:63) }}
{{ at ___cps.transform___(Native Method) }}

{noformat
} }

 


This happens with a very simple pipeline configuration with just two files in the git repo:

 

Jenkinsfile:

{ {pipeline { noformat } }
pipeline {
agent \ { dockerfile true } }}
{{  stages { }}
{{    stage('Test 1') { }}
{{      steps { }}
{{        sh '''#!/bin/sh -x }}
{{        echo "If we got to this point, the DockerClient was correctly initialized" }}
{{        cat /proc/self/cgroup }}
{{        '''
} }
{{      } }}
{{    } }}
{{  } }}
{
{ noformat } }}


Dockerfile:

{ { noformat}
FROM busybox }}
{{ RUN echo 'Add a dummy layer'

{noformat
} }

 

The unexpected "/user/jenkins/0" at the end of the cgroup path comes from the fact that we're logging in to a docker-based slave via SSH. It first started appearing when we upgraded our slave container's base OS from ubuntu:trusty to ubuntu:xenial. I've set up a basic demonstration of the difference in cgroup naming convention here: [ https://github.com/PeppyHare/demo-docker-workflow-issue ]

evan.bluhm@ibm.com (JIRA)

unread,
Apr 4, 2019, 4:02:02 PM4/4/19
to jenkinsc...@googlegroups.com

pjdarton@gmail.com (JIRA)

unread,
Apr 5, 2019, 5:24:02 AM4/5/19
to jenkinsc...@googlegroups.com
pjdarton updated an issue
 
Change By: pjdarton
In our configuration, we use the Docker -Workflow plugin as a build cloud to dynamically spawn Jenkins slaves ([https:// wiki plugins .jenkins.io/ display/JENKINS/Docker+Plugin] docker-workflow) . ]

{noformat}

pipeline {
    agent { dockerfile true }
    stages {
        stage('Test 1') {
        steps {
            sh '''#!/bin/sh -x
            echo "If we got to this point, the DockerClient was correctly initialized"
            cat /proc/self/cgroup
            '''
            }
        }
    }
}
{noformat}

Dockerfile:
{noformat}
FROM busybox
RUN echo 'Add a dummy layer'
{noformat}

The unexpected "/user/jenkins/0" at the end of the cgroup path comes from the fact that we're logging in to a docker-based slave via SSH. It first started appearing when we upgraded our slave container's base OS from ubuntu:trusty to ubuntu:xenial. I've set up a basic demonstration of the difference in cgroup naming convention here: [https://github.com/PeppyHare/demo-docker-workflow-issue]

andrewtaylor2@gmail.com (JIRA)

unread,
Jan 29, 2020, 3:11:03 PM1/29/20
to jenkinsc...@googlegroups.com
Andy Taylor assigned an issue to Andy Taylor
Change By: Andy Taylor
Assignee: Andy Taylor
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

andrewtaylor2@gmail.com (JIRA)

unread,
Jan 29, 2020, 3:12:02 PM1/29/20
to jenkinsc...@googlegroups.com
Andy Taylor commented on Bug JENKINS-56888
 
Re: Unexpected cgroup syntax when using docker-in-docker slave launched via SSH

This bug is causing us a lot of pain - is there any chance someone could review and merge the fix provided by Evan please?!

Reply all
Reply to author
Forward
0 new messages