[JIRA] (JENKINS-52576) Logs are flooded when sleep/touch are not available in the docker image

2 views
Skip to first unread message

andrea.giardini@camunda.com (JIRA)

unread,
Jul 16, 2018, 7:08:02 AM7/16/18
to jenkinsc...@googlegroups.com
Andrea Giardini created an issue
 
Jenkins / Bug JENKINS-52576
Logs are flooded when sleep/touch are not available in the docker image
Issue Type: Bug Bug
Assignee: Carlos Sanchez
Components: kubernetes-plugin
Created: 2018-07-16 11:07
Priority: Minor Minor
Reporter: Andrea Giardini

I have tried to build images on my k8s cluster using kaniko and the kubernetes-plugin, while for some images the process works just fine for others I encountered a misbehaviour.

In particular, the Jenkins build logs are flooded with the following lines, making the build fail : 

*12:51:38* sh: sleep: not found
*12:51:38* sh: touch: not found

I tried to debug the issue and eventually, I realized that Jenkins is running a command using nohup that is not able to find the sleep and touch utilities (the kaniko image is extremely minimal) because they are not in the folder where Jenkins expects them, but they are under the /busybox folder.

 

I managed to mitigate the problem adding an env variables in order to overwrite the path:

  stages {
    stage('Build with Kaniko') {
      environment {
        PATH = "/busybox:$PATH"
      }
      steps {
        container(name: 'kaniko', shell: '/busybox/sh') {
            sh '''#!/busybox/sh
            /kaniko/executor -v warn -f `pwd`/Dockerfile -c `pwd` -d 'gcr.io/camunda-public/bpmn-io-demo:tmp'
            '''
        }
      }
    }
  }

This approach partially fixes the problem. The logs are still flooded, but for a shorter period of time. Moreover, the following line appear at a certain point.

12:51:38 sh: touch: not found
12:51:39 sh: sleep: not found
12:51:39 sh: touch: not found
12:51:39 sh: sleep: Text file busy
12:51:39 sh: touch: Text file busy
12:51:39 sh: sleep: Text file busy

 
I am very confident that the responsible of this is the following line: https://github.com/jenkinsci/kubernetes-plugin/blob/6f5db53a096b241a7cb6bb9f26e898f5c2e04f38/src/main/java/org/csanchez/jenkins/plugins/kubernetes/pipeline/ContainerExecDecorator.java#L531 since I can see in the debug log that a small bash script using the sleep/touch commands is called by Jenkins using nohup.

I believe the problem is related (but I haven't verified) only to Docker multi-stage builds.

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396)

andrea.giardini@camunda.com (JIRA)

unread,
Jul 16, 2018, 7:09:01 AM7/16/18
to jenkinsc...@googlegroups.com
Andrea Giardini updated an issue
Change By: Andrea Giardini
I have tried to build images on my k8s cluster using kaniko and the kubernetes-plugin , while . While for some images the process works just fine for others I encountered a misbehaviour.


In particular, the Jenkins build logs are flooded with the following lines, making the build fail : 

{code}

*12:51:38* sh: sleep: not found
*12:51:38* sh: touch: not found
{code}


I tried to debug the issue and eventually, I realized that Jenkins is running a command using nohup that is not able to find the sleep and touch utilities (the kaniko image is extremely minimal) because they are not in the folder where Jenkins expects them, but they are under the /busybox folder.

I managed to mitigate the problem adding an env variables in order to overwrite the path:

{code}

  stages {
    stage('Build with Kaniko') {
      environment {
        PATH = "/busybox:$PATH"
      }
      steps {
        container(name: 'kaniko', shell: '/busybox/sh') {
            sh '''#!/busybox/sh
            /kaniko/executor -v warn -f `pwd`/Dockerfile -c `pwd` -d 'gcr.io/camunda-public/bpmn-io-demo:tmp'
            '''
        }
      }
    }
  }

{code}


This approach partially fixes the problem. The logs are still flooded, but for a shorter period of time. Moreover, the following line appear at a certain point.

{code}

12:51:38 sh: touch: not found
12:51:39 sh: sleep: not found
12:51:39 sh: touch: not found
12:51:39 sh: sleep: Text file busy
12:51:39 sh: touch: Text file busy
12:51:39 sh: sleep: Text file busy
{code}

 
I am very confident that the responsible of this is the following line: https://github.com/jenkinsci/kubernetes-plugin/blob/6f5db53a096b241a7cb6bb9f26e898f5c2e04f38/src/main/java/org/csanchez/jenkins/plugins/kubernetes/pipeline/ContainerExecDecorator.java#L531 since I can see in the debug log that a small bash script using the sleep/touch commands is called by Jenkins using nohup.

I believe the problem is related (but I haven't verified) only to Docker multi-stage builds.

andrea.giardini@camunda.com (JIRA)

unread,
Jul 16, 2018, 7:10:02 AM7/16/18
to jenkinsc...@googlegroups.com
Andrea Giardini updated an issue
I have tried to build images on my k8s cluster using kaniko and the kubernetes-plugin. While for some images the process works just fine for others I encountered a misbehaviour.


In particular, the Jenkins build logs are flooded with the following lines, making the build fail : 

{code}
*12:51:38* sh: sleep: not found
*12:51:38* sh: touch: not found
{code}

I tried to debug the issue and eventually, I realized that Jenkins is running a command using nohup that is not able to find the sleep and touch utilities (the kaniko image is extremely minimal) because they are not in the folder where Jenkins expects them, but they are under the /busybox folder.

I managed to mitigate the problem by adding an env variables environment variable in order to overwrite the path PATH :


{code}
  stages {
    stage('Build with Kaniko') {
      environment {
        PATH = "/busybox:$PATH"
      }
      steps {
        container(name: 'kaniko', shell: '/busybox/sh') {
            sh '''#!/busybox/sh
            /kaniko/executor -v warn -f `pwd`/Dockerfile -c `pwd` -d 'gcr.io/camunda-public/bpmn-io-demo:tmp'
            '''
        }
      }
    }
  }

{code}

This approach partially fixes the problem. The logs are still flooded, but for a shorter period of time. Moreover, the following line appear appears at a certain point.


{code}
12:51:38 sh: touch: not found
12:51:39 sh: sleep: not found
12:51:39 sh: touch: not found
12:51:39 sh: sleep: Text file busy
12:51:39 sh: touch: Text file busy
12:51:39 sh: sleep: Text file busy
{code}
 
I am very confident that the responsible of this problem is the following line: https://github.com/jenkinsci/kubernetes-plugin/blob/6f5db53a096b241a7cb6bb9f26e898f5c2e04f38/src/main/java/org/csanchez/jenkins/plugins/kubernetes/pipeline/ContainerExecDecorator.java#L531 since I can see in the debug log that a small bash script using the sleep/touch commands is called by Jenkins using nohup.


I believe the problem is related (but I haven't verified) only to Docker multi-stage builds.

andrea.giardini@camunda.com (JIRA)

unread,
Jul 16, 2018, 7:14:02 AM7/16/18
to jenkinsc...@googlegroups.com
Andrea Giardini updated an issue
I have tried to build images on my k8s cluster using kaniko and the kubernetes-plugin. While for some images the process works just fine for others I encountered a misbehaviour.

In particular, the Jenkins build logs are flooded with the following lines, making the build fail : 

{code}
*12:51:38* sh: sleep: not found
*12:51:38* sh: touch: not found
{code}

I tried to debug the issue and eventually, I realized that Jenkins is running a command using nohup that is not able to find the sleep and touch utilities (the kaniko image is extremely minimal) because they are not in the folder where Jenkins expects them, but they are under the /busybox folder.

I managed to mitigate the problem by adding an environment variable in order to overwrite the PATH:


{code}
  stages {
    stage('Build with Kaniko') {
      environment {
        PATH = "/busybox:$PATH"
      }
      steps {
        container(name: 'kaniko', shell: '/busybox/sh') {
            sh '''#!/busybox/sh
            /kaniko/executor -v warn -f `pwd`/Dockerfile -c `pwd` -d 'gcr.io/camunda-public/bpmn-io-demo:tmp'
            '''
        }
      }
    }
  }

{code}

This approach partially fixes the problem. The logs are still flooded, but for a shorter period of time. Moreover, the following line appears at a certain point.


{code}
12:51:38 sh: touch: not found
12:51:39 sh: sleep: not found
12:51:39 sh: touch: not found
12:51:39 sh: sleep: Text file busy
12:51:39 sh: touch: Text file busy
12:51:39 sh: sleep: Text file busy
{code}
 
I am very confident that the responsible of this problem is the following line: https://github.com/jenkinsci/kubernetes-plugin/blob/6f5db53a096b241a7cb6bb9f26e898f5c2e04f38/src/main/java/org/csanchez/jenkins/plugins/kubernetes/pipeline/ContainerExecDecorator.java#L531 since I can see in the debug log that a small bash script using the sleep/touch commands is called by Jenkins using nohup.

I believe the problem is related (but I haven't verified) only to Docker multi-stage builds.


EDIT: Here you have the command that I believe is responsible for the issue:

{code}
Executing command: "nohup" "sh" "-c" "{ while [ -d '/home/jenkins/workspace/bpmn-io_bpmn-io-demo_newCI-F5F74DO475B5SPC6Z6AH66CH2AZ6IJS4HMDREKRSG7AXXJGZAUCA@tmp/durable-48787ddf' -a \! -f '/home/jenkins/workspace/bpmn-io_bpmn-io-demo_newCI-F5F74DO475B5SPC6Z6AH66CH2AZ6IJS4HMDREKRSG7AXXJGZAUCA@tmp/durable-48787ddf/jenkins-result.txt' ]; do touch '/home/jenkins/workspace/bpmn-io_bpmn-io-demo_newCI-F5F74DO475B5SPC6Z6AH66CH2AZ6IJS4HMDREKRSG7AXXJGZAUCA@tmp/durable-48787ddf/jenkins-log.txt'; sleep 3; done } & jsc=durable-49cb86e613753ffa205fa30f607a20d9; JENKINS_SERVER_COOKIE=\$jsc '/home/jenkins/workspace/bpmn-io_bpmn-io-demo_newCI-F5F74DO475B5SPC6Z6AH66CH2AZ6IJS4HMDREKRSG7AXXJGZAUCA@tmp/durable-48787ddf/script.sh' > '/home/jenkins/workspace/bpmn-io_bpmn-io-demo_newCI-F5F74DO475B5SPC6Z6AH66CH2AZ6IJS4HMDREKRSG7AXXJGZAUCA@tmp/durable-48787ddf/jenkins-log.txt' 2>&1; echo \$? > '/home/jenkins/workspace/bpmn-io_bpmn-io-demo_newCI-F5F74DO475B5SPC6Z6AH66CH2AZ6IJS4HMDREKRSG7AXXJGZAUCA@tmp/durable-48787ddf/jenkins-result.txt.tmp'; mv '/home/jenkins/workspace/bpmn-io_bpmn-io-demo_newCI-F5F74DO475B5SPC6Z6AH66CH2AZ6IJS4HMDREKRSG7AXXJGZAUCA@tmp/durable-48787ddf/jenkins-result.txt.tmp' '/home/jenkins/workspace/bpmn-io_bpmn-io-demo_newCI-F5F74DO475B5SPC6Z6AH66CH2AZ6IJS4HMDREKRSG7AXXJGZAUCA@tmp/durable-48787ddf/jenkins-result.txt'; wait"
exit
{code}

andrea.giardini@camunda.com (JIRA)

unread,
Jul 17, 2018, 1:52:02 AM7/17/18
to jenkinsc...@googlegroups.com
Andrea Giardini commented on Bug JENKINS-52576
 
Re: Logs are flooded when sleep/touch are not available in the docker image

I can reproduce 100% of the time with this stripped down Dockerfile:

FROM node:8.11.3-alpine as builder
RUN env
FROM node:8.11.3-alpine
RUN sleep 10
RUN env

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Jul 17, 2018, 6:40:01 AM7/17/18
to jenkinsc...@googlegroups.com

The touch and sleep commands must come from durable pipeline plugin. The image has touch and sleep under busybox so it should work with the right configuration
Added better examples at https://github.com/jenkinsci/kubernetes-plugin/pull/360

andrea.giardini@camunda.com (JIRA)

unread,
Jul 17, 2018, 7:08:03 AM7/17/18
to jenkinsc...@googlegroups.com

The PATH setting that you added in pull request does not fix the problem and does not reproduce it since it looks like this involves specifically multi-stage builds

Do you have any suggestion on how i should fix this?

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Jul 17, 2018, 7:13:01 AM7/17/18
to jenkinsc...@googlegroups.com

I get successful builds with your multistage example despite the "not found" errors

andrea.giardini@camunda.com (JIRA)

unread,
Jul 17, 2018, 7:18:01 AM7/17/18
to jenkinsc...@googlegroups.com

That's true but the logging is quite aggressive and annoying... Should I open a bug report to the durable pipeline plugin directly?

andrea.giardini@camunda.com (JIRA)

unread,
Jul 18, 2018, 2:07:01 AM7/18/18
to jenkinsc...@googlegroups.com

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Jul 19, 2018, 3:29:01 AM7/19/18
to jenkinsc...@googlegroups.com

hypnoce@donarproject.org (JIRA)

unread,
Jul 30, 2018, 5:38:01 PM7/30/18
to jenkinsc...@googlegroups.com
Francis JAC commented on Bug JENKINS-52576
 
Re: Logs are flooded when sleep/touch are not available in the docker image

The problem occurs because kaniko deletes the full file system between two stages, including the /bin folder with touch and sleep.

andrea.giardini@camunda.com (JIRA)

unread,
Jul 31, 2018, 2:04:02 AM7/31/18
to jenkinsc...@googlegroups.com

What do you think it is the best way to solve this problem?

dbeck@cloudbees.com (JIRA)

unread,
Aug 12, 2018, 7:36:02 PM8/12/18
to jenkinsc...@googlegroups.com

v.behar@free.fr (JIRA)

unread,
Oct 1, 2018, 2:58:02 AM10/1/18
to jenkinsc...@googlegroups.com

Hi guys, FYI this has been fixed upstream in kaniko: https://github.com/GoogleContainerTools/kaniko/pull/369

 

So building multi-stages images won't delete the /busybox directory anymore, and the touch/sleep commands will still be available.

 

I believe this ticket can now be closed

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

NickHKraus@gmail.com (JIRA)

unread,
May 1, 2019, 1:55:02 PM5/1/19
to jenkinsc...@googlegroups.com

We were able to mitigate this issue by using a statically-linked Busybox binary, then adding the following to our Jenkins agent Docker image:

Dockerfile

FROM busybox:1.30.1 as busybox

COPY --from=busybox /bin /busybox

ENV PATH=/busybox:$PATH

# Docker volumes include an entry in /proc/self/mountinfo. This file is used
# when kaniko builds the list of whitelisted directories. Whitelisted
# directories are persisted between stages and are not included in the final
# Docker image.
VOLUME /busybox

 The Dockerfile for our kaniko Jenkins agent can be found here.

jglick@cloudbees.com (JIRA)

unread,
Jul 16, 2019, 3:43:39 PM7/16/19
to jenkinsc...@googlegroups.com
Jesse Glick assigned an issue to Unassigned
 
Change By: Jesse Glick
Assignee: Carlos Sanchez
Reply all
Reply to author
Forward
0 new messages