[JIRA] [gerrit-trigger-plugin] (JENKINS-35370) Workflow shell step ERROR: script returned exit code -1

1,240 views
Skip to first unread message

z.jasinski@samsung.com (JIRA)

unread,
Jun 6, 2016, 6:44:01 AM6/6/16
to jenkinsc...@googlegroups.com
zbigniew jasinski created an issue
 
Jenkins / Bug JENKINS-35370
Workflow shell step ERROR: script returned exit code -1
Issue Type: Bug Bug
Assignee: rsandell
Components: gerrit-trigger-plugin, workflow-plugin
Created: 2016/Jun/06 10:43 AM
Environment: jenkins 2.8, gerrit-trigger 2.21.0,
Priority: Minor Minor
Reporter: zbigniew jasinski

So I'm trying to use workflow and gerrit-trigger plugins together. But I'm getting script returned exit code -1. I've checked, and it's not 'git fetch' network related issue or anything like that. Any command I run through 'sh' which make some output delay like, git fetch or clone, or whatever - gives me -1 exit code. Groovy script is same like in gerrit-trigger example from https://wiki.jenkins-ci.org/display/JENKINS/Gerrit+Trigger
My log samples below:

Retriggered by user admin for Gerrit: https://url/gerrit/73010 in silent mode.
[Pipeline] node
Running on master in /var/jenkins_home/workspace/pipeline-docker-test
[Pipeline] {
[Pipeline] withDockerServer
[Pipeline] {
[Pipeline] sh
[pipeline-docker-test] Running shell script
+ docker inspect -f . dockers.local:5000/image:latest
.
[Pipeline] withDockerContainer
$ docker run -t -d -u 1000:1000 -w /var/jenkins_home/workspace/pipeline-docker-test -v /var/jenkins_home/workspace/pipeline-docker-test:/var/jenkins_home/workspace/pipeline-docker-test:rw -v /var/jenkins_home/workspace/pipeline-docker-test@tmp:/var/jenkins_home/workspace/pipeline-docker-test@tmp:rw -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** dockers.local:5000/image:latest cat
[Pipeline] {
[Pipeline] stage (cloning repo)
Entering stage cloning repo
Proceeding
[Pipeline] git
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url ssh://url:29418/project # timeout=10
Fetching upstream changes from ssh://url:29418/project
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress ssh://url:29418/project +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision 82beb7889ae2fea7a35a59575bb849c881302eec (refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 82beb7889ae2fea7a35a59575bb849c881302eec # timeout=10
 > git branch -a -v --no-abbrev # timeout=10
 > git branch -D master # timeout=10
 > git checkout -b master 82beb7889ae2fea7a35a59575bb849c881302eec
 > git rev-list 82beb7889ae2fea7a35a59575bb849c881302eec # timeout=10
[Pipeline] stage (Checkout patchset)
Entering stage Checkout patchset
Proceeding
[Pipeline] sh
[pipeline-docker-test] Running shell script
+ git fetch origin refs/changes/10/73010/1:change-73010-1
[Pipeline] }
$ docker stop d3499aacbbef9d2acbb0e9cdbac51c0ea620e91b9893d46ceda49d5344357c61
$ docker rm -f d3499aacbbef9d2acbb0e9cdbac51c0ea620e91b9893d46ceda49d5344357c61
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // withDockerServer
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code -1
Finished: FAILURE

Example below. I made 3 'sh' steps:

sh 'pwd'
sh 'sleep 1'
sh 'uname'

and log:

[Pipeline] {
[Pipeline] sh
[pipeline-docker-test] Running shell script

+ pwd
/var/jenkins_home/workspace/pipeline-docker-test
[Pipeline] sh

[pipeline-docker-test] Running shell script

+ sleep 1

[Pipeline] }
$ docker stop 93ea34143c0a9c7e28b6e580d620f33bcc8f0e0081e7885ea63ef9ad0d1fc57e
$ docker rm -f 93ea34143c0a9c7e28b6e580d620f33bcc8f0e0081e7885ea63ef9ad0d1fc57e
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // withDockerServer
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code -1
Finished: FAILURE
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)
Atlassian logo

z.jasinski@samsung.com (JIRA)

unread,
Jun 6, 2016, 6:47:01 AM6/6/16
to jenkinsc...@googlegroups.com
zbigniew jasinski updated an issue
Change By: zbigniew jasinski
Environment: jenkins 2.8, gerrit-trigger 2.21.0,    docker-workflow 1.4, pipeline 2.1

z.jasinski@samsung.com (JIRA)

unread,
Jun 15, 2016, 8:38:01 AM6/15/16
to jenkinsc...@googlegroups.com
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo

z.jasinski@samsung.com (JIRA)

unread,
Jul 1, 2016, 7:47:01 AM7/1/16
to jenkinsc...@googlegroups.com
zbigniew jasinski updated an issue
Change By: zbigniew jasinski
Component/s: docker-workflow-plugin

z.jasinski@samsung.com (JIRA)

unread,
Jul 1, 2016, 7:47:01 AM7/1/16
to jenkinsc...@googlegroups.com
 
Re: Workflow shell step ERROR: script returned exit code -1

I've run my configuration without using dockers and it works. So it seems that docker-workflow issue.

z.jasinski@samsung.com (JIRA)

unread,
Jul 1, 2016, 7:47:01 AM7/1/16
to jenkinsc...@googlegroups.com
zbigniew jasinski assigned an issue to Jesse Glick
 
Change By: zbigniew jasinski
Assignee: rsandell Jesse Glick

z.jasinski@samsung.com (JIRA)

unread,
Jul 1, 2016, 7:54:01 AM7/1/16
to jenkinsc...@googlegroups.com
zbigniew jasinski updated an issue
Change By: zbigniew jasinski
Environment: jenkins 2. 8 11 , gerrit-trigger 2.21. 0 1 , docker-workflow 1. 4 5 , pipeline 2. 1 2

jglick@cloudbees.com (JIRA)

unread,
Jul 25, 2016, 1:55:02 PM7/25/16
to jenkinsc...@googlegroups.com
Jesse Glick updated an issue
Change By: Jesse Glick
Component/s: gerrit-trigger-plugin
Component/s: workflow-plugin

jglick@cloudbees.com (JIRA)

unread,
Jul 25, 2016, 1:56:01 PM7/25/16
to jenkinsc...@googlegroups.com
Jesse Glick commented on Bug JENKINS-35370
 
Re: Workflow shell step ERROR: script returned exit code -1

Make sure you have updated all relevant plugins, especially durable-task, as there have been diagnostic improvements. Typically the problem is that the Docker daemon does not share a filesystem with the client, which is a prerequisite for Image.inside to work.

jglick@cloudbees.com (JIRA)

unread,
Sep 6, 2016, 10:01:13 AM9/6/16
to jenkinsc...@googlegroups.com
Jesse Glick resolved as Incomplete
 

Not enough information here.
Change By: Jesse Glick
Status: Open Resolved
Resolution: Incomplete

ben@mathews2000.com (JIRA)

unread,
Oct 31, 2016, 1:27:02 PM10/31/16
to jenkinsc...@googlegroups.com
Ben Mathews commented on Bug JENKINS-35370
 
Re: Workflow shell step ERROR: script returned exit code -1

I'm seeing a similar issue where the sh command inside a docker.image block always terminates after some delay. The point in the sh command that jenkins errors out varies which leads me to believe that jenkins is causing the problem. I'm on the latest version of jenkins and all plugins. Any tips on how to debug this?

jglick@cloudbees.com (JIRA)

unread,
Oct 31, 2016, 2:43:01 PM10/31/16
to jenkinsc...@googlegroups.com

Typically it means workspace sharing between host & container is failing. If the agent itself is inside a container, make sure --volumes-from is included on the command line; if not, nothing will work.

ben@mathews2000.com (JIRA)

unread,
Nov 1, 2016, 5:03:02 PM11/1/16
to jenkinsc...@googlegroups.com

The workspace is getting mounted (--volumes-from param is present) and is being interacted with. I've got a shell script that should take a couple minutes to complete. But within a couple seconds, the script will terminate and jenkins returns the above mentioned 

ERROR: script returned exit code -1
Finished: FAILURE

The Jenkins log has the call stack

hudson.AbortException: script returned exit code -1
	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.check(DurableTaskStep.java:285)
	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.run(DurableTaskStep.java:234)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Looking at the pipeline code in the call stack, it appears that Jenkins thinks my process is exiting. But (a) it never fails when I launch the process directly w/ "docker run ..." and (b) it fails at a different spot every time.

I've replaced my script w/ "for i in `seq 1 20`; do echo $i;date;sleep 5;done" and it never fails. So, it is apparent that something in the interaction between jenkins and my script is failing.

FWIW, the script is a series of "python setup.py develop" commands.

jglick@cloudbees.com (JIRA)

unread,
Nov 1, 2016, 5:50:02 PM11/1/16
to jenkinsc...@googlegroups.com

Ben Mathews Hmm, does not sound like a familiar issue. The fake exit status -1 means that Jenkins cannot find the PID of the controller sh script which tracks the output and exit code of your actual script (also sh, unless you specified a #!/bin/command). The typical reason for the failure to find this PID when using Image.inside is that the container does not share the right mount with the agent, for example because --volumes-from was not passed when it should have been. But in your case it was, and you say other shell scripts work, so something trickier is happening. If you can narrow it down to a reproducible test case, that would help a lot of course. Otherwise you will need to inspect the process tree inside the container, to see if the wrapper script is really still running or not; and inspect the …job@tmp sibling workspace that holds the control directory with the PID, output, and exit code.

ben@mathews2000.com (JIRA)

unread,
Nov 3, 2016, 1:12:03 PM11/3/16
to jenkinsc...@googlegroups.com

An update....

I cut the problem down to this repo

stage('run unit tests') {
      node() {
        docker.image("centos:6.7").inside() {
          sh 'for i in `seq 1 50`; do echo $i;date;sleep 2;done'
        }
}
}

This fails on both a docker using https://hub.docker.com/r/axltxl/jenkins-dood/ and jenkins installed on my ubuntu laptop via apt-get.

Before creating a ticket, I made one final test on our production jenkins server and it passed with flying colors four times consecutively. Obviously, something is strange with my box. Not sure what though. Any ideas would be appreciated. I'll keep poking around

$ uname -a
Linux mathewslaptop 4.4.0-45-generic #66-Ubuntu SMP Wed Oct 19 14:12:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/os-release 
NAME="Ubuntu"
VERSION="16.04.1 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.1 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
UBUNTU_CODENAME=xenial
$ docker -v
Docker version 1.12.1, build 23cf638

jglick@cloudbees.com (JIRA)

unread,
Nov 8, 2016, 1:02:01 PM11/8/16
to jenkinsc...@googlegroups.com

Are you using a (remote) agent on the production server but not in the test environment, or vice-versa? Generally should not matter except to the extent that it is the functioning of the docker command on the agent where the build is running which matters.

ben@mathews2000.com (JIRA)

unread,
Dec 24, 2016, 11:37:02 AM12/24/16
to jenkinsc...@googlegroups.com

The production server is running the builds on a centos 7 slave. My test environment is running jenkins in a docker container that is mounting /var/run/docker.sock from my laptop.

rodrigc@FreeBSD.org (JIRA)

unread,
Mar 15, 2017, 9:59:01 PM3/15/17
to jenkinsc...@googlegroups.com

Ben Mathews did you eventually solve this problem?  I'm seeing the same problem

where sh scripts are failing with -1.

 

I am also using:

docker.image(....).inside {

}

 

As a further note, my Jenkins instance is also a docker container.

 
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo

ben@mathews2000.com (JIRA)

unread,
Mar 16, 2017, 1:14:02 PM3/16/17
to jenkinsc...@googlegroups.com

Sorry, I should have followed up back in Dec. I've forgotten what happened with this. I stopped trying to do development locally, so I probably never resolved it.

rodrigc@FreeBSD.org (JIRA)

unread,
Mar 16, 2017, 3:21:02 PM3/16/17
to jenkinsc...@googlegroups.com

Ben Mathews fair enough.  I'm running into the same problem so was just curious.

My understanding of the problem is that if your Jenkins server is running inside a Docker container, and then you try to start a docker container on the same server (so you are doing Docker container inside Docker container), then that doesn't work so well.

The Durable Task plugin used by the pipeline **sh step has some complicated logic for how it figures out the process ID (pid) of the shell script that has been executed, and this logic gets confused when you do Docker inside Docker, and returns -1, even though your shell script is still running.  I ran into the -1 problems in JENKINS-32264 , (not in a Docker context).

 

rodrigc@FreeBSD.org (JIRA)

unread,
Mar 20, 2017, 7:29:02 PM3/20/17
to jenkinsc...@googlegroups.com
Craig Rodrigues reopened an issue
 

I have been having the same problems as Ben Mathews with the shell script returning -1 if it is run inside a docker.inside block in a pipeline.

I took Ben's testcase here: https://issues.jenkins-ci.org/browse/JENKINS-35370?focusedCommentId=275803&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-275803

and could reproduce the problem.

 

The environment I used to reproduce the problem was:

  • host environment was a VM running Debian stretch
  • docker version:Client:

Version:      1.13.1

API version:  1.26

Go version:   go1.7.5

Git commit:   092cba372

Built:        Wed Feb  8 06:44:30 2017

OS/Arch:      linux/amd64

 

Server:

Version:      1.13.1

API version:  1.26 (minimum version 1.12)

Go version:   go1.7.5
Git commit:   092cba372
Built:        Wed Feb  8 06:44:30 2017
OS/Arch:      linux/amd64

So the host environment, and the docker container are on the same machine and sharing the same file system.
Change By: Craig Rodrigues
Resolution: Incomplete
Status: Resolved Reopened

rodrigc@FreeBSD.org (JIRA)

unread,
Mar 20, 2017, 7:32:04 PM3/20/17
to jenkinsc...@googlegroups.com
 
Re: Workflow shell step ERROR: script returned exit code -1

I'm also using:

  • Jenkins 2.51
  • Pipeline 2.5
  • Durable task plugin 1.13
  • Docker pipeline 1.10

cstroe@gmail.com (JIRA)

unread,
Jun 8, 2018, 12:31:03 AM6/8/18
to jenkinsc...@googlegroups.com

I'm seeing this issue in Jenkins ver. 2.63.  I am running a python script inside a docker container and getting:

...
[Pipeline] stage
[Pipeline] { (Publish Classes)
[Pipeline] sh
[***] Running shell script
+ ./post_classes.py
...
ERROR: script returned exit code -1
Finished: FAILURE

I've tried to reproduce it with a simpler script, but I can't.  It happens only with certain builds.

cstroe@gmail.com (JIRA)

unread,
Jun 8, 2018, 12:50:05 AM6/8/18
to jenkinsc...@googlegroups.com
Cosmin Stroe edited a comment on Bug JENKINS-35370
I'm seeing this issue in Jenkins ver. 2.63.  I am running a python script inside a docker container and getting:

{code:java}

...
[Pipeline] stage
[Pipeline] { (Publish Classes)
[Pipeline] sh
[***] Running shell script
+ ./post_classes.py
...
ERROR: script returned exit code -1
Finished: FAILURE{code}

I've tried to reproduce it with a simpler script, but I can't.  It happens only with certain builds.


 

Update:

The docker container I was building didn't contain {{ps}} (thank you [~iiosifidis] for your message above).  After adding ps, it fixed the issue.

wei.lan@zte.com.cn (JIRA)

unread,
Sep 26, 2018, 2:39:03 AM9/26/18
to jenkinsc...@googlegroups.com
wei lan commented on Bug JENKINS-35370

I had the same issue，and I had checked my docker container，“ps” command can run in it，so I thought it had no relation with the cmmand “ps” installed in docker image，any other solutions？
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

jglick@cloudbees.com (JIRA)

unread,
Sep 28, 2018, 2:48:03 PM9/28/18
to jenkinsc...@googlegroups.com

There could be many reasons, not necessarily related to one another.
Reply all
Reply to author
Forward
0 new messages