[JIRA] (JENKINS-59907) sh steps stuck indefinitely on uncommon architectures (e.g. s390x)

38 views
Skip to first unread message

jakub.loucky@gmail.com (JIRA)

unread,
Oct 24, 2019, 5:03:02 AM10/24/19
to jenkinsc...@googlegroups.com
Jakub L updated an issue
 
Jenkins / Bug JENKINS-59907
sh steps stuck indefinitely on uncommon architectures (e.g. s390x)
Change By: Jakub L
Summary: sh steps stuck indefinitely after upgrading durable-task to v1 on uncommon architectures (e . 31 g. s390x)
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

psu.ftbl@gmail.com (JIRA)

unread,
Oct 25, 2019, 8:28:03 AM10/25/19
to jenkinsc...@googlegroups.com
Avy Harvey commented on Bug JENKINS-59907
 
Re: sh steps stuck indefinitely on uncommon architectures (e.g. s390x)

I'm seeing this issue on FreeBSD 11.1 as well.

 

$ ./durable_task_monitor_1.31_unix_64
ELF binary type "0" not known. 
bash: ./durable_task_monitor_1.31_unix_64: cannot execute binary file: Exec format error

Downgrading to version 1.30 works.

 

 

psu.ftbl@gmail.com (JIRA)

unread,
Oct 25, 2019, 8:30:01 AM10/25/19
to jenkinsc...@googlegroups.com
Avy Harvey edited a comment on Bug JENKINS-59907
I'm seeing this issue on FreeBSD 11.1 (64-bit) as well.

 
{code:java}

$ ./durable_task_monitor_1.31_unix_64
ELF binary type "0" not known.
bash: ./durable_task_monitor_1.31_unix_64: cannot execute binary file: Exec format error
{code}

Downgrading to version 1.30 works.

 

 

cchiou@cloudbees.com (JIRA)

unread,
Oct 27, 2019, 11:35:02 PM10/27/19
to jenkinsc...@googlegroups.com

cchiou@cloudbees.com (JIRA)

unread,
Oct 28, 2019, 3:56:07 AM10/28/19
to jenkinsc...@googlegroups.com
Carroll Chiou commented on Bug JENKINS-59907
 
Re: sh steps stuck indefinitely on uncommon architectures (e.g. s390x)

Sorry this took so long to address. The binary was not intended to run on non-x86 architectures. Instead, when a non-x86 and non-*NIX architecture is detected, the original shell wrapper was supposed to launch the script. I have a PR up right now that is changing that behavior. I will also update the changelog (that is currently being migrated to github) for this information.

The PR can be found here: https://github.com/jenkinsci/durable-task-plugin/pull/114

reiner.wirtz@ser.de (JIRA)

unread,
Oct 28, 2019, 10:04:02 AM10/28/19
to jenkinsc...@googlegroups.com

We have a similar problem on AIX 7.2.
The binary to start is of type ELF.

cchiou@cloudbees.com (JIRA)

unread,
Oct 28, 2019, 1:36:03 PM10/28/19
to jenkinsc...@googlegroups.com
Carroll Chiou edited a comment on Bug JENKINS-59907
Sorry this took so long to address. The binary was not intended to run on non-x86 architectures. Instead, when  a non-x86 and non-*NIX architecture is detected, the original shell wrapper was supposed to launch the script. I have  a PR up right now that  is changing that behavior. I will also update the changelog (that  is currently being migrated to github) for  this  information.

The PR can be found here: https://github.com/jenkinsci/durable-task-plugin/pull/114


UPDATE: more work is being done to this PR to handle a few more cases (such as freebsd)

rafi@guengel.ch (JIRA)

unread,
Oct 29, 2019, 6:38:02 AM10/29/19
to jenkinsc...@googlegroups.com

Just for the record, this issue will also affect Solaris, NetBSD and OpenBSD, regardless of the architecture.

As a side note, the name of the binary is misleading. It should be called durable_task_monitor_X.YZ_linux_amd64, because that is the only OS it can run natively.

jesse@novi.systems (JIRA)

unread,
Oct 29, 2019, 9:37:02 AM10/29/19
to jenkinsc...@googlegroups.com

I am running an agent on ppc64le and ran into this issue.  I will attempt to revert to 1.3.0.

Thank you, Jenkins team, for your ongoing support.

cchiou@cloudbees.com (JIRA)

unread,
Oct 29, 2019, 5:26:02 PM10/29/19
to jenkinsc...@googlegroups.com
Carroll Chiou started work on Bug JENKINS-59907
 
Change By: Carroll Chiou
Status: Open In Progress

cchiou@cloudbees.com (JIRA)

unread,
Oct 29, 2019, 5:29:02 PM10/29/19
to jenkinsc...@googlegroups.com

Version 1.33 has now been released. There is stricter checking on the platforms it runs on. I know not every case has been covered here. The binary is disabled by default so behavior should be simliar, if not same to 1.30

jakub.loucky@gmail.com (JIRA)

unread,
Oct 31, 2019, 2:18:03 PM10/31/19
to jenkinsc...@googlegroups.com
Jakub L commented on Bug JENKINS-59907

I can confirm that v1.33 works fine on s390x. Thanks for the fix!

cchiou@cloudbees.com (JIRA)

unread,
Oct 31, 2019, 2:23:04 PM10/31/19
to jenkinsc...@googlegroups.com
Carroll Chiou closed an issue as Incomplete
 
Change By: Carroll Chiou
Status: In Progress Closed
Resolution: Incomplete

cchiou@cloudbees.com (JIRA)

unread,
Oct 31, 2019, 2:24:06 PM10/31/19
to jenkinsc...@googlegroups.com
Carroll Chiou reopened an issue
Change By: Carroll Chiou
Resolution: Incomplete
Status: Closed Reopened

cchiou@cloudbees.com (JIRA)

unread,
Oct 31, 2019, 2:24:06 PM10/31/19
to jenkinsc...@googlegroups.com
Change By: Carroll Chiou
Status: Fixed but Unreleased Resolved
Released As: 1.33

cchiou@cloudbees.com (JIRA)

unread,
Oct 31, 2019, 2:24:07 PM10/31/19
to jenkinsc...@googlegroups.com
Change By: Carroll Chiou
Status: Reopened Fixed but Unreleased
Resolution: Fixed

rrajcorp@gmail.com (JIRA)

unread,
Feb 28, 2020, 6:28:04 AM2/28/20
to jenkinsc...@googlegroups.com
Rahul Raj reopened an issue
 

The issue is still there with v1.33

logs:

https://gist.github.com/rahul-raj/ddeaa1407827f191e0d1b94966a58a0b

Please suggest on a fix for this. 

 

 

 

 

 

Change By: Rahul Raj
Resolution: Fixed
Status: Resolved Reopened
This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38)
Atlassian logo

rrajcorp@gmail.com (JIRA)

unread,
Feb 28, 2020, 6:30:03 AM2/28/20
to jenkinsc...@googlegroups.com
The issue is still there with v1.33

logs:

[https://gist.github.com/rahul-raj/ddeaa1407827f191e0d1b94966a58a0b]


!image Version from my Jenkins plugin list:
[https://user
- 2020 images.githubusercontent.com/517415/75545150 - 02 8a721200 - 28 5a4b - 16 11ea - 56 8b49 - 57-930 02d69669184f .png |width=592,height=283! ]

 

Please suggest on a fix for this. 

 

 

 

 

 

cchiou@cloudbees.com (JIRA)

unread,
Feb 28, 2020, 5:05:04 PM2/28/20
to jenkinsc...@googlegroups.com

Rahul Raj it appears you are using x86, and not an "uncommon architecture" like this ticket is describing. I would probably advise setting LAUNCH_DIAGNOSTICS=true as suggested in the output log and that can tell us better. The default behavior for this plugin should be using the original script wrappers. If you can't ascertain what is going there, I would probably post this to jenkinsci-users mailing list while we're still investigating.

don@hardwarehacks.org (JIRA)

unread,
Mar 6, 2020, 11:07:03 AM3/6/20
to jenkinsc...@googlegroups.com
Don L commented on Bug JENKINS-59907

I've found this also reproduces when using build agents in Kubernetes. The problem here is that Kubernetes launches two containers into a pod with a shared mount: a JNLP slave container, which Jenkins does have permission to write the cache directory in, and a build container (in my case kubectl, but could be any container without a Jenkins user) where it does not necessarily have the same permission, in which code actually runs. The plugin runs its test inside the JNLP container, enables the wrapper, and then exhibits the same hanging behavior when commands are run in the kubectl container.

In the JNLP container:

bash-4.4$ cd /home/jenkins/agent/caches
bash-4.4$ ls -l
total 0
drwxr-xr-x    2 jenkins  jenkins          6 Mar  6 15:47 durable-task 

In the kubectl container:

I have no name!@<REDACTED>:/home/jenkins/agent/caches$ ls -l
total 0
drwxr-xr-x 2 1000 1000 6 Mar  6 15:47 durable-task

I have no name!@<REDACTED>:/home/jenkins/agent/caches$ id
uid=1001 gid=0(root) groups=0(root)

cchiou@cloudbees.com (JIRA)

unread,
Mar 6, 2020, 2:18:04 PM3/6/20
to jenkinsc...@googlegroups.com

Don L can we move this over to JENKINS-59903? That would be the relevant ticket. Could you also confirm that you are running v1.33? and not v1.31-32

don@hardwarehacks.org (JIRA)

unread,
Mar 6, 2020, 2:43:03 PM3/6/20
to jenkinsc...@googlegroups.com

matthew.pigram2@gmail.com (JIRA)

unread,
Mar 11, 2020, 5:37:03 PM3/11/20
to jenkinsc...@googlegroups.com

I'm still running into this same issue, whilst trying to run a pyinstaller docker image.

The Sh command gives me the following:

```

ERROR: The container started but didn't run the expected command. Please double check your ENTRYPOINT does execute the command passed as docker run argument, as required by official docker images (see https://github.com/docker-library/official-images#consistency for entrypoint consistency requirements).
Alternatively you can force image entrypoint to be disabled by adding option `--entrypoint=''`.
[Pipeline]

{ [Pipeline] sh process apparently never started in /var/jenkins_home/workspace/simple-python-pyinstaller-app@tmp/durable-87eb5f90 (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer) [Pipeline] }

$ docker stop --time=1 0d2e194b04bb5a12016da7f4dd92019127837debf082d18ba9fdf4cbbf6abbd7
$ docker rm -f 0d2e194b04bb5a12016da7f4dd92019127837debf082d18ba9fdf4cbbf6abbd7
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] End of Pipeline
ERROR: script returned exit code -2
Finished: FAILURE

```

Running latest jenkins/blueocean image and I'm currently using the following set up for my Jenkinsfile.

 

```

pipeline {
agent none
options {| |skipStagesAfterUnstable()| |}
stages {
stage('Build') {
agent {
docker {| |image 'python:2-alpine'| |}
}
steps {| |sh 'python -m py_compile sources/add2vals.py sources/calc.py'| |}
}
stage('Test') {
agent {
docker {| |image 'qnib/pytest'| |}
}
steps {| |sh 'py.test --verbose --junit-xml test-reports/results.xml sources/test_calc.py'| |}
post {
always {| |junit 'test-reports/results.xml'| |}
}
}
stage('Deliver') {
agent {
docker {| |image 'cdrx/pyinstaller-linux:python2'| |args 'docker run -v "/var/jenkins_home/workspace/simple-python-pyinstaller-app/sources:/src/" --name pyinstaller --entrypoint= cdrx/pyinstaller-linux:python2'| |}
}
steps {| |sh 'pyinstaller --onefile sources/add2vals.py'| |}
post {
success {| |archiveArtifacts 'dist/add2vals'| |}
}
}
}

}

```

 

Reply all
Reply to author
Forward
0 new messages