[JIRA] (JENKINS-44930) Allow sh to return exit status, stdout and stderr all at once

4,306 views
Skip to first unread message

mellery451@gmail.com (JIRA)

unread,
Feb 13, 2018, 1:41:02 PM2/13/18
to jenkinsc...@googlegroups.com
Mike Ellery commented on Improvement JENKINS-44930
 
Re: Allow sh to return exit status, stdout and stderr all at once

I would like to point out that this would be particularly useful for powershell scripts since the proposed workaround of "just redirect your stuff to a file" just doesn't work in powershell. I've tried umpteen different ways to get powershell to just log all my output and it won't have any of it. Start/Stop Transcript, classic redirection, none of them work consistently or reliably in the jenkins pipeline. Just having the ability to capture the output AND the return code to know if we failed would be outstanding. As it stands right now, I either get my output or I get an exception and I have to do some pipeline trickery to handle both cases. Ugh.

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo

me+jenkins@childno.de (JIRA)

unread,
Feb 14, 2018, 5:33:03 AM2/14/18
to jenkinsc...@googlegroups.com

Mike Ellery: just a question: isn't the powershell (introduced by JENKINS-34581) step more suitable for your needs? https://github.com/jenkinsci/workflow-durable-task-step-plugin/blob/master/src/main/java/org/jenkinsci/plugins/workflow/steps/durable_task/PowershellScriptStep.java

If yes, I see a different issue, that there is no functional parity between sh and powershell

mellery451@gmail.com (JIRA)

unread,
Feb 14, 2018, 11:11:03 AM2/14/18
to jenkinsc...@googlegroups.com

Marcel: I'm using `powershell` in a pipeline, as described here: https://jenkins.io/doc/pipeline/steps/workflow-durable-task-step/#code-powershell-code-powershell-script. I think it mostly has parity with the `sh` step. My comment above was mainly in response to the workaround proposed in 26133 (to just redirect output from your shell to file). That's easy to do in bash and not so easy to do in powershell, at least not the way it's currently wrapped in jenkins - that has been my experience so far. Thanks.

mark.pettigrew@harris.com (JIRA)

unread,
Jun 7, 2018, 11:26:05 AM6/7/18
to jenkinsc...@googlegroups.com

+1

 

Trying to think of a competing CI tool or programming language that doesn't have this obvious feature.  returnExit OR stdout thoroughly violates the principle of least surprise.

leemeador@java.net (JIRA)

unread,
Sep 24, 2018, 7:25:04 PM9/24/18
to jenkinsc...@googlegroups.com

One solution is to have sh() return the output in the case of returnStdout: true as usual and, in the case of failure, throw an exception that could be a subclass of the current failure exception from which you could extract the error return code if you catch it and the console output is present in the exception as well (so you could extract it too).

 

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

fnasser@redhat.com (JIRA)

unread,
Nov 2, 2018, 4:12:03 PM11/2/18
to jenkinsc...@googlegroups.com

Simple solution, create a returnStderr: true

As the empty String (i.e., nothing in stderr) would mean false, it would indicate "no errors"

If it is not empty, one not only knows it failed but also knows WHY it failed.

 
String status = sh(returnStderr: true, script: "git merge --no-edit $branches > merge_output.txt")if (status) {
currentBuild.result = 'FAILED' def output = readFile('merge_output.txt').trim()
slackSend channel: SLACK_CHANNEL, message: "<${env.JOB_URL}|${env.JOB_NAME}> ran into an error merging the PR branches into the ${TARGET_BRANCH} branch:\n```\n${output}\n```\n<${env.BUILD_URL}/console|See the full output>", color: 'warning', tokenCredentialId: 'slack-token' error status}
sh 'rm merge_output.txt'

fnasser@redhat.com (JIRA)

unread,
Nov 2, 2018, 4:13:02 PM11/2/18
to jenkinsc...@googlegroups.com
Fernando Nasser edited a comment on Improvement JENKINS-44930
Simple solution, create a returnStderr: true

As the empty String (i.e., nothing in stderr) would mean false, it would indicate "no errors"

If it is not empty, one not only knows it failed but also knows WHY it failed.

 
{code:java}
String status = sh(returnStderr: true, script: "git merge --no-edit $branches > merge_output.txt")
if (status) {
  currentBuild.result = 'FAILED'

  def output = readFile('merge_output.txt').trim()
  slackSend channel: SLACK_CHANNEL, message: "<${env.JOB_URL}|${env.JOB_NAME}> ran into an error merging the PR branches into the ${TARGET_BRANCH} branch:\n```\n${output}\n```\n<${env.BUILD_URL}/console|See the full output>", color: 'warning', tokenCredentialId: 'slack-token'

  error status
}
sh 'rm merge_output.txt'
{code}

fnasser@redhat.com (JIRA)

unread,
Nov 28, 2018, 11:04:06 AM11/28/18
to jenkinsc...@googlegroups.com

Another option:

def status = sh(returnStatus: true, returnStdout: true, returnStderr: true, script: "git merge --no-edit $branches")

Would return a Map, which can be seen as

{ 'rc': <return code value>, 'stdout': <string with stdout content>, 'stderr': <string with stderr content> }

A compromise would be to have both stderr and stdout together (easy to capture?)

{ 'rc': <return code value>, 'output': <string with stdout+stderr contents> }

P.S.: The 'returnStderr' does not exist yet, I'll file a JIRA for it as it is useful independently.

mdealer@gmail.com (JIRA)

unread,
Jan 10, 2019, 3:15:09 AM1/10/19
to jenkinsc...@googlegroups.com

Inability to return both status and stdout is a huge drawback for us. Separate stdout and stderr would be a plus.

Once we want both it could just return a map instead of a value. This seems trivial to implement and won't break anything as returning both doesn't work already.

mdealer@gmail.com (JIRA)

unread,
Jan 10, 2019, 3:15:09 AM1/10/19
to jenkinsc...@googlegroups.com
Edgars Batna edited a comment on Improvement JENKINS-44930
This should also be implemented for Windows batch 'bat' step. Inability to return both status and stdout is a huge drawback for us. Separate stdout and stderr would be a plus.


Once we want both it could just return a map instead of a value. This seems trivial to implement and won't break anything as returning both doesn't work already.

sverre.moe@gmail.com (JIRA)

unread,
Jan 10, 2019, 2:48:07 PM1/10/19
to jenkinsc...@googlegroups.com
Sverre Moe edited a comment on Improvement JENKINS-44930
It would be better using a specific POJO than a Map.
{code}
def returnObj = sh(returnStatus: true, returnStdout: true, returnStderr: true, script: "git merge --no-edit $branches")
def status = returnObj.getStatus()
def stdout = returnObj.getStdout()
{code}

Using a Map you would need to know the
values keys therein, but with a POJO you can use the javadoc to know the getters.

sverre.moe@gmail.com (JIRA)

unread,
Jan 10, 2019, 2:48:07 PM1/10/19
to jenkinsc...@googlegroups.com

It would be better using a specific POJO than a Map.

def returnObj = sh(returnStatus: true, returnStdout: true, returnStderr: true, script: "git merge --no-edit $branches")
def status = returnObj.getStatus()
def stdout = returnObj.getStdout()

Using a Map you would need to know the values therein, but with a POJO you can use the javadoc to know the getters.

sverre.moe@gmail.com (JIRA)

unread,
Jan 10, 2019, 2:51:10 PM1/10/19
to jenkinsc...@googlegroups.com
Sverre Moe edited a comment on Improvement JENKINS-44930
It would be better using a specific POJO than a Map.
{code}

def returnObj = sh(returnStatus: true, returnStdout: true, returnStderr: true, script: "git merge --no-edit $branches")
def status = returnObj.getStatus()
def stdout = returnObj.getStdout()
{code}

Using a Map you would need to know the keys therein, but with a POJO you can use the javadoc to know the getters.

Perhaps the parameter flags returnStatus, returnStdout, returnStderr would not be necessary. It would just return the POJO, and you could either use it or not.

sverre.moe@gmail.com (JIRA)

unread,
Jan 10, 2019, 2:51:10 PM1/10/19
to jenkinsc...@googlegroups.com
Sverre Moe edited a comment on Improvement JENKINS-44930
It would be better using a specific POJO than a Map.
{code}
def returnObj = sh(returnStatus: true, returnStdout: true, returnStderr: true, script: "git merge --no-edit $branches")
def status = returnObj.getStatus()
def stdout = returnObj.getStdout()
def stderr = returnObj.getStderr()
{code}

Using a Map you would need to know the keys therein, but with a POJO you can use the javadoc to know the getters.
Perhaps the parameter flags returnStatus, returnStdout, returnStderr would not be necessary. It would just return the POJO, and you could either use it or not.

sverre.moe@gmail.com (JIRA)

unread,
Jan 10, 2019, 2:52:03 PM1/10/19
to jenkinsc...@googlegroups.com
Sverre Moe edited a comment on Improvement JENKINS-44930
It would be better using a specific POJO than a Map.
{code}
def returnObj = sh(returnStatus: true, returnStdout: true, returnStderr: true, script: "git merge --no-edit $branches")
def status = returnObj.getStatus()
def stdout = returnObj.getStdout()
def stderr = returnObj.getStderr()
{code}

Using a Map you would need to know the keys therein, but with a POJO you can use the javadoc to know the getters.
Perhaps the parameter flags returnStatus, returnStdout, returnStderr would not be necessary. It would could just return the POJO, and you could either use it or not.

fnasser@redhat.com (JIRA)

unread,
Jan 10, 2019, 3:46:03 PM1/10/19
to jenkinsc...@googlegroups.com

I'd rather keep the switches to avoid unnecessary overhead in the cases these are not necessary.

benjamin.heilbrunn@sap.com (JIRA)

unread,
Jan 23, 2019, 11:42:13 AM1/23/19
to jenkinsc...@googlegroups.com

We already encountered multiple use cases where we needed to parse STDOUT in case a certain exit code happened. Would be great to have this feature.

ramkumar.bangaru@gmail.com (JIRA)

unread,
Jan 29, 2019, 11:47:07 AM1/29/19
to jenkinsc...@googlegroups.com

In my case, I need it as my pipeline scripts need to branch out to different path based on the status and if the status is successful, based on the output of it. It makes a lot of sense to have both status and output returned by this function. 

juan.perezesteban@gmail.com (JIRA)

unread,
Feb 5, 2019, 12:56:03 PM2/5/19
to jenkinsc...@googlegroups.com

We are using lots of python code called from the shared library, is a pain sh only returns an error code we would need a way to bubble the error message. Those solutions sounds good.

jenkins-trik@conn.cx (JIRA)

unread,
Apr 9, 2019, 6:21:04 PM4/9/19
to jenkinsc...@googlegroups.com

Just sharing my workaround for others:

pipeline {
  agent {
    docker {
      image 'debian'
    }
  }

  stages {
    stage('script') {
      steps {
        script {
          status = sh(
            returnStatus: true,

            script: '''#!/bin/bash
              exec > >(tee output.log) 2>&1
              echo 'one: stdout'
              >&2 echo 'one: stderr'
              exit 1
            '''
          )

          output = readFile('output.log').trim()
          echo output

          if (status != 0) {
            currentBuild.result = 'UNSTABLE'
          }
        }
      }
    }
  }

  post {
    cleanup {
      deleteDir()
    }
  }
}

[Pipeline] {
[Pipeline] stage
[Pipeline] { (script)
[Pipeline] script
[Pipeline]

Unknown macro: { [Pipeline] sh one}

[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline]

Unknown macro: { (Declarative}

[Pipeline] // stage
[Pipeline] }
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: UNSTABLE

jenkins-trik@conn.cx (JIRA)

unread,
Apr 9, 2019, 6:22:11 PM4/9/19
to jenkinsc...@googlegroups.com
JD Friedrikson edited a comment on Improvement JENKINS-44930
Just sharing my workaround for others:
{code:groovy}

pipeline {
  agent {
    docker {
      image 'debian'
    }
  }

  stages {
    stage('script') {
      steps {
        script {
          status = sh(
            returnStatus: true,

            script: '''#!/bin/bash
              exec > >(tee output.log) 2>&1
              echo 'one: stdout'
              >&2 echo 'one: stderr'
              exit 1
            '''
          )

          output = readFile('output.log').trim()
          echo output

          if (status != 0) {
            currentBuild.result = 'UNSTABLE'
          }
        }
      }
    }
  }

  post {
    cleanup {
      deleteDir()
    }
  }
}
{code}

{code
:text }

[Pipeline] {
[Pipeline] stage
[Pipeline] { (script)
[Pipeline] script
[Pipeline] {
[Pipeline] sh
one: stdout
one: stderr
[Pipeline] readFile
[Pipeline] echo
one: stdout
one: stderr
[Pipeline] }

[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Declarative: Post Actions)
[Pipeline] deleteDir (show)

[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: UNSTABLE
{code}

jenkins-trik@conn.cx (JIRA)

unread,
Apr 9, 2019, 6:22:11 PM4/9/19
to jenkinsc...@googlegroups.com
{ panel code:text }
{ panel code }

giovanni.tirloni@gmail.com (JIRA)

unread,
Apr 11, 2019, 8:35:21 AM4/11/19
to jenkinsc...@googlegroups.com

By not making the Jenkins code "more complex" to support a common use case, Jenkins users everywhere need to make their code more complex. I'm certain the sum of "complexity" is much greater now than it would be if `sh` implemented this request. Talk about externalities.

nelhimoud@gmail.com (JIRA)

unread,
May 14, 2019, 2:59:14 AM5/14/19
to jenkinsc...@googlegroups.com

int exitStatus = sh(script: "curl -s -m ${timeoutSeconds} -w %{http_code} ${url}", returnStatus: true)
String httpStatus = sh(script: "curl -s -m ${timeoutSeconds} -w %{http_code} ${url}", returnStdout: true)

Curl: exitStatus, httpStatus. Chose one!

nelhimoud@gmail.com (JIRA)

unread,
May 14, 2019, 3:00:17 AM5/14/19
to jenkinsc...@googlegroups.com
Nils El-Himoud edited a comment on Improvement JENKINS-44930
{code:java}

int exitStatus = sh(script: "curl -s -m ${timeoutSeconds} -w %{http_code} ${url}", returnStatus: true)
String int httpStatus = sh(script: "curl -s -m ${timeoutSeconds} -w %{http_code} ${url}", returnStdout: true)

{code}

Curl: exitStatus, httpStatus. Chose one!

Aaron.Marasco@BIA-Boeing.com (JIRA)

unread,
May 14, 2019, 6:53:05 AM5/14/19
to jenkinsc...@googlegroups.com

Nils El-Himoud where are you going with that? We don't want to run every command twice.

mahmoud.al-ashi@intel.com (JIRA)

unread,
May 24, 2019, 10:14:03 AM5/24/19
to jenkinsc...@googlegroups.com

I believe command "sh" should always return an object with all relevant output (stdout, stderr, exit_code) and at the same time it should also print the output live to the console unless some argument (no_stdout) or so is given. This way you can access all at once.

The other problem that I see now is that when "returnStdout" is provided, the output is not printed to the console. I would prefer to print the output to the console live anyway but also get the output to parse it or do something special with it.

nolange79@gmail.com (JIRA)

unread,
May 28, 2019, 5:32:08 AM5/28/19
to jenkinsc...@googlegroups.com

I would really like this to be supported, as the use of the warnings plugin is otherwise really cumbersome and invasive, see JENKINS-54832 for the issue and workaround.

The usecase of

  • using the returncode to fail a build
  • using the output for displaying progress
  • re-using the output later in further plugins

should be supported by Jenkins natively, and not require workarounds in the buildscripts

fnasser@redhat.com (JIRA)

unread,
May 28, 2019, 10:30:03 AM5/28/19
to jenkinsc...@googlegroups.com

Although I am one of the supporters for the implementation of a solution to this JIRA I have to object to a solution like:

" I believe command "sh" should always return an object with all relevant output (stdout, stderr, exit_code) "

Output of sysout and syserr may be too extensive, have all sort of characters, etc.  We had a system that included the output as one of the fields of a JSON output and that was always causing problems, requiring retrying, etc.

The workaround for this JIRA that we are using for quite some time and it is working very well is to, if output is requested, write it to a file in the WORSPACE.  We do it by piping the output (sysout and/or syserr) but that should be much clear if done by the command itself.

 

nolange79@gmail.com (JIRA)

unread,
May 28, 2019, 10:46:19 AM5/28/19
to jenkinsc...@googlegroups.com

Fernando Nasser: I would like and prefer a dump into a file into workspace, while also having normal output.

I am not sure what you mean by "command", you you mean the pipeline command ("sh"), or the native script?
The native script is cumbersome, there might even be different ways with different shells,
if you meant the pipeline command then I am all for it.

but some other idea would be to use a context wrappers like

output(stdout: "build.log",  stderr: "err.log") {
   sh 'cmake --build .'
}

surajsharma121@gmail.com (JIRA)

unread,
May 28, 2019, 6:50:04 PM5/28/19
to jenkinsc...@googlegroups.com

Maybe this small function can help take care of exit status and stdout.

def runCommand(script) {    
    echo "[runCommand:script] ${script}"

    def stdoutFile = "rc.${BUILD_NUMBER}.out"    
    script = script + " > " + stdoutFile

    def res = [:]    
    res["exitCode"] = sh(returnStatus: true, script: script)    
    res["stdout"] = sh(returnStdout: true, script: "cat " + stdoutFile)

    sh(returnStatus: true, script: "rm -f " + stdoutFile)

    echo "[runCommand:response] ${res}"    
    return res
}

surajsharma121@gmail.com (JIRA)

unread,
May 28, 2019, 6:52:21 PM5/28/19
to jenkinsc...@googlegroups.com
Suraj Sharma edited a comment on Improvement JENKINS-44930
Maybe this small function can help take care of exit status and stdout.
{code:java}

def runCommand(script) {    
    echo "[runCommand:script] ${script}"

    def stdoutFile = "rc.${BUILD_NUMBER}.out"    
    script = script + " > " + stdoutFile

    def res = [:]    
    res["exitCode"] = sh(returnStatus: true, script: script)    
    res["stdout"] = sh(returnStdout: true, script: "cat " + stdoutFile)

    sh(returnStatus: true, script: "rm -f " + stdoutFile)

    echo "[runCommand:response] ${res}"    
    return res
}
{code}

Example Usage
{code}
    def response = runCommand("date")
    echo response["exitCode"]
    echo response["stdout"]
{code}

fnasser@redhat.com (JIRA)

unread,
May 28, 2019, 8:25:09 PM5/28/19
to jenkinsc...@googlegroups.com

Norbert Lange   The "command" was in a quote, but yes, we were talking about the 'sh' step.

In fact we use "tee" after the '|' so we are also showing the command output in the log.  

I agree that the output should also go to the log.

akostadinov@java.net (JIRA)

unread,
May 30, 2019, 9:12:05 AM5/30/19
to jenkinsc...@googlegroups.com

Suraj Sharma, and if you have a multiline string that function works in a surprising way. It needs to be done on the lower level by the plugin to be reliable.
I actually wonder if we can have a console wrapper that will capture whatever output produced inside. Like AnsiColor Plugin. It will not be able to differentiate stdin from stderr though.

ogondza@gmail.com (JIRA)

unread,
Jun 5, 2019, 8:52:08 AM6/5/19
to jenkinsc...@googlegroups.com

david@famriemens.nl (JIRA)

unread,
Sep 25, 2019, 4:47:04 AM9/25/19
to jenkinsc...@googlegroups.com
David Riemens commented on Improvement JENKINS-44930
 
Re: Allow sh to return exit status, stdout and stderr all at once

I like the solution by Suraj Sharma for smaller outputs. However I have a few calls (using 'bat') that run quite long. I am looking to:

  • capture the exitcode to see if the step passed/failed
  • capture (grep) for a few specific lines in the output from stdout or stderr that contain more info.
    I want to store that info for reporting detailed results at the end
  • see the progress of the output 'live', rather than a single dump at the end.
    Yeah, write to file with 'tee' works, but there should be better solutions ... 

So although I'd vote for having the 'object' returned as described by Fernando Nasser it would be great if I would also be able to optionally pass a method that does the magic I need. In my case grep for some pattern in stdout, and store it such that it is available after the bat/sh step.

I've been looking at using a 'listener' as described in eg

   https://stackoverflow.com/questions/53172023/get-console-logger-or-tasklistener-from-pipeline-script-method

but have not found a (working) way yet to apply this in a scripted pipeline. Anyone

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

alexander.samoylov@gmail.com (JIRA)

unread,
Oct 21, 2019, 9:38:06 AM10/21/19
to jenkinsc...@googlegroups.com

Using a temp file is an ugly workaround which has at least these obvious disadvantages:
1. You never have 100% guarantee that you don't break parallelism. Every time you introduce one more parallel "axe" (such as release/debug conf, branch, os whatever) you must take care that the temp file has the corresponding suffix.
2. This method won't work in some cases if a command already contains output redirection. For example, on Linux this works: rm 123 > temp.txt 2>&1. I am not sure if we can do the same on Windows. There may be more complex cases with tricky double/single quote combinations and multiple output redirections, of a command which consists of several commands, semicolon separated. At the end we always loose generality and platform-independency if we use a temp file.
3. You must remove the temp file after the command execution, but how do you do it on the node if the pipeline stuff is executed on master? You cannot use Java method File.createTempFile() for this, because it creates the temp file on master. It means that the code which generates the file name must run on master and then you pass this name to the "sh" or "bat" when you remove it. Distinguishing to "sh" and "bat" is also losing of platform-independency by the way.

My current solution with which tried looks like this:

def getTempDirOnNode() {
    if (isUnix()) {
        return env.TMPDIR != null ? env.TMPDIR : '/tmp'
    } else {
         return env.TEMP
    }
}

/*  May not work if "cmd" already contains output redirection or more complex shell syntax. */
def runCmdOnNodeSavingExitCodeAndStdout(cmd) {
    def rc = 0
    def stdout = null
    print("Using temp directory: " + getTempDirOnNode())

    def tempFileName = 'runIgnoreExitCodeStdout_' + UUID.randomUUID() + '.txt'
    def tempFilePath = getTempDirNode() + getOSPathSep() + tempFileName
    def tempFileHandle = new File(tempFilePath)
    
    print("Using temp file: " + tempFilePath)
    
    if (isUnix()) {
        rc = sh(script: cmd + ' > ' + tempFilePath, returnStatus: true)
    } else {
        rc = bat(script: cmd + ' > ' + tempFilePath, returnStatus: true);
    }
    stdout = readFile(tempFilePath).trim()

    // Delete temporary file from the node
    if (isUnix()) {
        sh(script: 'rm -f ' + tempFilePath, returnStatus: true)
    } else {
        bat(script: 'del /q ' + tempFilePath, returnStatus: true);
    }
    
    return [ rc, stdout ]
}

This workaround looks super ugly and of course I would want just say response = sh(cmd); response.getStatus() etc. without the tones of lines.
I vote +1 for this feature. It is such a basic thing that I am surprised what else can be discussed here. just must be.

If pipeline maintainers refuse to implement it we will probably need to write a plugin. I tried to implement a Java function which executes another arbitrary Java code of the given node, but it does not work directly from the pipeline script due to another limitation (see Pipeline scripts fails to call Java code on slave: Failed to deserialize the Callable object, however from a plugin it must work. But I still hope this feature will be implemented, because it is in high demand and I don't see any reason to not have it.

alexander.samoylov@gmail.com (JIRA)

unread,
Oct 21, 2019, 9:38:39 AM10/21/19
to jenkinsc...@googlegroups.com
Alexander Samoylov edited a comment on Improvement JENKINS-44930
Using a temp file is an ugly workaround which has at least these obvious disadvantages:
1. You never have 100% guarantee that you don't break parallelism. Every time you introduce one more parallel "axe" (such as release/debug conf, branch, os whatever) you must take care that the temp file has the corresponding suffix.
2.  This method won't work in some cases if a command already contains output redirection. For example, on Linux this works: rm 123 > temp.txt 2>&1. I am not sure if we can do the same on Windows. There may be more complex cases with tricky double/single quote combinations and multiple output redirections, of a command which consists of several commands, semicolon separated. At the end we always loose generality and platform-independency if we use a temp file.
3. You must remove the temp file after the command execution, but how do you do it on the node if the pipeline stuff is executed on master? You cannot use Java method File.createTempFile() for this, because it creates the temp file on master. It means that the code which generates the file name must run on master and then you pass this name to the "sh" or "bat" when you remove it. Distinguishing to "sh" and "bat" is also losing of platform-independency by the way.

My current solution with which tried looks like this:

{code:java}

def getTempDirOnNode() {
    if (isUnix()) {
        return env.TMPDIR != null ? env.TMPDIR : '/tmp'
    } else {
         return env.TEMP
    }
}

/*  May not work if "cmd" already contains output redirection or more complex shell syntax. */
def runCmdOnNodeSavingExitCodeAndStdout(cmd) {
    def rc = 0
    def stdout = null
    print("Using temp directory: " + getTempDirOnNode())

    def tempFileName = ' runIgnoreExitCodeStdout_ runCmdOnNodeSavingExitCodeAndStdout_ ' + UUID.randomUUID() + '.txt'

    def tempFilePath = getTempDirNode() + getOSPathSep() + tempFileName
    def tempFileHandle = new File(tempFilePath)
    
    print("Using temp file: " + tempFilePath)
    
    if (isUnix()) {
        rc = sh(script: cmd + ' > ' + tempFilePath, returnStatus: true)
    } else {
        rc = bat(script: cmd + ' > ' + tempFilePath, returnStatus: true);
    }
    stdout = readFile(tempFilePath).trim()

    // Delete temporary file from the node
    if (isUnix()) {
        sh(script: 'rm -f ' + tempFilePath, returnStatus: true)
    } else {
        bat(script: 'del /q ' + tempFilePath, returnStatus: true);
    }
    
    return [ rc, stdout ]
}
{code}


This workaround looks super ugly and of course I would want just say response = sh(cmd); response.getStatus() etc. without the tones of lines.
I vote +1 for this feature. It is such a basic thing that I am surprised what else can be discussed here. just must be.

If pipeline maintainers refuse to implement it we will probably need to write a plugin. I tried to implement a Java function which executes another arbitrary Java code of the given node, but it does not work directly from the pipeline script due to another limitation (see [Pipeline scripts fails to call Java code on slave: Failed to deserialize the Callable object|https://issues.jenkins-ci.org/browse/JENKINS-59778], however from a plugin it must work. But I still hope this feature will be implemented, because it is in high demand and I don't see any reason to not have it.

alexander.samoylov@gmail.com (JIRA)

unread,
Oct 21, 2019, 9:40:20 AM10/21/19
to jenkinsc...@googlegroups.com
Alexander Samoylov edited a comment on Improvement JENKINS-44930
Using a temp file is an ugly workaround which has at least these obvious disadvantages:
1. You never have 100% guarantee that you don't break parallelism. Every time you introduce one more parallel "axe" (such as release/debug conf, branch, os whatever) you must take care that the temp file has the corresponding suffix.
2.  This method won't work in some cases if a command already contains output redirection. For example, on Linux this works: rm 123 > temp.txt 2>&1. I am not sure if we can do the same on Windows. There may be more complex cases with tricky double/single quote combinations and multiple output redirections, of a command which consists of several commands, semicolon separated. At the end we always loose generality and platform-independency if we use a temp file.
3. You must remove the temp file after the command execution, but how do you do it on the node if the pipeline stuff is executed on master? You cannot use Java method File.createTempFile() for this, because it creates the temp file on master. It means that the code which generates the file name must run on master and then you pass this name to the "sh" or "bat" when you remove it. Distinguishing to "sh" and "bat" is also losing of platform-independency by the way.

My current solution with which tried looks like this:

{code:java}
def getTempDirOnNode() {
    if (isUnix()) {
        return env.TMPDIR != null ? env.TMPDIR : '/tmp'
    } else {
         return env.TEMP
    }
}

/*  May not work if "cmd" already contains output redirection or more complex shell syntax. */
def runCmdOnNodeSavingExitCodeAndStdout(cmd) {
    def rc = 0
    def stdout = null
    print("Using temp directory: " + getTempDirOnNode())

    def tempFileName = 'runCmdOnNodeSavingExitCodeAndStdout_' + UUID.randomUUID() + '.txt'

    def tempFilePath = getTempDirNode() + getOSPathSep() + tempFileName
    def tempFileHandle = new File(tempFilePath)
    
    print("Using temp file: " + tempFilePath)
    
    if (isUnix()) {
        rc = sh(script: cmd + ' > ' + tempFilePath, returnStatus: true)
    } else {
        rc = bat(script: cmd + ' > ' + tempFilePath, returnStatus: true);
    }
    stdout = readFile(tempFilePath).trim()

    // Delete temporary file from the node
    if (isUnix()) {
        sh(script: 'rm -f ' + tempFilePath, returnStatus: true)
    } else {
        bat(script: 'del /q ' + tempFilePath, returnStatus: true);
    }
    
    return [ rc, stdout ]
}
{code}

This workaround looks super ugly and of course I would want just say response = sh(cmd); response.getStatus() etc. without the tones of lines.
I vote +1 for this feature. It is such a basic thing that I am surprised what else can be discussed here. just must be.

If pipeline maintainers refuse to implement it we will probably need to write a plugin. I tried to implement a Java function which executes another arbitrary Java code of the given node, but it does not work directly from the pipeline script due to another limitation (see [Pipeline scripts fails to call Java code on slave: Failed to deserialize the Callable object|https://issues.jenkins-ci.org/browse/JENKINS-59778], however from a plugin it must work. But I still hope this feature will be implemented, because it is in high demand and I don't see any reason to not have it.

alexander.samoylov@gmail.com (JIRA)

unread,
Oct 21, 2019, 9:41:17 AM10/21/19
to jenkinsc...@googlegroups.com
Alexander Samoylov edited a comment on Improvement JENKINS-44930
Using a temp file is an ugly workaround which has at least these obvious disadvantages:
1. You never have 100% guarantee that you don't break parallelism. Every time you introduce one more parallel "axe" (such as release/debug conf, branch, os whatever) you must take care that the temp file has the corresponding suffix.
2.  This method won't work in some cases if a command already contains output redirection. For example, on Linux this works: rm 123 > temp.txt 2>&1. I am not sure if we can do the same on Windows. There may be more complex cases with tricky double/single quote combinations and multiple output redirections, of a command which consists of several commands, semicolon separated. At the end we always loose generality and platform-independency if we use a temp file.
3. You must remove the temp file after the command execution, but how do you do it on the node if the pipeline stuff is executed on master? You cannot use Java method File.createTempFile() for this, because it creates the temp file on master. It means that the code which generates the file name must run on master and then you pass this name to the "sh" or "bat" when you remove it. Distinguishing to "sh" and "bat" is also losing of platform-independency by the way.

My current solution with which tried looks like this:

{code:java}
def getOSPathSep() {
    if (isUnix()) {
        return '/'
    } else {
         return '\\'
    }
}

def
getTempDirOnNode() {

    if (isUnix()) {
        return env.TMPDIR != null ? env.TMPDIR : '/tmp'
    } else {
         return env.TEMP
    }
}

/*  May not work if "cmd" already contains output redirection or more complex shell syntax. */
def runCmdOnNodeSavingExitCodeAndStdout(cmd) {
    def rc = 0
    def stdout = null
    def tempFileName = 'runCmdOnNodeSavingExitCodeAndStdout_' + UUID.randomUUID() + '.txt'
    def tempFilePath = getTempDirNode() + getOSPathSep() + tempFileName
    def tempFileHandle = new File(tempFilePath)
    
    print("Using temp file: " + tempFilePath)
    
    if (isUnix()) {
        rc = sh(script: cmd + ' > ' + tempFilePath, returnStatus: true)
    } else {
        rc = bat(script: cmd + ' > ' + tempFilePath, returnStatus: true);
    }
    stdout = readFile(tempFilePath).trim()

    // Delete temporary file from the node
    if (isUnix()) {
        sh(script: 'rm -f ' + tempFilePath, returnStatus: true)
    } else {
        bat(script: 'del /q ' + tempFilePath, returnStatus: true);
    }
    
    return [ rc, stdout ]
}
{code}

This workaround looks super ugly and of course I would want just say response = sh(cmd); response.getStatus() etc. without the tones of lines.
I vote +1 for this feature. It is such a basic thing that I am surprised what else can be discussed here. just must be.

If pipeline maintainers refuse to implement it we will probably need to write a plugin. I tried to implement a Java function which executes another arbitrary Java code of the given node, but it does not work directly from the pipeline script due to another limitation (see [Pipeline scripts fails to call Java code on slave: Failed to deserialize the Callable object|https://issues.jenkins-ci.org/browse/JENKINS-59778], however from a plugin it must work. But I still hope this feature will be implemented, because it is in high demand and I don't see any reason to not have it.

ilya.shaisultanov@gmail.com (JIRA)

unread,
Jan 5, 2020, 6:53:19 PM1/5/20
to jenkinsc...@googlegroups.com
ilya s commented on Improvement JENKINS-44930

Here's my use case for this:

aws ecr describe-repositories --repository-names foo

I want to check if a Docker repository exists. On success, this command returns a JSON blob in stdout. On failure, it returns a string into stderr. As of right now, what should be a simple case of pattern-matching on the stderr content when exit code != 0 turns into something completely different. I've not yet came to a solution, but it's disappointing that what looks like shell invocation step behaves nothing like the shell in practice.

erikblomqvist3@gmail.com (JIRA)

unread,
Feb 17, 2020, 7:08:04 AM2/17/20
to jenkinsc...@googlegroups.com

I can't believe that you've been debating this since 2017. It's a basic feature and wanting to do processing based on the return code and the output is not an "uncommon use case". Any chance we could get this ticket moving? 

uliul.carpatin@gmail.com (JIRA)

unread,
Mar 10, 2020, 12:02:10 PM3/10/20
to jenkinsc...@googlegroups.com

+1 for this feature ..

A solution to not break the compatibility would be to:

  • add an extra parameter, 'returnBothOutputs'
  • return as a list the two items [ returnStatusOptionContent , returnStdoutOptionContent ]
This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38)
Atlassian logo

meghan.blanchard@fmr.com (JIRA)

unread,
Apr 14, 2020, 11:28:06 AM4/14/20
to jenkinsc...@googlegroups.com

I signed up just to be able to up-vote this ticket.  It is an incredibly fundamental requirement, I don't understand the controversy. 

Nancy.Robertson@ca.ibm.com (JIRA)

unread,
Apr 15, 2020, 9:15:04 AM4/15/20
to jenkinsc...@googlegroups.com

I like Uliul Carpatin's solution that does not break existing code.

It or a variant thereof could also include stdErr...

Reply all
Reply to author
Forward
0 new messages