[JIRA] [publish-over-ssh-plugin] (JENKINS-26439) Build does not abort when «Send files or execute commands over SSH» fails

[chris.adams@itential.com (JIRA)]

Chris Adams edited a comment on JENKINS-26439

Re: Build does not abort when «Send files or execute commands over SSH» fails Same problem. I have tested by explicitly exiting with non-zero status and still get: {{ Hint: Some lines were ellipsized, use -l to show in full. SSH: EXEC: completed after 23,017 ms SSH: Disconnecting configuration [cen...@52.20.86.102] ... SSH: Transferred 1 file(s) Finished: SUCCESS}} My guess is that it's exiting with the original scp/sftp transfer set RC and not the Exec command status. Add Comment

This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)

[chris.adams@itential.com (JIRA)]

Chris Adams commented on JENKINS-26439

[chris.adams@itential.com (JIRA)]

Chris Adams edited a comment on JENKINS-26439

Re: Build does not abort when «Send files or execute commands over SSH» fails

Same problem. I have tested by explicitly exiting with non-zero status and still get: {{

SSH: EXEC: completed after 23,017 ms

SSH: Disconnecting configuration [centos@ 52.20.86.102 <IP> ] ...

SSH: Transferred 1 file(s) Finished: SUCCESS

}} My guess is that it's exiting with the original scp/sftp transfer set RC and not the Exec command status.

Add Comment

[liyao.capehope@gmail.com (JIRA)]

Yao Li commented on JENKINS-26439

Re: Build does not abort when «Send files or execute commands over SSH» fails

Hit the problem in Jenkins 2.0 shell step also. ssh <credential>@ip << EOF command_1 command_2 EOF Run aborted on command_1 error, but job still finished "SUCCESS"

Add Comment

[liyao.capehope@gmail.com (JIRA)]

Yao Li updated an issue

Jenkins / JENKINS-26439

Build does not abort when «Send files or execute commands over SSH» fails

Change By: Yao Li Comment:

Hit the problem in Jenkins 2.0 shell step also. ssh <credential>@ip << EOF    command_1    command_2 EOF Run aborted on command_1 error, but job still finished "SUCCESS"

Add Comment

[slide.o.mix@gmail.com (JIRA)]

Alex Earl assigned an issue to Alex Earl

Jenkins / JENKINS-26439 Build does not abort when «Send files or execute commands over SSH» fails

Change By: Alex Earl Assignee: bap Alex Earl

Add Comment

This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)

[slide.o.mix@gmail.com (JIRA)]

Alex Earl commented on JENKINS-26439

Re: Build does not abort when «Send files or execute commands over SSH» fails This would be a change in behavior to the current plugin, I am not sure how many people are relying on the current behavior. I'll have to think about this some more.

Add Comment

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

[m@phrk.de (JIRA)]

Stephan Watermeyer commented on JENKINS-26439

Re: Build does not abort when «Send files or execute commands over SSH» fails

I am encountering simliar issues. I am executing some bash kung-fu on the target server and if this fails, i would like to exit the "step"/plugin with a RC != 0. The pipeline does not respect that the step has failed and continues the pipeline, so that the following step fails. [Pipeline] step SSH: Connecting from host [53d1c1e2cbf8] SSH: Connecting with configuration [swpsws47] ... SSH: EXEC: STDOUT/STDERR from command [ if [ -d /export/yyy/install/xxxmx-soapui/ ]; then cd /export/yyy/install/xxxmx-soapui/ rm -fv /export/yyy/install/soapui.tar.gz tar -zcf ../soapui.tar.gz ./* else echo "This build does not contain SoapUI tests to execute. Please update your branch!" exit 9 fi ] ... This build does not contain SoapUI tests to execute. Please update your branch! SSH: EXEC: completed after 201 ms SSH: Disconnecting configuration [swpsws47] ... ERROR: Exception when publishing, exception message [Exec exit status not zero. Status [9]] [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Download Tests) [Pipeline] sh [xxx-smoketest-swpsws47] Running shell script + wget -q http://swpsws47/yyy/install/soapui.tar.gz [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Verify Server Ready) Stage "Verify Server Ready" skipped due to earlier failure(s) The pipeline looks like pipeline { agent any stages { stage('Prepare Tests') { steps { cleanWs notFailBuild: true sshPublisher(publishers: [sshPublisherDesc(configName: "$ServerName", transfers: [sshTransfer(excludes: '', execCommand: ''' if [ -d /export/yyy/install/xxxmx-soapui/ ]; then cd /export/yyy/install/xxxmx-soapui/ rm -fv /export/yyy/install/soapui.tar.gz tar -zcf ../soapui.tar.gz ./* else echo "This build does not contain have SoapUI tests to execute. Please update your branch!" exit 9 fi ''', execTimeout: 18000, flatten: false, makeEmptyDirs: false, noDefaultExcludes: false, patternSeparator: '[, ]+', remoteDirectory: '', remoteDirectorySDF: false, removePrefix: '', sourceFiles: '')], usePromotionTimestamp: false, useWorkspaceInPromotion: false, verbose: false)]) } } stage('Download Tests') { steps { sh 'wget -q http://$ServerName/yyy/install/soapui.tar.gz' sh 'tar -xzf soapui.tar.gz --directory .' } } stage('Verify Server Ready') { ... } } }

Add Comment

[m@phrk.de (JIRA)]

Stephan Watermeyer commented on JENKINS-26439

Re: Build does not abort when «Send files or execute commands over SSH» fails

Here are working example. Just set the variable $ServerName I would expect that the pipeline does not continue if the sshPublisher plugin returns with something != 0 pipeline { agent any stages { stage('Do sth remote') { steps { cleanWs notFailBuild: true

sshPublisher(publishers: [sshPublisherDesc(configName: "$ServerName", transfers: [sshTransfer(excludes: '', execCommand: '

'' echo "i am a failure" exit 9 ''', execTimeout: 18000, flatten: false, makeEmptyDirs: false, noDefaultExcludes: false, patternSeparator: '[, ]+', remoteDirectory: '', remoteDirectorySDF: false, removePrefix: '', sourceFiles: '')], usePromotionTimestamp: false, useWorkspaceInPromotion: false, verbose: false)]) } } stage('Do sth on jenkins') { steps { echo "Should not run any more" } } } } Outcome: [Pipeline] } [Pipeline] // stage [Pipeline] withEnv [Pipeline] { [Pipeline] stage [Pipeline] { (Do sth remote) [Pipeline] cleanWs [WS-CLEANUP] Deleting project workspace...[WS-CLEANUP] done [Pipeline] step SSH: Connecting from host [53d1c1e2cbf8] SSH: Connecting with configuration [swpsws47] ... SSH: EXEC: STDOUT/STDERR from command [ echo "i am a failure" exit 9 ] ... i am a failure SSH: EXEC: completed after 201 ms SSH: Disconnecting configuration [swpsws47] ... ERROR: Exception when publishing, exception message [Exec exit status not zero. Status [9]] [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Do sth on jenkins) [Pipeline] echo Should not run any more [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline Finished: UNSTABLE

Add Comment

[slide.o.mix@gmail.com (JIRA)]

Alex Earl commented on JENKINS-26439

Re: Build does not abort when «Send files or execute commands over SSH» fails

Maybe I can provide an option that will fail the build if there is a non-zero exit status. This would keep the current default, but allow people who want this behavior to have it. I'll look into it.

Add Comment

[slide.o.mix@gmail.com (JIRA)]

Alex Earl commented on JENKINS-26439

Re: Build does not abort when «Send files or execute commands over SSH» fails

It looks like there is already a flag for this, can you try changing you pipeline as follows: The change is adding failOnError: true before the publishers list.

pipeline { agent any stages { stage('Do sth remote') { steps { cleanWs notFailBuild: true

sshPublisher(failOnError: true, publishers: [sshPublisherDesc(configName: "$ServerName", transfers: [sshTransfer(excludes: '', execCommand: ''' echo "i am a failure"

exit 9 ''', execTimeout: 18000, flatten: false, makeEmptyDirs: false, noDefaultExcludes: false, patternSeparator: '[, ]+', remoteDirectory: '', remoteDirectorySDF: false, removePrefix: '', sourceFiles: '')], usePromotionTimestamp: false, useWorkspaceInPromotion: false, verbose: false)]) } } stage('Do sth on jenkins') { steps { echo "Should not run any more" } } } }

Add Comment

[slide.o.mix@gmail.com (JIRA)]

Alex Earl commented on JENKINS-26439

Re: Build does not abort when «Send files or execute commands over SSH» fails

The thing that needs to be changed is the option to fail the build when no files were copied.

Add Comment

[m@phrk.de (JIRA)]

Stephan Watermeyer commented on JENKINS-26439

Re: Build does not abort when «Send files or execute commands over SSH» fails

Even if i change failOnError to true it still executes the next step. SSH: Connecting from host [3e88a1716d53] SSH: Connecting with configuration [swpsws46] ... SSH: EXEC: STDOUT/STDERR from command [ echo "i am a failure" exit 9 ] ... i am a failure SSH: EXEC: completed after 201 ms SSH: Disconnecting configuration [swpsws46] ... ERROR: Exception when publishing, exception message [Exec exit status not zero. Status [9]] [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Do sth on jenkins) [Pipeline] echo Should not run any more [Pipeline] } [Pipeline] // stage

[Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node

[Pipeline] End of Pipeline Finished: FAILURE

Add Comment

[m@phrk.de (JIRA)]

Stephan Watermeyer commented on JENKINS-26439

Re: Build does not abort when «Send files or execute commands over SSH» fails

We can see that there is this Exception thrown but its not aborting the next steps in the pipeline. I am not too deep in the details, but maybe this exception must not be catched? https://github.com/jenkinsci/publish-over-ssh-plugin/blob/master/src/main/java/jenkins/plugins/publish_over_ssh/BapSshClient.java#L283-L300 private void exec(final BapSshTransfer transfer) { ChannelExec exec = null; try { exec = openExecChannel(); exec.setPty(transfer.isUsePty()); exec.setAgentForwarding(transfer.isUseAgentForwarding()); exec.setInputStream(null); exec.setOutputStream(buildInfo.getListener().getLogger(), true); exec.setErrStream(buildInfo.getListener().getLogger(), true); connectExecChannel(exec, Util.replaceMacro(transfer.getExecCommand(), buildInfo.getEnvVars())); waitForExec(exec, transfer.getExecTimeout()); final int status = exec.getExitStatus(); if (status != 0) throw new BapPublisherException(Messages.exception_exec_exitStatus(status)); } finally { disconnectExecQuietly(exec); } }

Add Comment

[m@phrk.de (JIRA)]

Stephan Watermeyer edited a comment on JENKINS-26439

Re: Build does not abort when «Send files or execute commands over SSH» fails

We can see that there is this Exception thrown but its not aborting the next steps in the pipeline. I am not too deep in the details, but maybe this exception must not be catched? https://github.com/jenkinsci/publish-over-ssh-plugin/blob/master/src/main/java/jenkins/plugins/publish_over_ssh/BapSshClient.java#L283-L300

{code:java}

private void exec(final BapSshTransfer transfer) {         ChannelExec exec = null;         try {             exec = openExecChannel();             exec.setPty(transfer.isUsePty());             exec.setAgentForwarding(transfer.isUseAgentForwarding());             exec.setInputStream(null);             exec.setOutputStream(buildInfo.getListener().getLogger(), true);             exec.setErrStream(buildInfo.getListener().getLogger(), true);             connectExecChannel(exec, Util.replaceMacro(transfer.getExecCommand(), buildInfo.getEnvVars()));             waitForExec(exec, transfer.getExecTimeout());             final int status = exec.getExitStatus();             if (status != 0)                 throw new BapPublisherException(Messages.exception_exec_exitStatus(status));         } finally {             disconnectExecQuietly(exec);         }     }

{code} Ping [~slide_o_mix]

Add Comment

[ivan.ta@gmail.com (JIRA)]

Ivan Torres commented on JENKINS-26439

Re: Build does not abort when «Send files or execute commands over SSH» fails

I think i have the same problem, pipeline just keeps going, it does mark it as failed but it shouldn't continue the flow, have you found a fix for that Stephan Watermeyer ?

Add Comment

[m@phrk.de (JIRA)]

Stephan Watermeyer commented on JENKINS-26439

Re: Build does not abort when «Send files or execute commands over SSH» fails

No. No Solution so far.

Add Comment

[m@phrk.de (JIRA)]

Stephan Watermeyer edited a comment on JENKINS-26439

Re: Build does not abort when «Send files or execute commands over SSH» fails

No. No Solution so far.

The interesting thing is, that the pipeline continues but further sshPublisher steps are canceled with message: Pipeline {code}

pipeline {    agent any    stages {        stage('Do sth remote') {            steps {                cleanWs notFailBuild: true                sshPublisher(failOnError: true, publishers: [sshPublisherDesc(configName: "$ServerName", transfers: [sshTransfer(excludes: '', execCommand: '''

echo "i am a failure"

exit 9               ''', execTimeout: 18000, flatten: false, makeEmptyDirs: false, noDefaultExcludes: false, patternSeparator: '[, ]+', remoteDirectory: '', remoteDirectorySDF: false, removePrefix: '', sourceFiles: '')], usePromotionTimestamp: false, useWorkspaceInPromotion: false, verbose: true)])

}        }                stage('Do sth on jenkins') {             steps {

echo "Should not run any more"             }        }

stage('a') {             steps {                 echo "before"                 sshPublisher(failOnError: true, publishers: [sshPublisherDesc(configName: "$ServerName", transfers: [sshTransfer(excludes: '', execCommand: 'echo $HOSTNAME', execTimeout: 180000, flatten: false, makeEmptyDirs: false, noDefaultExcludes: false, patternSeparator: '[, ]+', remoteDirectory: '/export/pce/inbox', remoteDirectorySDF: false, removePrefix: '', sourceFiles: '*.zip')], usePromotionTimestamp: false, useWorkspaceInPromotion: false, verbose: true)])                 echo "after"             }        }        stage('1') {             steps {                sh 'exit 9'                sh 'echo after exit'             }        }        stage('2') {             steps {                sh 'echo finally'             }        }            } } {code} Outcome {code} SH: Connecting from host [f561680303be]

SSH: Connecting with configuration [swpsws46] ...

SSH: Creating session: username [root], hostname [10.226.2.95], port [22] SSH: Connecting session ... SSH: Connected SSH: Opening exec channel ... SSH: EXEC: channel open

SSH: EXEC: STDOUT/STDERR from command [                    echo "i am a failure"                    exit 9               ] ...

SSH: EXEC: connected i am a failure

ERROR: Exception when publishing, exception message [Exec exit status not zero. Status [9]] [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Do sth on jenkins) [Pipeline] echo Should not run any more [Pipeline] } [Pipeline] // stage

[Pipeline] stage [Pipeline] { (a) [Pipeline] echo before [Pipeline] step SSH: Current build result is [FAILURE], not going to run. [Pipeline] echo after

[Pipeline] } [Pipeline] // stage [Pipeline] stage

[Pipeline] { (1) [Pipeline] sh + exit 9

[Pipeline] } [Pipeline] // stage [Pipeline] stage

[Pipeline] { (2) Stage "2" skipped due to earlier failure(s)

[Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline

ERROR: script returned exit code 9 Finished: FAILURE {code}

Add Comment

[m@phrk.de (JIRA)]

Stephan Watermeyer edited a comment on JENKINS-26439

Re: Build does not abort when «Send files or execute commands over SSH» fails

No. No Solution so far. The interesting thing is, that the pipeline continues but further sshPublisher steps are canceled with message:

*SSH: Current build result is [FAILURE], not going to run.*

[m@phrk.de (JIRA)]

I don't know how and where to fix this. I would assume that somewhere a _AbortException_ or something must be raised, but the current Interface does not allow such modifications. This is something for a more experienced guy ;-)

Add Comment

[m@phrk.de (JIRA)]

I don't know how and where to fix this. I would assume that somewhere a _AbortException_ or something must be raised, but the current Interface for the _endTransfer_ Method does not allow such modifications.

This is something for a more experienced guy ;-)

Add Comment

[ivan.ta@gmail.com (JIRA)]

Ivan Torres commented on JENKINS-26439

Re: Build does not abort when «Send files or execute commands over SSH» fails

Since the plugin mark the build as failure, you could check if the result is failure after it ran   if ('FAILURE'.equals(currentBuild.result)) {     error('ssh error') }

Add Comment

[slide.o.mix@gmail.com (JIRA)]

Alex Earl assigned an issue to Unassigned

Jenkins / JENKINS-26439

Build does not abort when «Send files or execute commands over SSH» fails

Change By: Alex Earl Assignee: Alex Earl

Add Comment