sshj-ssh giving "Failed: Unknown: java.nio.channels.ClosedByInterruptException" when sudo command

[Russ Robinson]

When trying to use Rundeck 4.11 ; it also appears that sshj-ssh is no longer working with remote commands containing sudo.  If the job's remote command just contains the command "date"; all works fine.  For our existing jobs with sudo in the remote command (ex.: sudo su - root -c date ); the job fails with:

...

[net.schmizz.concurrent.Promise] Setting <<authenticated>> to `true`

[net.schmizz.sshj.userauth.UserAuthImpl] `password` auth successful

[sshj-ssh] authentication set

[sshj-ssh] connection done

[sshj-ssh] starting session

[net.schmizz.sshj.connection.ConnectionImpl] Attaching `session` channel (#0)

[net.schmizz.concurrent.Promise] Awaiting <<chan#0 / open>>

[net.schmizz.sshj.connection.ConnectionImpl] Received GLOBAL_REQUEST `hostk...@openssh.com`; want reply: false

[net.schmizz.sshj.connection.channel.direct.SessionChannel] Initialized - < session channel: id=0, recipient=0, localWin=[winSize=2097152], remoteWin=[winSize=0] >

[net.schmizz.concurrent.Promise] Setting <<chan#0 / open>> to `SOME`

[sshj-ssh] setting environments

[sshj-ssh] RD_NODE_CHEF_IS_HEALTHY => n_a

[net.schmizz.sshj.connection.channel.direct.SessionChannel] Sending channel request for `env`

[net.schmizz.concurrent.Promise] Awaiting <<chan#0 / chanreq for env>>

[net.schmizz.concurrent.Promise] <<chan#0 / chanreq for env>> woke to: net.schmizz.sshj.connection.ConnectionException: Request failed

[sshj-ssh] Env variable RD_NODE_CHEF_IS_HEALTHY cannot by set: Request failed

[sshj-ssh] RD_NODE_LIFECYCLE_STATUS => development

[net.schmizz.sshj.connection.channel.direct.SessionChannel] Sending channel request for `env`

[net.schmizz.concurrent.Promise] Awaiting <<chan#0 / chanreq for env>>

[net.schmizz.concurrent.Promise] <<chan#0 / chanreq for env>> woke to: net.schmizz.sshj.connection.ConnectionException: Request failed

[sshj-ssh] Env variable RD_NODE_LIFECYCLE_STATUS cannot by set: Request failed

...

...

[net.schmizz.concurrent.Promise] <<chan#0 / chanreq for env>> woke to: net.schmizz.sshj.connection.ConnectionException: Request failed

[sshj-ssh] Env variable RD_NODE_CPP_CI_FUNCTION cannot by set: Request failed

[sshj-ssh] RD_JOB_FILTER => nodename: ${option.linux_servers}

[net.schmizz.sshj.connection.channel.direct.SessionChannel] Sending channel request for `env`

[net.schmizz.concurrent.Promise] Awaiting <<chan#0 / chanreq for env>>

[net.schmizz.concurrent.Promise] <<chan#0 / chanreq for env>> woke to: net.schmizz.sshj.connection.ConnectionException: Request failed

[sshj-ssh] Env variable RD_JOB_FILTER cannot by set: Request failed

[sshj-ssh] executing command sudo su - root -c date

[net.schmizz.sshj.connection.channel.direct.SessionChannel] Sending channel request for `shell`

[net.schmizz.concurrent.Promise] Awaiting <<chan#0 / chanreq for shell>>

[net.schmizz.sshj.connection.channel.direct.SessionChannel] Received window adjustment for 2097152 bytes

[net.schmizz.sshj.connection.channel.Window$Remote] Increasing by 2097152 up to 2097152

[net.schmizz.concurrent.Promise] Setting <<chan#0 / chanreq for shell>> to `SOME`

[wf:2a3c7054-2134-48c0-a236-e2064b4b009e] Interrupted: Engine interrupted, stopping engine...

[wf:2a3c7054-2134-48c0-a236-e2064b4b009e] WillShutdown: Workflow engine shutting down (interrupted? true)

[wf:2a3c7054-2134-48c0-a236-e2064b4b009e] OperationFailed: operation failed: java.util.concurrent.CancellationException: Task was cancelled.

[sshj-ssh] closing session

[net.schmizz.sshj.connection.channel.direct.SessionChannel] Sending close

[net.schmizz.concurrent.Promise] Awaiting <<chan#0 / close>>

[sshj-ssh] error closing null

[sshj-ssh] disconnected

SSH command execution error: Unknown: java.nio.channels.ClosedByInterruptException

[net.schmizz.sshj.transport.TransportImpl] Disconnected - BY_APPLICATION

[net.schmizz.sshj.connection.ConnectionImpl] Notified of net.schmizz.sshj.transport.TransportException: [BY_APPLICATION] Disconnected

[net.schmizz.sshj.connection.channel.direct.SessionChannel] Channel #0 got notified of net.schmizz.sshj.transport.TransportException: [BY_APPLICATION] Disconnected

[net.schmizz.sshj.connection.ConnectionImpl] Forgetting `session` channel (#0)

[net.schmizz.concurrent.Promise] Setting <<chan#0 / close>> to `SOME`

[net.schmizz.sshj.transport.TransportImpl] Sending SSH_MSG_DISCONNECT: reason=[BY_APPLICATION], msg=[]

[net.schmizz.concurrent.Promise] Setting <<transport close>> to `SOME`

[net.schmizz.sshj.transport.Reader] Stopping

Failed: Unknown: java.nio.channels.ClosedByInterruptException

[workflow] finishExecuteNodeStep(myserver.mytest.com): NodeDispatch: Unknown: java.nio.channels.ClosedByInterruptException

Our project does contain setting:

project.sudo-command-enabled=true

as well as:

project.retry-counter=3
project.ssh-authentication=password
project.ssh-command-timeout=0
project.ssh-connect-timeout=0
project.ssh-keypath=/var/lib/rundeck/.ssh/id_rsa
project.ssh.user=${option.username}

service.FileCopier.default.provider=sshj-scp
service.NodeExecutor.default.provider=sshj-ssh

Any work-around to get sudo working in newer version?

[rac...@rundeck.com]

Hey Russ,

Try with a previous SSHJ plugin version in the following way:

1. Sign in as an administrator user.
2. Click on the Gear icon (up to right) > "Plugins" > "Installed Plugins."
3. Remove the SSHJ node executor plugin (0.1.6).
4. Remove the sshj-0.1.6 directory from the /var/lib/rundeck/libext/cache directory.
5. Download the SSHJ 0.1.4 from here (put the jar file on the /var/lib/rundeck/libext directory).

Now, by default Rundeck regenerates the built in plugins after each reboot, this includes the SSHJ 0.1.6 plugin. If you need to restart the rundeck instance, repeat the steps above to put the SSHJ 0.1.4 plugin.

Of course, another workaround is to rollback to previous Rundeck version.

Hope it helps!

[Russ Robinson]

Should I open another bug regarding the sshj-ssh plugin?

[rac...@rundeck.com]

That's not necessary, This issue is included into the previous one.