Sudo su testing with Rundeck 4.14.0

[Russ Robinson]

Team,

  I was trying out Rundeck 4.14.0 release and seeing if the "sudo su -" issues from previous releases are fixed (as I noticed the upgrade to sshj plugin).  When trying to run the inline script (with invocation string of: sudo su - root); it receives the error:

Input closed

SSH command execution error: Unknown: java.io.EOFException: Input closed

The debug output is below.  It looks like it copies the inline script to /tmp and there is an /tmp/ssh-<xxxxxx>/agent.<yyyy> file.  Note: This still works with Rundeck 4.10 and 4.7 releases.  Any suggestions?

[sshj-ssh] executing command sudo su - root /tmp/58-51928-mytestserver.com-dispatch-script.tmp.sh

[net.schmizz.sshj.connection.channel.direct.SessionChannel] Sending channel request for `shell`

[net.schmizz.concurrent.Promise] Awaiting <<chan#0 / chanreq for shell>>

[net.schmizz.sshj.connection.channel.direct.SessionChannel] Received window adjustment for 2097152 bytes

[net.schmizz.sshj.connection.channel.Window$Remote] Increasing by 2097152 up to 2097152

[net.schmizz.concurrent.Promise] Setting <<chan#0 / chanreq for shell>> to `SOME`

[net.schmizz.sshj.transport.TransportImpl] Dying because - Broken transport; encountered EOF

[net.schmizz.sshj.transport.TransportImpl] Disconnected - UNKNOWN

[net.schmizz.sshj.transport.KeyExchanger] Got notified of net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF

[net.schmizz.sshj.connection.ConnectionImpl] Notified of net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF

[net.schmizz.sshj.connection.channel.direct.SessionChannel] Channel #0 got notified of net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF

[net.schmizz.sshj.connection.ConnectionImpl] Forgetting `session` channel (#0)

[net.schmizz.concurrent.Promise] Setting <<chan#0 / close>> to `SOME`

[net.schmizz.sshj.transport.TransportImpl] Setting active service to null-service

[net.schmizz.concurrent.Promise] Setting <<transport close>> to `SOME`

[net.schmizz.sshj.transport.Reader] Stopping

Input closed

[sshj-ssh] closing session

[sshj-ssh] disconnected

SSH command execution error: Unknown: java.io.EOFException: Input closed

[Russ Robinson]

After importing this job into a project with all of our standard ssh and sudo settings; the job gets further along.  In debug mode, I do see where the job ssh's into the server but then an operation times out.  Im assumintg this is sshj post-sudo waiting on a command prompt.  Note: I do not have regexp('~.*\$') defined in any of the project, framework, or node settings.  Suggestions?

[sshj-ssh] Env variable RD_JOB_FILTER cannot by set: Request failed

[sshj-ssh] executing command sudo su - root /tmp/109-51947-myserver.com-dispatch-script.tmp.sh

[net.schmizz.sshj.connection.channel.direct.SessionChannel] Sending channel request for `shell`

[net.schmizz.concurrent.Promise] Awaiting <<chan#0 / chanreq for shell>>

[net.schmizz.sshj.connection.channel.direct.SessionChannel] Received window adjustment for 2097152 bytes

[net.schmizz.sshj.connection.channel.Window$Remote] Increasing by 2097152 up to 2097152

[net.schmizz.concurrent.Promise] Setting <<chan#0 / chanreq for shell>> to `SOME`

[net.schmizz.sshj.connection.channel.Window$Local] Consuming by 175 down to 2096977

Last failed login: Tue Jun 20 18:40:00 UTC 2023 from myserver.com on ssh:notty

There were 2 failed login attempts since the last successful login.

[net.schmizz.sshj.transport.TransportImpl] Dying because - Broken transport; encountered EOF

[net.schmizz.sshj.transport.TransportImpl] Disconnected - UNKNOWN

[net.schmizz.sshj.transport.KeyExchanger] Got notified of net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF

[net.schmizz.sshj.connection.ConnectionImpl] Notified of net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF

[net.schmizz.sshj.connection.channel.direct.SessionChannel] Channel #0 got notified of net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF

[net.schmizz.sshj.connection.ConnectionImpl] Forgetting `session` channel (#0)

[net.schmizz.concurrent.Promise] Setting <<chan#0 / close>> to `SOME`

[net.schmizz.sshj.transport.TransportImpl] Setting active service to null-service

[net.schmizz.concurrent.Promise] Setting <<transport close>> to `SOME`

[net.schmizz.sshj.transport.Reader] Stopping

Expect operation fails (timeout: 30000000 ms) for matcher: regexp('~.*\$')

[sshj-ssh] closing session

[rac...@rundeck.com]

Hi Russ,

I tested Rundeck 4.14+SSHJ+sudo as invocation string successfully, I'm seeing a normal behavior. I'm missing something for sure, could you share a job definiton example to replicate your enviroment? Also send the remote node entry.

Thanks.

[Russ Robinson]

From digging through your files, it appears that "PROMPT_PATTERN" has the line:

private final String PROMPT_PATTERN = "~.*\\$";

in https://github.com/rundeck-plugins/sshj-plugin/blob/main/src/main/java/com/plugin/sshjplugin/sudo/SudoCommand.java

My related project settings are:

project.retry-counter=3
project.ssh-authentication=password
project.ssh-command-timeout=0
project.ssh-connect-timeout=0
project.ssh-keypath=/var/lib/rundeck/.ssh/id_rsa
project.ssh.user=${option.username}
project.sudo-command-enabled=true
service.FileCopier.default.provider=sshj-scp
service.NodeExecutor.default.provider=sshj-ssh

For my nodes within the inventory, I have the following settings:

    "sudo-command-enabled": "true",
    "sudo-password-option": "option.sshPassword",
    "username": "${option.username}",
    "password-option": "option.sshPassword",

My prompts are:

[rurobin@myserver tmp]$ sudo su - oci_fmw /tmp/temp_russ.sh
[sudo] password for rurobin:
This is a test
Tue Jun 20 20:42:36 UTC 2023
exit
[rurobin@myserver tmp]$ sudo su - root /tmp/temp_russ.sh
This is a test
Tue Jun 20 20:42:49 UTC 2023
exit
[rurobin@myserver tmp]$

Any suggestion that I'm overlooking?

[Russ Robinson]

Example job:

- defaultTab: nodes

description: ''

executionEnabled: true

id: df5a98ab-c9a0-4e8f-b3be-3ef284fa8b8e

loglevel: INFO

name: ssh sudo su test job

nodeFilterEditable: false

nodefilters:

dispatch:

excludePrecedence: true

keepgoing: false

rankOrder: ascending

successOnEmptyNodeFilter: false

threadcount: '100'

filter: 'os_name: Linux !function: database_server'

nodesSelectedByDefault: true

options:

- label: username

name: username

- label: sshPassword

name: sshPassword

secure: true

plugins:

ExecutionLifecycle: null

scheduleEnabled: true

sequence:

commands:

- description: date

exec: date

- description: Sleep Test

fileExtension: .sh

interpreterArgsQuoted: false

script: |

#!/bin/bash

my_start_date=`date`

my_max=10000

my_curr_try=0

my_sleep=480

echo "Starting the script...."

while [ $my_curr_try -lt $my_max ]

do

my_curr_date=`date`

(( my_curr_try = my_curr_try + 1 ))

echo "Start: ${my_start_date} Curr: ${my_curr_date} Try: ${my_curr_try} of ${my_max}"

echo "Sleeping ${my_sleep}"

sleep ${my_sleep}

done

scriptInterpreter: sudo su - root

keepgoing: false

strategy: node-first

uuid: df5a98ab-c9a0-4e8f-b3be-3ef284fa8b8e

[Russ Robinson]

Node entry:

[
  {
    "nodename": "myserver.com",
    "type": "Node",
    "tags": [

    ],
    "os": "Linux",
    "os_name": "Linux",
    "memory": "120240",
    "hostname": "myserver.com",
    "lifecycle_status": "development",
    "function": "test_server",
    "osFamily": "unix",

    "sudo-command-enabled": "true",
    "sudo-password-option": "option.sshPassword",
    "username": "${option.username}",
    "password-option": "option.sshPassword"
  },

[Russ Robinson]

In digging further, I commented out the following line in SudoCommand.java:

expect.expect(regexp(PROMPT_PATTERN));

and compiled my own jar for testing.  When the sudo command is tried to be issued, it receives:

sudo: sorry, you must have a tty to run sudo

I do have the following project setting:

project.always-set-pty=true

Did something change in the newer versions of the sshj plugin whereby tty sessions are not being set?

[Russ Robinson]

I found the problem.  In comparing 0.2 version of. sshj plugin with latest 0.7; the following line was removed from SSHJExec.java at some point:

session.allocateDefaultPTY();

Around line 55, it should be:

session = ssh.startSession();

session.allocateDefaultPTY();

If I add that line back in, compile a jar, and use it; my sudo commands run fine.

Can we add this line back in?

[rac...@rundeck.com]

Hi Russ,

Amazing catch, could you create a new pull request on the SSHJ github repository?  (create a branch, apply your changes and then create the Pull Request) The dev team will check it.

Thanks!

[Russ Robinson]

Here is pull request: https://github.com/rundeck-plugins/sshj-plugin/pull/19

[rac...@rundeck.com]

Thanks so much for your contribution, Russ.

[Russ Robinson]

As I know everyone is very busy, any ideas how long it will take for the above pull request to be merged into the plugin and shipped with a Rundeck release?