Unable to connect to Linux Box Using Rundeck --Getting: NonZeroResultCode: Remote command failed with exit status -1

[versha kamthan]

Hi Team,

I am facing a strange issue wherein I have configured a Linux Box to connect via Rundeck  , and i am trying to connect to it via a user using public-private key pair authentication.
I am able to connect to this box using below command manually (running it directly on the server via putty)

ssh -i /var/lib/rundeck/.ssh/public_key user@<hostname>  'echo test'

But if I go to Rundeck UI -> Commands
and type below:

Command  : echo test
Nodes: hostname

Its giving me below error:

[workflow] beginExecuteNodeStep(name_hostname): NodeDispatch: com.dtolabs.rundeck.execution.ExecutionItemFactory$4@43fbbc67
Using ssh keyfile: /var/lib/rundeck/.ssh/public_key
Starting SSH Connection: user@hostname (name_hostname)
Set timeout to 0
Connecting to hostname:22
Remote command failed with exit status -1
Failed: NonZeroResultCode: Remote command failed with exit status -1


Would be great if someone can help me here.

Thanks,
Versha

[Tommy Tsui]

Hi all,

So it sounds like the ssh plugin on rundeck is not connecting, whereas the ssh on the Linux box is connecting properly.

Is there a way to turn on debugging on the ssh plugin to see what's happening?

Tommy

[Ashish Mishra]

From the o/p can't really say, if the SSH connection itself failed or it's the command. Can you try using "sudo echo test"? Also try to start the debug logs.

[versha kamthan]

Hi Ashish,

Thanks for your reply.
Here is what is happening:


1.We are trying to connect via  rundeck user "user" to a linux box using a public-private key pair using below command from the Rundeck Linux machine:

 ssh -i /var/lib/rundeck/.ssh/public_key user@<hostname>  'sudo echo test'
And we get below output:

This system is intended for authorised users only.
stdin: is not a tty
test

2. If I configure my this <hostname> in my resources.xml file like below:

<node name="MyHost" description="MyHost" tags="MyHost"
hostname="<hostname>"
ssh-keypath="/var/lib/rundeck/.ssh/id_rsa"
ssh-authentication="privateKey"
username="rundeck"
/>

and try to execute similar command(sudo echo test) from Rundeck->Commands
sudo echo test
I am getting below error(same as before):

[workflow] beginExecuteNodeStep(MyHost): NodeDispatch: com.dtolabs.rundeck.execution.ExecutionItemFactory$4@43fbbc67
Using ssh keyfile: /var/lib/rundeck/.ssh/public_key
Starting SSH Connection: rundcek@hostname (MyHost)

Set timeout to 0
Connecting to hostname:22
Remote command failed with exit status -1
Failed: NonZeroResultCode: Remote command failed with exit status -1

3.However when we checked the logs on the other linux box (on MyHost)  it says:
"Accepted public key from <rundeckServer>"
"connection timed out by the client"


4.Also if I configure my node with node-exec and script-exec entries in resources.xml file, I am able to connect to this <MyHost>

<node name="MyHost" description="MyHost" tags="MyHost"
hostname="<hostname>"
ssh-keypath="/var/lib/rundeck/.ssh/id_rsa"
ssh-authentication="privateKey"
username="rundeck"
node-executor="script-exec"
script-exec="ssh -i /var/lib/rundeck/.ssh/id_rsa ${node.username}@${node.hostname} ${exec.command}"
script-exec-shell="bash -c"
/>

Now, if I go to Rundeck->Commands and execute
echo test or sudo echo test; I get below success:

[workflow] Begin step: 1,NodeDispatch
1: Workflow step executing: com.dtolabs.rundeck.execution.ExecutionItemFactory$4@3f972a2
preparing for sequential execution on 1 nodes
Executing command on node: MyHost, NodeEntryImpl{attributes={script-exec-shell=bash -c, hostname=<hostname>, ssh-authentication=privateKey, username=rundeck,nodename=MyHost, node-executor=script-exec, script-exec=ssh -i /var/lib/rundeck/.ssh/id_rsa  ${node.username}@${node.hostname} ${exec.command}, ssh-keypath=/var/lib/rundeck/.ssh/id_rsa }, project='null'}
[workflow] beginExecuteNodeStep(MyHost): NodeDispatch: com.dtolabs.rundeck.execution.ExecutionItemFactory$4@3f972a2
[script-exec] executing: bash -c ssh -i /var/lib/rundeck/.ssh/id_rsa rundeck@<hostname> sudo echo test
****  WARNING ****
This system is intended for authorised users only. 
stdin: is not a tty
test
[script-exec]: result code: 0, success: true
[workflow] finishExecuteNodeStep(MyHost): NodeDispatch: Succeeded
1: Workflow step finished, result: Dispatch successful (1 nodes)
[workflow] Finish step: 1,NodeDispatch
[workflow] Finish execution:  rundeck-workflow-node-first: [Workflow , Node failures: MyHost=[]}]


5.I have selected the Debug Mode in Rundeck Job Config;Can you please tell me if there is way to increase SSH logging?

Thanks in advance.

Regards,
Versha

[Ashish Mishra]

Hi Versha,

Can you also check if the line "#Defaults    requiretty" is not uncommented on your client server in the /etc/sudoers file. 

None of the failure in case seems to be because of SSH. Seems to be a sudo issue. Did you give "user" all the sudo permissions? Add "user ALL=(ALL) NOPASSWD: ALL" to the sudo file. 

--Ashish

[versha kamthan]

Hi Ashish,

Thanks for your reply.

I have checked both the values and they are same as what you are saying, but still no luck :(
Can you tell me if there is a way to get more verbose ssh logging ?


Thanks,
Versha

[Ashish Mishra]

Hi Versha,

Setting the loglevel in Rundeck to Debug should show the ssh connection messages as well. Apart from that, I don't think there is any other method to turn on the verbose for SSH.  

All I can think of now is a plug-in issue. Do you have all the required plugins in your rundeck? Node Execution plug-ins and SSH plug-ins? Not all comes bundled with Rundeck.

Thanks,

Ashish   

[versha kamthan]

Thanks Ashish.
Yeah its a bit strange as this is already working for so many other linux servers, i mean we are able to connect/ssh using rundeck on some other linux servers but this issue is coming on only few servers.

So cant think of any missing plugin as well. :(
we are planning to delete the "user" from the linux box and get it created again, just to see if this has something to do with the user profile or something like that.

Thanks,
Versha

--
You received this message because you are subscribed to a topic in the Google Groups "rundeck-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rundeck-discuss/I8h-17GbDTA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rundeck-discu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rundeck-discuss/5fb3e60c-95f6-4f96-b8c8-c92ca2792c54%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ashish Mishra]

Good luck. :) Let me know how it goes. 

[Sam]

IMHO, 2 issues are causing this

1. debug log shows rundeck is using <rundeck> to remote into instance rather than <user>, you need to change that in framework properties or project properties.

2. SSH key path used by rundeck is not the one you are expecting /var/lib/rundeck/.ssh/id_rsa

Make sure you are using private key to ssh and also using the same for rundeck. At the start you mentioned you are trying to ssh with public key and detailed debug message shows private key ??? I am confused about that just make sure that is set and you should be all good to go.

[versha kamthan]

Hi Sam,

Thanks for you time to look in to this and sorry for the late reply.
But the 2 points which you have mentioned are so beacus i was trying to frame my question in first places hence i wrote that by mistake.
I am actually connecting using username = <rundeck> only and with the private key as </var/lib/rundeck/.ssh/id_rsa">.

I , howevere tried to change all the log levels to "debug" in log4j.properties but couldnt increase more "Log Output" in Rundeck UI.
Can you tell me how can i get more information/logs for a particular job.( I have already selected "LogLevel = Debug" in rundeck job configuration)

Thanks for your help.

Regards,
Versha

[Sam]

Did you check the rundeck.log and service.log after running the command. The best way I can think of given the limited information is that you create a job  and configure your resource for the project, and run the job with a date or echo command. This should give you a detailed log as to why it is failing. 

Things to cross check are the project's resource config, default node executor (should be ssh) and finally the private key (path etc).