JSchException: Auth cancel when executing command with ctl

[dimitar]

I am new with ControlTier, but I found the installation and
configuration of the server and client straightforward and not
complicated, which just makes my frustration bigger when trying to
execute unsuccessfully the following command from the server to asses
if JBoss is down on the client machine.

ctl -v -p MyProject -t JBossServer -o myservice -c
assertServiceIsDown

SSH works with both rsa and dsa keys. CT is set-up to use dsa. I have
seen similar posts where people fixed the problem by correcting their
SSH PKI configuration. In my case nothing worked making me believe the
ctl on the server side is not using the keys of the CT dedicated
account. is it possible?

Thanks,

Dimitar

ctl -v -p MyProject -t JBossServer -o myservice -c
assertServiceIsDown
parsing buildfile /opt/software/ctier/pkgs/ctl-1.4.8/lib/ant/ctl/
nodedispatch.xml with URI = file:/opt/software/ctier/pkgs/ctl-1.4.8/
lib/ant/ctl/nodedispatch.xml
Project base dir set to: /opt/software/ctier/pkgs/ctl-1.4.8/lib/ant/
ctl
Build sequence for target(s) `execute' is [execute]
Complete build sequence is [execute, ]

execute:
Property "opts.keepgoing" has not been set
Property "opts.failed-nodes.file" has not been set
Override ignored for property "context.user"
authenticated user: ctier
creating an ObjectCommandProxyDispatcher
applying nodeset filter...
number of nodes to dispatch to: 1, (1 threads)
preparing for sequential execution...
returning RemoteCommand object for command: assertServiceIsDown
SSHExec node1.mydomain.com -> "ctl -p MyProject -t JBossServer -o
myservice -c assertServiceIsDown"
dispatching to proxy on node: wasintqa0506.wiley.com
Connecting to node1.mydomain.com:22

Command failed:
/opt/software/ctier/pkgs/ctl-1.4.8/lib/ant/ctl/nodedispatch.xml:19:
com.jcraft.jsch.JSchException: Auth cancel
at
com.controltier.ctl.tasks.controller.Controller.performExecuteAction
(Controller.java:359)
at com.controltier.ctl.tasks.controller.Controller.execute
(Controller.java:311)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:
288)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.tools.ant.dispatch.DispatchUtils.execute
(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.taskdefs.Sequential.execute(Sequential.java:
62)
at net.sf.antcontrib.logic.IfTask.execute(IfTask.java:197)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.tools.ant.dispatch.DispatchUtils.execute
(DispatchUtils.java:106)
at org.apache.tools.ant.TaskAdapter.execute(TaskAdapter.java:154)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:
288)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.tools.ant.dispatch.DispatchUtils.execute
(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:357)
at org.apache.tools.ant.Target.performTasks(Target.java:385)
at org.apache.tools.ant.Project.executeSortedTargets(Project.java:
1340)
at org.apache.tools.ant.Project.executeTarget(Project.java:1309)
at com.controltier.ctl.common.AntProject.execute(AntProject.java:481)
at com.controltier.ctl.cli.CtlMain$mainDispatchAction.execute
(CtlMain.java:472)
at com.controltier.ctl.cli.CtlMain$mainDispatchAction.perform
(CtlMain.java:350)
at com.controltier.ctl.cli.CtlMain.go(CtlMain.java:292)
at com.controltier.ctl.cli.AbstractCtlMain.run(AbstractCtlMain.java:
236)
at com.controltier.ctl.cli.CtlMain.main(CtlMain.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at launcher.CtlLauncher.main(CtlLauncher.java:34)
Caused by: com.jcraft.jsch.JSchException: Auth cancel
at org.apache.tools.ant.taskdefs.optional.ssh.SSHExec.execute
(SSHExec.java:188)
at com.controltier.ctl.tasks.controller.node.SshCommand.execute
(SshCommand.java:178)
at com.controltier.ctl.cli.CtlExec.executeNodedispatch(CtlExec.java:
950)
at
com.controltier.ctl.tasks.controller.node.ActionProxyDispatcher.dispatch
(ActionProxyDispatcher.java:123)
at
com.controltier.ctl.tasks.controller.node.BaseExecutableTask.execute
(BaseExecutableTask.java:37)
at
com.controltier.ctl.tasks.controller.node.NodeDispatchAction.perform
(NodeDispatchAction.java:73)
at com.controltier.ctl.types.controller.ExecuteAction.perform
(ExecuteAction.java:136)
at
com.controltier.ctl.tasks.controller.Controller.performExecuteAction
(Controller.java:353)
... 38 more
Caused by: com.jcraft.jsch.JSchException: Auth cancel
at com.jcraft.jsch.Session.connect(Session.java:453)
at com.jcraft.jsch.Session.connect(Session.java:142)
at org.apache.tools.ant.taskdefs.optional.ssh.SSHBase.openSession
(SSHBase.java:212)
at org.apache.tools.ant.taskdefs.optional.ssh.SSHExec.execute
(SSHExec.java:158)
... 45 more

[Alex-SF]

Hi,
What version of CT are you using and also what OSes?
Thanks

[Anthony Shortland]

Hi Dimitar,

I find that the JSch stack traces often get in the way of identifying the root cause of a given problem. You can take Ctl out of the picture as follows:

1) Check which key Ctl is configured to use:

[ctier@centos54 ~]$ fgrep framework.ssh.keypath $CTIER_ROOT/ctl/etc/framework.properties
framework.ssh.keypath = /opt/ctier/.ssh/id_dsa

2) Run the system's ssh directly to test as follows:

[ctier@centos54 ~]$ ssh -v -i /opt/ctier/.ssh/id_dsa ctier@localhost id
OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to localhost [127.0.0.1] port 22.
debug1: Connection established.
debug1: identity file /opt/ctier/.ssh/id_dsa type 2
debug1: loaded 1 keys
debug1: Remote protocol version 2.0, remote software version OpenSSH_4.3
debug1: match: OpenSSH_4.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_4.3
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-cbc hmac-md5 none
debug1: kex: client->server aes128-cbc hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host 'localhost' is known and matches the RSA host key.
debug1: Found key in /opt/ctier/.ssh/known_hosts:1
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-with-mic,password
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
Unknown code krb5 195

debug1: Unspecified GSS failure.  Minor code may provide more information
Unknown code krb5 195

debug1: Unspecified GSS failure.  Minor code may provide more information
Unknown code krb5 195

debug1: Next authentication method: publickey
debug1: Offering public key: /opt/ctier/.ssh/id_dsa
debug1: Server accepts key: pkalg ssh-dss blen 433
debug1: read PEM private key done: type DSA
debug1: Authentication succeeded (publickey).
debug1: channel 0: new [client-session]
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
debug1: Sending command: id
uid=101(ctier) gid=103(ctier) groups=103(ctier)
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: channel 0: free: client-session, nchannels 1
debug1: Transferred: stdin 0, stdout 0, stderr 0 bytes in 0.1 seconds
debug1: Bytes per second: stdin 0.0, stdout 0.0, stderr 0.0
debug1: Exit status 0

If this doesn't isolate the problem, then we may actually be facing a JSch problem (wouldn't be the first time!).

Anthony.

--
You received this message because you are subscribed to the Google Groups "ControlTier" group.
To post to this group, send email to contr...@googlegroups.com
To unsubscribe from this group, send email to controltier...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/controltier?hl=en
http://wiki.controltier.org

[Dimitar Georgievski]

Sorry, I forgot to provide this important information

- ControlTier 3.4.8
- RHEL 4.0
- Sun Java 1.5.0_17

Dimitar

[Dimitar Georgievski]

Hi Anthony,

I already did that. CT is using id_dsa and I also verified with the SSH client that this key is used to establish the connection between the two machines.

I also configured ctl client (through framework.ssh.keypath setting) to use id_rsa key and received the same error when I re-ran the ctl command.

Would it be possible to extend the debugging output of JSch to determine the user name used to establish the SSH connection?

Thanks,

Dimitar

[Anthony Shortland]

Here's my attempt at re-producing your setup (using 3.4.9 by the way):

1) I have a two node setup: the ControlTier server and a client.

2) I used Workbench to configure a Site object deployed to the server with a Service object as its resource deployed to the client.

3) Ran this command to test node dispatching:

[ctier@centos54 ~]$ ctl -p Development -t Site -o mySite -c dispatchCmd -- -command Properties
dispatching command: "Properties " to: myService[Service]...
dispatching "Properties " to object: myService[Service]
Connecting to centos54:22

Command failed: The following error occurred while executing this line:
/opt/ctier/ctl/depots/Development/modules/Mediator/commands/dispatchCmd.xml:168: The following error occurred while executing this line:
/opt/ctier/ctl/depots/Development/modules/Mediator/commands/dispatchCmd.xml:188: The following error occurred while executing this line:
/opt/ctier/ctl/depots/Development/modules/Mediator/commands/dispatchCmd.xml:61: com.jcraft.jsch.JSchException: Auth cancel

4) Re-running in verbose node yielded the following output (which, as you note, does not include the user name):

.

.

.

SSHExec centos54 -> "ctl -p Development -t Service -o myService -c Properties"
dispatching to proxy on node: user1.centos54
Connecting to centos54:22

5) Consulting the client node's properties in Workbench to get its "Ctl Username", I composed the following ssh command to reveal that, in this case, the password prompt is the cause of the Jsch authentication problem (since I don't have user1's password stored in my Node object):

[ctier@centos54 ~]$ ssh -i /opt/ctier/.ssh/id_dsa user1@centos54 id
user1@centos54's password: 
uid=504(user1) gid=504(user1) groups=103(ctier),504(user1)

6) After equivalencing the users using public key authentication (I could also have added user1's password to the Node object using Workbench or ProjectBuilder) I get:

[ctier@centos54 ~]$ ssh -i /opt/ctier/.ssh/id_dsa user1@centos54 id
uid=504(user1) gid=504(user1) groups=103(ctier),504(user1)

7) ... and then re-running the dispatchCmd:

[ctier@centos54 ~]$ ctl -p Development -t Site -o mySite -c dispatchCmd -- -command Properties
dispatching command: "Properties " to: myService[Service]...
dispatching "Properties " to object: myService[Service]
Connecting to centos54:22
cmd : ctl -p Development -t Service -o myService -c Properties
.
.
.

dispatched command: Properties completed for objects: myService[Service]

Works as expected!

Now; looking at your original verbose output, Dimitar, I see:

dispatching to proxy on node: wasintqa0506.wiley.com
Connecting to node1.mydomain.com:22

Please consult Workbench, find the "wasintqa0506.wiley.com" Node object get its "Ctl Username" and "Hostname" (is "node1.mydomain.com" really what you want?) to use in composing an ssh command like:

ssh -i /opt/ctier/.ssh/id_dsa ????@node1.mydomain.com id

(One other note: you have a different Node object name to your Hostname. We've dealt with a number of bugs where the node name was erroneously used instead of the hostname. Can you setup a node with the same node and hostname (i.e. the hostname you want)? Alternatively; upgrade to 3.4.9 where - as you can see from my example - these things are definitely resolved).

Thanks,

Anthony.

[Anthony Shortland]

Hi Dimitar,

Reflecting on this a little further, I should also ask you to check whether you've managed to get an invalid "Ctl Password" into that Node? This can happen indirectly if you load node tags via ProjectBuilder for example. We did identify a bug in that regard around 3.4.8 that ended up saving an unexpanded property into the Node. Note that password authentication takes precedence over public key authentication for remote dispatching ... 

Thanks,

Anthony.

[Dimitar Georgievski]

Hi Anthony,

I'll test your recommendations today. Because of Thanksgiving holidays I wasn't able to come close to my computer.

Thanks,

Dimitar

[Dimitar Georgievski]

Hi Anthony,

The upgrade to CT 3.4.9 seems to fix the JSch issue. I have repeated all the steps performed when installing the previous version - 3.4.8. I am providing all the installation and configuration steps at the end of my message but for clarity I'll provide the information about the ctl command I have executed successfully at the end:

[ctier@ct-client /opt/software/ctier]$ ctl  -p MyProject -t JBossServer -o JB-Serrvice  -c assertServiceIsDown
Connecting to ct-client.mydomain.com:22
cmd : ctl -p MyProject -t JBossServer -o JB-Serrvice -c assertServiceIsDown
[ct...@ct-client.mydomain.com MyProject.JBossServer.JB-Serrvice assertServiceIsDown][INFO] JBoss is DOWN.

Although I didn't check "Ctl Password" property, from your description below the precedence of the password authentication looks like the likely cause of the authentication issue in CT 3.4.8.

Two notes:
1) The 3.4.9 installation sets in .ctierrc incorrectly the CT_HOME to the previous version:
 export CTL_HOME=/opt/software/ctier/pkgs/ctl-1.4.8
 instead of
 export CTL_HOME=/opt/software/ctier/pkgs/ctl-1.4.9

I had to change this setting manually on both the server and client.

2) After creating JBossServer service through the workbench I had to rerun on the client nore the ctl-depot installation command to update the project depot on the client node. This is a minor thing but for a novice it would be helpful if this information is available on the CT Wiki page so I don't have to guess it. Maybe this info is  somewhere on the Wiki site but I didn't find it.

$ ctl-depot -p MyProject -a install
"Install" command running for object: JB-Sercvice[JBossServer]

The client and server installation/configuration steps for CT 3.4.9:
(names of nodes, project, and services were modified for publishing)
================================================
1. Install CT 3.4.9 Server on ct-server
Commands executed as ctier user.
 CTIER_ROOT=/opt/software/ctier
 
 Note: both client and server installation set CTL_HOME incorrectly to:
 export CTL_HOME=/opt/software/ctier/pkgs/ctl-1.4.8
 instead of
 export CTL_HOME=/opt/software/ctier/pkgs/ctl-1.4.9

 a) unzip ControlTier.
 b) ./install.sh -Dserver.jetty.port=5080 -Dserver.jackrabbit.rmi.port=5099 -d /opt/software/ctier
 
 
2) client installation (CT 3.4.9) on ct-client
Commands executed as ctier user
 CTIER_ROOT=/opt/software/ctier
 
 a) unzip ControlTier
 b) ./install.sh --client -Dserver.hostname=ct-server.mydomain.com -Dclient.hostname=ct-client.mydomain.com -Dserver.jetty.port=5080 -Dserver.jackrabbit.rmi.port=5099 -d /opt/software/ctier

3) Test SSH from command line
 a) dsa
 ssh -v -i ~/.ssh/id_dsa ct-client.mydomain.com
 .....
     debug1: Authentications that can continue: publickey,password,keyboard-interactive

    debug1: Next authentication method: publickey

    debug1: Offering public key: /home/ctier/.ssh/id_dsa
    debug1: Server accepts key: pkalg ssh-dss blen 432

    debug1: read PEM private key done: type DSA
    debug1: Authentication succeeded (publickey).
    debug1: channel 0: new [client-session]
    debug1: Entering interactive session.

    [ctier@ct-client ctier]$
   
  b) rsa (default SSH)
    ssh -v -i ~/.ssh/id_rsa ct-client.mydomain.com
      debug1: Authentications that can continue: publickey,password,keyboard-interactive

    debug1: Next authentication method: publickey

    debug1: Offering public key: /home/ctier/.ssh/id_rsa
    debug1: Server accepts key: pkalg ssh-rsa blen 149
    debug1: read PEM private key done: type RSA

    debug1: Authentication succeeded (publickey).
    debug1: channel 0: new [client-session]
    debug1: Entering interactive session.

    [ctier@ct-client ctier]$

4) Jetty server started on the server node
Software Install Info:

    * CTIER_ROOT: /opt/software/ctier
    * JETTY_HOME: /opt/software/ctier/pkgs/jetty-6.1.14
    * Jetty Version: 6.1.14
    * Java Version: 1.5
    * Java Home: /usr/java/5
    * CTL Version: 1.4.9
    * CTL_HOME: /opt/software/ctier/pkgs/ctl-1.4.9
    * CTL_BASE: /opt/software/ctier/ctl
   
5) log-in as default user.

6) Create project on CT server: MyProject

7) Create project depot on the CT client:
[ctier@ct-client ctier]$ctl-depot -p MyProject -a create
Project depot structure created: /opt/software/ctier/ctl/depots/MyProject
Invoking external setup script: /opt/software/ctier/pkgs/ctl-1.4.9/bin/commander-depotsetup.xml
Beginning client setup ...
Running CTL depot setup: /opt/software/ctier/pkgs/ctl-1.4.9/lib/ant/controllers/ctl/depotsetupCmd.xml ...
Trying to override old definition of task document-property
CTL depot setup procedure completed.
Beginning node registration ...
Registering ct-client.mydomain.com[Node] with ctl.base: /opt/software/ctier/ctl, ctl.home: /opt/software/ctier/pkgs/ctl-1.4.9 ...
Registered node "ct-client.mydomain.com" with Workbench @ http://ct-server.mydomain.com:5080/itnav
Completed client setup. Node ct-client.mydomain.com registered in project: "MyProject".

client hostname = client node name = ct-client.mydomain.com

CT Server Workbench/Node :
ct-client.mydomain.com  ControlTier managed node.
    Operating System
    OS Family:     unix    
    OS Name:     Linux    
    OS Arch:     i386    
    OS Version:     2.4.21-4.ELsmp    
    Network
    Hostname:     ct-client.mydomain.com    
    CTL Base:     /opt/software/ctier/ctl    
    CTL Home:     /opt/software/ctier/pkgs/ctl-1.4.9    
    CTL Username:     ctier    
    CTL Password:        
    Tags
    Tags:    
   
   

8) create JBossServer service

9) re-run on client
   ctl-depot -p MyProject -a install
to add JBossServer service to client

10) execute (command executed successfully)
[ctier@ct-client /opt/software/ctier]$ ctl  -p MyProject -t JBossServer -o JB-Serrvice  -c assertServiceIsDown
Connecting to ct-client.mydomain.com:22
cmd : ctl -p MyProject -t JBossServer -o JB-Serrvice -c assertServiceIsDown
[ct...@ct-client.mydomain.com MyProject.JBossServer.JB-Serrvice assertServiceIsDown][INFO] JBoss is DOWN.

Many thanks for you help!

Dimitar