Rundeck SSH Bastion tunnelling

[James Scollard]

I have a series of cloud based (mostly unreachable) networks that are only ssh accessible through bastion hosts.  I know that I can configure sshd to forward requests to say devnode9* for dev east, or qanode8* for qa west, but how do I set up rundeck to tunnel to nodes behind them?  I have the ec2-nodes plugin installed and it works beautifully.  Unfortunately I am pretty sure that there is currently no way to create a project by project ssh config file, and then set the Rundeck service to use it.  I believe I am only able to dictate the required SSH key for each bastion host at the project level.  An additional consideration is that our DNS nodes are configured per region/zone and only contain records for their zone, which makes these bastion hosts the only reasonable static resource that can reliably provide the true connection details for the discovered nodes.  If there is a way to tunnel through these nodes then matching against wildcard regex strings can accurately determine which bastion host to connect to if there is a place to configure this.

Is there a way to set each project up in a way that Rundeck will execute commands from the bastion hosts or tunnel though them using the ec2-nodes plugin discovered node attributes?

Any ideas or suggestions are welcome.

Thanks.

[Peter C]

I'm also very interested in a Rundeck solution for resource nodes behind a bastion host. Like James said, if we can have Rundeck evaluate/execute commands in the context of bastion hosts, it would be perfect. Is this the right place for feature request?

Thanks,

Peter

[Martin Zimmermann]

You can use the Rundeck script Plugin (http://rundeck.org/docs/manual/plugins.html#script-plugin), in combination with your local openssh client (http://backdrift.org/transparent-proxy-with-ssh), to access nodes through bastion hosts. 

Node definition:

 mynode:

    node-executor: script-exec

    script-exec: /usr/bin/ssh -F ssh_config ${node.username}@${node.hostname} -- ${exec.command}

Local ssh_config File:

 Host superchunk.example.org

    ProxyCommand  ssh us...@bastion.example.org nc %h %p

[Peter C]

Thanks for the tip Martin, it does work as you said! In the comments of the page you linked to, someone pointed out OpenSSH 5.4+ have netcat built in, so ssh_config is a bit shorter:

ProxyCommand ssh -W %h:%p us...@bastion.example.org

[Jedblack]

If I have variables in my ssh command (ie. ${exec.command}) is there anyway to expand them?

For example,

${option.httpd}

sudo /sbin/service ${option.httpd} restart

It seem that it errors out with "bad substitution" because its not expanding the variables when the command is run on the remote nodes.

[Zachary Buckholz]

I don't understand why the jsch-ssh plugin doesn't inherit the ~/.ssh/config file and do the proper thing.

Am I missing something?

My rundeck user has a ~/ssh/config file that resembles something like the following

#########  STAGE-AP-SOUTHEAST-1   ################

Host *.stage.ap-southeast-1

 ProxyCommand ssh -W %h:%p 52.x.x.x

 StrictHostKeyChecking no

 IdentityFile ~/.ssh/stage-ap-southeast-1.pem

Host nat-1-1.stage.ap-southeast-1

 Hostname 52.x.x.x

 User ec2-user

 StrictHostKeyChecking no

 IdentityFile ~/.ssh/stage-ap-southeast-1.pem

 ControlMaster auto

 ControlPath ~/.ssh/ansible-%r@%h

 ControlPersist 120m

[K West]

@Zachary

If you want to use an OpenSSH configuration file (~/.ssh/config by default), don't use jsch-ssh (which is just an implementation of SSH AFAIK, not an implementation of OpenSSH). Use pure OpenSSH "ssh" and "scp" commands :

Option 1: at the node definition level (http://rundeck.org/docs/plugins-user-guide/script-plugin.html)

mynode:
    node-executor: script-exec
    script-exec-shell: bash -c
    script-exec: ssh -o "some quoted option" ${node.username}@${node.hostname} ${exec.command}

Option 2: Set the option globally at the Project level. This is what i personally use on my projects:

• Default Node Executor

  The Node Executor is responsible for executing commands and scripts on remote nodes.
  Script Execution Delegates command execution to an external script. Can be configured project-wide or on a per-node basis.

  Command: /usr/bin/ssh ${node.username}@${node.hostname} ${exec.command}

• Default File Copier

  The File Copier is responsible for copying scripts as files to remote nodes before they are executed.
  Script Execution Delegates file copying to an external script. Can be configured project-wide or on a per-node basis.

  Command: /usr/bin/scp ${file-copy.file} ${node.username}@${node.hostname}:${file-copy.destination}

Rq1: if your config file is not at the default location, use ssh -F <path_to_your_ssh_config_file> ...)
Rq2: as you can see above in option 2, you can put ssh and/or scp with their absolute path if you want/need (ex: /usr/bin/ssh ...)

Kofi.