Steps for ssh forwarding?

[Ananda Debnath]

Hi Ansiblers,

I have an Ansible controller machine that contains my playbooks, custom libraries, key files etc.
I have an inventory of host machines and CentOS VM's that run on these host machines. The VMs aren't directly accessible from the Ansible box. To get to each VM manually, I need to ssh into each host machine and then ssh into the specific VM.

While in a shell on the Ansible machine, I can reach the VM's by using: ssh -o "ForwardAgent=yes" -tt ki...@56.66.3.10 ssh redis...@10.0.0.1
I.e. the chain of access is Ansible-machine --> dom0 --> vm

I'd like to specify the VMs in my inventory file - but unsure what all things need to be set in ansible.cfg, ~/.ssh/config etc.

Inventory:

[dom0-host]
56.66.3.10

[dom0-host:vars]
ansible_ssh_user=king
ansible_ssh_private_key_file=/home/ansible_user/.ssh/dom0_id_rsa.private_key

[redis-vm]
# Can only access this from dom0-host
10.0.0.1 ansible_ssh_user=redis_user ansible_ssh_pass="reallybadidea"

[web-vm]
# Can only access this from dom0-host
10.0.0.2 ansible_ssh_user=redis_user ansible_ssh_pass="anotherbadidea"

I tried googling around, but couldn't find a summary of the steps that would allow me to do this. I'm hoping this is possible and that someone can enumerate the steps here.

Many thanks!
Ananda

[Giovanni Tirloni]

On Mon, 02 Feb 2015 10:38 -0800, Ananda Debnath
<ananda....@gmail.com> wrote:
> Hi Ansiblers,
>
> I have an Ansible controller machine that contains my playbooks, custom
> libraries, key files etc.
> I have an inventory of host machines and CentOS VM's that run on these
> host
> machines. The VMs aren't directly accessible from the Ansible box. To get
> to each VM manually, I need to ssh into each host machine and then ssh
> into
> the specific VM.

Your scenario seems similar to having a bastion host so the SSH
ProxyCommand directive might help.

https://www.google.com/search?q=ansible+bastion+host

Giovanni

[Ananda Debnath]

Many thanks!

After a few hours of wresting with this and my setup, and a typo that my eyes kept skimming over I finally got this working.

Thanks again for point it out to me Giovanni!

[Ananda Debnath]

Does anyone know of an example to do this using password authentication?

My inventory file has ansible_ssh_user and ansible_ssh_pass values set and my ssh config file has password authentication set to yes.

my.ssh.config

Host 56.66.3.10
    User                   king
    HostName               56.66.3.10
    ProxyCommand           none
    BatchMode              yes
    PasswordAuthentication yes

# Also tried ssh -aY ki...@56.66.3.10 'nc -w 14400 %h %p'
Host *
    ServerAliveInterval    60
    TCPKeepAlive           yes
    ProxyCommand           ssh -AY ki...@56.66.3.10 'nc -w 14400 %h %p'
    ControlMaster          auto
    ControlPath            ~/.ssh/mux-%r@%h:%p
    ControlPersist         15s
    PasswordAuthentication yes

But this isn't helping. I have a test role that applies to 56.66.3.10 first and then other roles to the ones behind the bastion - redis-vm, etc. Neither the bastion nor the vms behind it will let me through with passwords.
The ansible.cfg file looks like:
[defaults]
host_key_checking = False
# Tried both with and without this
transport=ssh

[ssh_connection]
ssh_args = -F my.ssh.config
scp_if_ssh = True
control_path = ~/.ssh/mux-%%r@%%h:%%p

The tail of the bastion login failure reads:

debug2: key: /home/adebnath/.ssh/id_rsa ((nil)),
debug2: key: /home/adebnath/.ssh/id_dsa ((nil)),
debug2: key: /home/adebnath/.ssh/id_ecdsa ((nil)),
debug2: key: /home/adebnath/.ssh/id_ed25519 ((nil)),
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred
debug1: No more authentication methods to try.
Permission denied (publickey,password).

[Michael Peters]

Not the answer you're looking for, but why don't you just use ssh
keys? It's some minor work upfront with huge security and automation
benefits easier.

> --
> You received this message because you are subscribed to the Google Groups
> "Ansible Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ansible-proje...@googlegroups.com.
> To post to this group, send email to ansible...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/ansible-project/c776ec2d-c640-4e2e-bb50-ecbd24548673%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Ananda Debnath]

Thanks for looking.

There are too many current processes dependent on passwords that I'm migrating to Ansible - while converting to keys is partly underway, it won't be complete for a while.

There's also a second bootstrapping problem. I'm using Ansible to run baremetal bringup scripts on xen hosts (doing double duty as the jumpbox/bastion host above) which in turn creates the VM's from an image. I cannot bake keys into the image and need to get them on there after they boot up. Hence - a chicken and egg problem. How would I get the keys onto the VMs? Xen hosts have numerous problems running Ansible scripts directly because they run an old version of Python - so I don't think I can call the authorized_key module on the host and have it inject them into the VMs. The action would need to be triggered outside in a machine/vm that supports Ansible - which in turn would need to tunnel into the VMs to do just this - hence chicken and egg.

Ultimately, the key pairs would be created and injected into the images instead of the passwords we do today - but as I mention above, this won't be for a while.

[Ananda Debnath]

I did manage to get it to work - but it's really ugly :/

I had to change the ProxyCommand directive in the ssh config for the wildcard section to:

ProxyCommand sshpass -p 'reallybadpassword' ssh -A ro...@56.66.3.10 'nc -w 14400 %h %p'

I guess Ansible has no way of overriding the ProxyCommand specified in the ssh_config file in order to use sshpass with the password specified in the inventory file?

[Michael Spiegle]

As an additional datapoint, here's a brief summary of how I deal with this.  To complicate matters, my machines are split across various labs in different locations which each have their own bastion/jumpbox.  I use ssh keys sometimes, and hard coded passwords for some other machines:

ansible.cfg:

[defaults]
hostfile = hosts
error_on_undefined_vars = True
host_key_checking = False
transport = ssh
jinja2_extensions = jinja2.ext.do

[ssh_connection]
ssh_args = -F ssh.config
pipelining = True

ssh.config (referenced in ansible.cfg).  Note that if you have a new enough version of ssh, you can use the -W flag instead of nc:

#jumpboxes first (most specific hosts first)
Host jumpbox01 10.1.0.10
 ControlMaster yes
 ControlPath ~/.ssh/master-%r@jumpbox01:%p
 StrictHostkeyChecking no
 ProxyCommand none

Host jumpbox02 10.2.0.10
 ControlMaster yes
 ControlPath ~/.ssh/master-%r@jumpbox02:%p
 StrictHostkeyChecking no
 ProxyCommand none

Host jumpbox03 10.3.0.10
 ControlMaster yes
 ControlPath ~/.ssh/master-%r@jumpbox03:%p
 StrictHostkeyChecking no
 ProxyCommand none

# groups of machines that can be accessed by the above jumpboxes
Host *.west.domain.com 10.1.0.*
 ControlMaster no
 ProxyCommand ssh -S ~/.ssh/master-*@jumpbox01:%p remote nc %h %p

Host *.central.domain.com 10.2.0.*
 ControlMaster no
 ProxyCommand ssh -S ~/.ssh/master-*@jumpbox02:%p remote nc %h %p

Host *.east.domain.com 10.3.0.*
 ControlMaster no
 ProxyCommand ssh -S ~/.ssh/master-*@jumpbox03:%p remote nc %h %p

# this makes ansible faster by reusing connections
Host *

 ControlMaster auto
 ControlPersist 300s
 ControlPath ~/.ssh/ansible-%r@%h:%p

Once I have those configs setup, I have to run the following to establish a tunnel to a jumpbox/bastion before I can run ansible:

$ ssh -F ssh.config -fN user@jumpbox01

When I run the above, it asks for the password (or uses my SSH key), then SSH goes into the background and then the tunnel is established.

I do all of my deployments this way by creating a Jenkins job that establishes the tunnel, runs ansible, then tears down the tunnel using something like:

ssh -O exit -TS ~/.ssh/path-to-socket

Some of the jumpboxes use dumb passwords, some of them use keys, and one of them requires an RSA token (2-factor auth).  For the RSA machine, my Jenkins job presents the user with a form that has 2 fields: 2-Factor Username, and 2-Factor Passcode.  The passcode is generated by an RSA token keyfob (or smartphone app).  In order to make this work, I had to write a custom expect script because the SSH prompt for the RSA token reads "Enter PASSCODE" instead of "Password" which is what sshpass is hardcoded to look for.  Here's my expect script to catch all the variations:

#!/usr/bin/env expect
set timeout 30
set userhost [lindex $argv 0]
spawn ssh -fN -F ssh.config $userhost

expect {
  "Enter PASSCODE:" {
    send "$env(SSH_PASSWORD)\n"
    send "\n"
  }

  "Password:" {
    send "$env(SSH_PASSWORD)\n"
    send "\n"
  }

  "password:" {
    send "$env(SSH_PASSWORD)\n"
    send "\n"
  }
}

sleep 5

I need the sleep 5 at the end of the script as a hack because the jumpbox with RSA token don't establish the control socket until a few seconds after the login happens.  If my script exits too soon, then the tunnel won't get established.  I'm not sure how to properly deal with this.

Like I said, some of the machines that I run ansible on have hardcoded passwords and it works fine when ssh_user and ssh_pass is set as facts for your host.

[Michael Spiegle]

About half of my machines are in Amazon/EC2.  In order to solve the chicken/egg problem, I write out a "user_data" script which installs some SSH keys for me to the root user of the VM upon first boot.  This allows me to run my initial bootstrap and get the machine joined to the domain, then I can optionally remove the root keys.  This is all done with Ansible.

[Ananda Debnath]

Many thanks Michael for the detailed writeup!

I'm going to experiment with my setup (quite similar to your's - multiple jumpboxes in multiple locations, some AWS usage as well) to see how much of your approach I can adapt to mine.

[Benny Kusman]

this is a great step.

im able to establish the tunneling with the jumphost.

but i was wondering, what did you put in the hosts filfe ?

im still not able to reach the server

[Michael Spiegle]

By "hosts" file, do you mean /etc/hosts or the hosts in ssh.config?  Also, if you just run SSH by hand to login to a remote host, what happens?

Ex:

$ ssh -F ssh.config -fN user@some_jumpbox
$ ssh -F ssh.config user@some_server_behind_jumpbox

[Sosys]

Hi

ssh -F ssh.config -fN user@some_jumpbox  --> im able to establish this and send it back to the background

ssh -F ssh.config user@someserverbehindjumpbox --> does not work. the message is: ssh_exchange_identification: Connection closed by remote host

the "hosts" im referring to is in the ssh.config. This is my ssh.config

==============================================

#jumpboxes first (most specific hosts first)

Host jumpboxserver

 ControlMaster yes

 ControlPath ~/.ssh/master-%r@ jumpboxserver :%p

 StrictHostkeyChecking no

 ProxyCommand none

 ControlPersist 10m

# groups of machines that can be accessed by the above jumpboxes

 

Host *.someserverbehindjumpbox.com

 ControlMaster no

 ProxyCommand ssh -S ~/.ssh/master-*@jumpboxserver:%p -W  %h:%p

# this makes ansible faster by reusing connections

Host *

 ControlMaster auto

 ControlPersist 300s

 ControlPath ~/.ssh/ansible-%r@%h:%p

==============================================

--

You received this message because you are subscribed to the Google Groups "Ansible Project" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscribe@googlegroups.com.
To post to this group, send email to ansible-project@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/53c9364c-3062-4f8c-8f14-be041690e8fa%40googlegroups.com.

[Benny Kusman]

This is the -vvv output:

[root@WW-GVXQLC2 ansible]# ssh -F ssh.config bkusman@serverbehindjumpbox -p 670 -vvv

OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017

debug1: Reading configuration data ssh.config

debug1: ssh.config line 28: Applying options for *

debug1: Executing proxy command: exec ssh -S ~/.ssh/master-*@jumpboxserver:670 -W  serverbehindjumpbox:670

debug1: permanently_set_uid: 0/0

debug1: permanently_drop_suid: 0

debug1: key_load_public: No such file or directory

debug1: identity file /root/.ssh/id_rsa type -1

debug1: key_load_public: No such file or directory

debug1: identity file /root/.ssh/id_rsa-cert type -1

debug1: key_load_public: No such file or directory

debug1: identity file /root/.ssh/id_dsa type -1

debug1: key_load_public: No such file or directory

debug1: identity file /root/.ssh/id_dsa-cert type -1

debug1: key_load_public: No such file or directory

debug1: identity file /root/.ssh/id_ecdsa type -1

debug1: key_load_public: No such file or directory

debug1: identity file /root/.ssh/id_ecdsa-cert type -1

debug1: key_load_public: No such file or directory

debug1: identity file /root/.ssh/id_ed25519 type -1

debug1: key_load_public: No such file or directory

debug1: identity file /root/.ssh/id_ed25519-cert type -1

debug1: Enabling compatibility mode for protocol 2.0

debug1: Local version string SSH-2.0-OpenSSH_7.4

usage: ssh [-1246AaCfGgKkMNnqsTtVvXxYy] [-b bind_address] [-c cipher_spec]

           [-D [bind_address:]port] [-E log_file] [-e escape_char]

           [-F configfile] [-I pkcs11] [-i identity_file]

           [-J [user@]host[:port]] [-L address] [-l login_name] [-m mac_spec]

           [-O ctl_cmd] [-o option] [-p port] [-Q query_option] [-R address]

           [-S ctl_path] [-W host:port] [-w local_tun[:remote_tun]]

           [user@]hostname [command]

ssh_exchange_identification: Connection closed by remote host

[Michael Spiegle]

I think this is because the SSH command always expects a hostname even if there isn't a hostname to use.  In your ssh -vvv debug output, you can see that every single option is inside of [optional] brackets EXCEPT for the hostname.  SSH won't actually use this hostname for anything, it just wants some dumb string in that field.  In my config, I use the string "remote" like
the following.  Note that the host "remote" doesn't exist and you'd probably receive an error under normal circumstances:

Host *.someserverbehindjumpbox.com
 ControlMaster no

 ProxyCommand ssh -S ~/.ssh/master-*@jumpboxserver:%p -W %h:%p

​ re

mote

​

--
You received this message because you are subscribed to a topic in the Google Groups "Ansible Project" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ansible-project/-AFEyk69T8k/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ansible-project+unsubscribe@googlegroups.com.

To post to this group, send email to ansible-project@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/688b9432-4580-4369-ad37-4ea106977cdd%40googlegroups.com.

[Benny Kusman]

Michael,

you are SO RIGHT!

now the tunneling works after your clue! 

completely missed out this "remote"

To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/53c9364c-3062-4f8c-8f14-be041690e8fa%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Ansible Project" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ansible-project/-AFEyk69T8k/unsubscribe.

To unsubscribe from this group and all its topics, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.