Ansible provisioning through bastion SSH host

[and...@edlio.com]

Hi folks,

I'm encountering issues provisioning with Ansible via SSH from outside our datacenter (well, AWS Virtual Private Cloud).

Our bastion hosts ("jump boxes") are configured to accept SSH keys from staff members.  Boxes behind the bastion hosts accept only the SSH keys created for the jumpbox user on the bastion hosts.  This makes our administration easier in the event of key compromise staff turnover (only need to rotate out on the jumpboxes).

The ProxyCommand examples I've seen for Ansible rely on agent forwarding and netcat.  I want Ansible to SSH in to the bastion host, then run a second SSH command with the private key of the bastion host.  Thus agent forwarding is useless here.    I've tried replacing the netcat ProxyCommand with "ssh -q -t awsjumpbox1 ssh -p %p %h" and I get mysterious failures that even -vvvv doesn't provide much insight on.  For instance:

fatal: [172.31.17.144] => SSH Error: -bash: line 1: SSH-2.0-OpenSSH_6.6.1_hpn13v11: command not found
It is sometimes useful to re-run the command using -vvvv, which prints SSH debug output to help diagnose the issue.

Our ansible.cfg looks like:

[defaults]
host_key_checking = False

[ssh_connection]
ssh_args = -o ControlPersist=15m -F ssh.config -q
scp_if_ssh = True
control_path = ~/.ssh/mux-%%r@%%h:%%p

Our ssh.config (simplified) looks like:

Host jumpbox
    HostName (redacted)
    User (redacted)
    BatchMode              yes
    PasswordAuthentication no

Host *
    ServerAliveInterval    60
    TCPKeepAlive           yes
    #ProxyCommand           ssh -q -A -t jumpbox nc %h %p  # have tried this one too
    ProxyCommand           ssh -q -t jumpbox ssh -p %p %h
    ControlMaster          auto
    ControlPath            ~/.ssh/ansible-%r@%h:%p
    ControlPersist         8h
    User                   (redacted)

Our boxes are running the standard Amazon AMI (which I am led to believe are CentOS-based).  We are open to rethinking the aforementioned strategy re: the keys if that's the only way.

I'll be interested in anyone's thoughts on the matter.  Thank you.

Best regards,

Andrew Merenbach
Senior DevOps Engineer at Edlio Inc

[and...@edlio.com]

Correction: the non-working command is "ssh -q -t jumpbox1 ssh -p %p %h" (not awsjumpbox1)

[Paul Markham]

I had a play with this and I can't get it to work either.

Whilst running 'ssh -tt bastion01 ssh -tt server01' does works from the command line, it doesn't work when used as a ProxyCommand in .ssh/config. 

This makes sense though. When using ProxyCommand you're telling your ssh client to connect to a remote server, the hops in between are transparent to it. So, the private key needs to be on your local machine, not on some intermediate machine.

If you're managing accounts and keys through Ansible, changing keys or removing an account shouldn't be a big deal.