Cant get ansible to properly route through bastion host

[Emil Safonov]

Hey Everyone,

I've been having difficulty getting my ansible playbooks to route through a bastion host (without ssh keys). For testing I have a cisco virl (virtual internet routing lab) simulation up and running with virtualized network devices, and the bastion host should be the server that is running the simulations. 

I can confirm that, after defining an ssh.cfg file, I can ssh into the virtualized network device via the command "ssh -F ssh.cfg xxx.xxx.1.xxx -v" just fine, and first input the password for the bastion host and then input the password to the cisco virtualized device. But each time I try and run the playbook I get either a timeout or "ssh protocol banner cant be read". How can I confirm ansible is trying to connect to the bastion host first? I never see that in the terminal stdout, even with -vvvvvv it never shows it connecting to the bastion host but just to the device defined in hosts.virl. 

Command to run playbook: ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook playbook.yml -i hosts.virl -k -vvvvvv

Below are my files (playbook, host, ansible.cfg and ssh.cfg + groupvars/all.yml and groupvars/ios.yml).

playbook.yml

---

- name: test through bastion-host,  show version

  hosts: ios

  gather_facts: false

  connection: network_cli

  tasks:

    - name: show version

      ios_command:

        commands:

          - show version

      register: version_out

    - debug:

        var: version_out

hosts.virl

[ios]

xxx.xxx.1.xxx

[ios:vars]

ansible_user=cisco

ansible_password=cisco

ansible_become_pass=cisco

The user, pass and enable pass are always default cisco on a simulated device in virl, so I just keep them default to save the time for vaulting the passwords during testing

ansible.cfg

[defaults]

host_key_checking = False

scp_if_ssh = True

[paramiko_connection]

host_key_auto_add = True

look_for_keys = False

host_key_checking = False

[ssh_connection]

ssh_args = -F /home/<MY_USER>/projects/ansible/development/ssh.cfg -o ControlMaster=auto -o ControlPersist=15m -o StrictHostKeyChecking=no -o USerKnownHostsFile=/dev/null

control_path = /tmp/ansible-%%r@%%h:%%p

Paramiko caused trouble before so I had to add those paramiko_connection values

ssh.cfg

Host xxx.xxx.xxx.55*

    Hostname xxx.xxx.xxx.55

    Port 10015

    User <BASTION_HOST_USERNAME>

    TCPKeepAlive yes

    ControlMaster auto

    ControlPath ~/.ssh/ansible-%r%h:%p

    ControlPersist 15m

    StrictHostKeyChecking no

    ProxyCommand none

    LogLevel Error

Host xxx.xxx.1.*

    User cisco

    StrictHostKeyChecking no

    ProxyCommand ssh -p 10015 -q -A -W %h:%p <BASTION_HOST_USERNAME>@xxx.xxx.xxx.55 -v

Host *
    ProxyCommand none
    Port 22
    StrictHostKeyChecking no

Port has to be set for the bastion host since each individual simulation runs on its own port, starting with 10001 up to however many sims there are. Current simulation is 10015. I also had to add the * to the Host at the top since it has to match to any of the sims ports.

group_vars/all and group_vars/ios combined

ansible_become: yes

ansible_become_method: enable

ansible_connection: network_cli

ansible_python_interpreter: "/home/<MY_USER>/projects/ansible/python-3.6-ansible/bin/python"
ansible_network_os: ios

Error output:

(python-3.6-ansible) [<USER>@<DEVICE> development]$ ansible-playbook test-bacchus.yml -i hosts.virl -vvvvvvv

ansible-playbook 2.9.7

  config file = /home/<USERNAME>/projects/ansible/development/ansible.cfg

  configured module search path = ['/home/<USERNAME>/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']

  ansible python module location = /home/<USERNAME>/projects/ansible/python-3.6-ansible/lib/python3.6/site-packages/ansible

  executable location = /home/<USERNAME>/projects/ansible/python-3.6-ansible/bin/ansible-playbook

  python version = 3.6.8 (default, Nov 27 2019, 14:21:59) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39.0.1)]

Using /home/<USERNAME>/projects/ansible/development/ansible.cfg as config file

setting up inventory plugins

host_list declined parsing /home/<USERNAME>/projects/ansible/development/hosts.virl as it did not pass its verify_file() method

script declined parsing /home/<USERNAME>/projects/ansible/development/hosts.virl as it did not pass its verify_file() method

auto declined parsing /home/<USERNAME>/projects/ansible/development/hosts.virl as it did not pass its verify_file() method

yaml declined parsing /home/<USERNAME>/projects/ansible/development/hosts.virl as it did not pass its verify_file() method

Parsed /home/<USERNAME>/projects/ansible/development/hosts.virl inventory source with ini plugin

Loading callback plugin default of type stdout, v2.0 from /home/<USERNAME>/projects/ansible/python-3.6-ansible/lib/python3.6/site-packages/ansible/plugins/callback/default.py

PLAYBOOK: test-bacchus.yml ********************************************************************************************************************************************************************************

Positional arguments: test-bacchus.yml

verbosity: 7

connection: smart

timeout: 10

become_method: sudo

tags: ('all',)

inventory: ('/home/<USERNAME>/projects/ansible/development/hosts.virl',)

forks: 5

1 plays in test-bacchus.yml

PLAY [test bacchus show version] **************************************************************************************************************************************************************************

META: ran handlers

TASK [show version] ***************************************************************************************************************************************************************************************

task path: /home/<USERNAME>/projects/ansible/development/test-bacchus.yml:7

<xxx.xxx.1.xxx> attempting to start connection

<xxx.xxx.1.xxx> using connection plugin network_cli

<xxx.xxx.1.xxx> local domain socket does not exist, starting it

<xxx.xxx.1.xxx> control socket path is /home/<USERNAME>/.ansible/pc/c0bb744b49

<xxx.xxx.1.xxx> local domain socket listeners started successfully

<xxx.xxx.1.xxx> loaded cliconf plugin ios from path /home/<USERNAME>/projects/ansible/python-3.6-ansible/lib/python3.6/site-packages/ansible/plugins/cliconf/ios.py for network_os ios

<xxx.xxx.1.xxx> 

<xxx.xxx.1.xxx> local domain socket path is /home/<USERNAME>/.ansible/pc/c0bb744b49

fatal: [xxx.xxx.1.xxx]: FAILED! => {

    "changed": false,

    "msg": "timed out"

}

PLAY RECAP ************************************************************************************************************************************************************************************************

xxx.xxx.1.xxx               : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0 

Sorry for the massive post / code-vomit. Im at a complete loss at what to do here, and the verbose mode of ansible never states that it even goes through the bastion host at all so its been difficult troubleshooting. Thanks for all the help

[Venu Devannagari]

Hi @Emil Safonov

Did you resolve the issue??

If yes can you please help me here I am also facing the same issue.

Thanks,

D.Venu.

[Emil Safonov]

Hey Venu,

I was able to resolve the issue using a linux module called sshpass. I'll show below:

hosts.through_bastion

[ios]

10.*.*.1

10.*.*.2

[ios:vars]

ansible_user=<username>

ansible_password=<password>

ansible_become_pass=<password>

ansible_network_os=ios

network_os=ios

ansible_ssh_common_args=" -o ProxyCommand=\"sshpass -p '<password_to_bastion>' ssh -q -W %h:%p <bastion_username>@<bastion_ip_or_dns\""

So whats happening here is we are using ansible_ssh_common_args="the above string". That means ansible will take that aboove string and instead of performing "ssh username@hostname" it will perform "sshpass -p <password> etc." as its ssh command. 

The sshpass module allows you to input a password that will act on the bastion host, then proxy all commands through that instead of just using ssh. This is the only way I could get it to authenticate WITHOUT an ssh key. One problem you might see is exposing your usernames/passwords in these files, however what you can do is use ansible vault to save credentials within your group_vars/all.yml file, set them to a variable and then use those vaulted credentials during playbook execution. 

Hope this helps!