problem with ansible bastion hosts /playbook connectivity

Simon Tyler

unread,

Jul 8, 2014, 4:05:26 PM7/8/14

to ansible...@googlegroups.com

Hello,

I was running an older version of ansible, and I couldn't find a way to get it to connect through a bastion host to run scripts against a destination server, so I tried upgrading ansible. (yum upgrade to 1.6.2)

Now it appears that some things don't work the way they did eg. it doesn't seem to recognize the hosts list properly. It used to be that the first line in the .yml file was:

- hosts: $Server_name

but, now that entry results in an error

[prod_deploy@srvtoolbox simontest]$ ansible-playbook fetch_mysql_config2.yml --extra-vars="Server_name=jruby01"

[WARNING]: The version of gmp you have installed has a known issue regarding

timing vulnerabilities when used with pycrypto. If possible, you should update

it (ie. yum update gmp).

PLAY [$Server_name] ***********************************************************

skipping: no hosts matched

so, I can no longer pass the host name at the command line? or the syntax has changed. fine that's a new issue i guess.

If I enter the host name in the playbook itself, it appears to work, if the host name is in the ansible list.

I've tried a number of methods to get it to recognize host names in ~/.ssh/config but all failed?

eg. [prod_deploy@srvtoolbox simontest]$ ansible-playbook -c ssh fetch_mysql_config2.yml

[WARNING]: The version of gmp you have installed has a known issue regarding

timing vulnerabilities when used with pycrypto. If possible, you should update

it (ie. yum update gmp).

PLAY [tmdb02] *****************************************************************

skipping: no hosts matched

PLAY RECAP ********************************************************************

my .ssh/config file:

[prod_deploy@srvtoolbox simontest]$ cat ~/.ssh/config

Host terremarkjump

HostName 208.39.105.26

User prod_deploy

Host solr4-dih

User prod_deploy

HostName 10.124.33.204

ProxyCommand ssh -q terremarkjump ncat %h 22

Host sprt_liferay_10

User prod_deploy

HostName ec2-54-202-175-96.us-west-2.compute.amazonaws.com

Host tmdb02

User prod_deploy

HostName 10.124.33.250

ProxyCommand ssh -q terremarkjump ncat %h 22

my test playbook:

- hosts: tmdb02

user: prod_deploy

sudo: yes

tasks:

- name: get INVENTORY

action: fetch src=/etc/my.cnf dest=~

I looked briefly at /usr/lib/python2.6/site-packages/ansible/runner/connection_plugins/ssh.py but, I don't know enough python to be able to tell if this has anything to do with using the ~/.ssh/config file.

Is anyone aware if the current state of ansible/ssh (not paramiko) documented in plain english, or is there a way to use jump servers via ssh with ansible 1.6.2?

It may well be that I'm doing something silly,

much obliged for any assist

Simon

Brian Coca

unread,

Jul 8, 2014, 10:37:58 PM7/8/14

to ansible...@googlegroups.com

The $ variables have been dropped from current ansible, they have been deprecated for a loong time. The new way to do the host variables would be writing plays like this:

hosts: "{{server_name}}"

and pass on the command line as before, there is some more advanced stuff you can do now, like using --limit or doing this

hosts: "{{server_name|default('all')}}"

the ssh config should be picked up normally, I still use it for some bastion jump hosts w/o issues.

Simon Tyler

unread,

Jul 9, 2014, 9:48:13 AM7/9/14

to ansible...@googlegroups.com

ok i havent used Ansible before. Are you saying I don't have to pass any switch to use SSH; it uses Paramiko and if it doesn't find a hostname, it tries ssh config? it just keeps giving me the error"No hosts matched",

Henry Finucane

unread,

Jul 9, 2014, 9:58:48 AM7/9/14

to ansible...@googlegroups.com

On Jul 9, 2014 6:48 AM, "Simon Tyler" <simon.syst...@gmail.com> wrote:
>
> ok i havent used Ansible before. Are you saying I don't have to pass any switch to use SSH; it uses Paramiko and if it doesn't find a hostname, it tries ssh config? it just keeps giving me the error"No hosts matched",

Current versions of ansible use openssh as a library for connections by default. You can also ask ansible to use paramiko instead. If you're using openssh, your ssh config will get picked up, if you're using paramiko, it won't.

"No hosts matched" is, I believe, unrelated- it means that there are no hosts in your inventory in the specified group.

> On Tuesday, July 8, 2014 10:37:58 PM UTC-4, Brian Coca wrote:
>>
>> The $ variables have been dropped from current ansible, they have been deprecated for a loong time. The new way to do the host variables would be writing plays like this:
>>
>> hosts: "{{server_name}}"
>>
>> and pass on the command line as before, there is some more advanced stuff you can do now, like using --limit or doing this
>>
>> hosts: "{{server_name|default('all')}}"
>>
>>
>> the ssh config should be picked up normally, I still use it for some bastion jump hosts w/o issues.
>>
>

> --
> You received this message because you are subscribed to the Google Groups "Ansible Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
> To post to this group, send email to ansible...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/6935e31d-d83e-4987-9f07-bf0f0937806d%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

Simon Tyler

unread,

Jul 9, 2014, 10:56:28 AM7/9/14

to ansible...@googlegroups.com

ok clearly I am doing something dumb.

I have an entry in ~/.ssh/config for destination host: tmdb02

Host tmdb02

User prod_deploy

HostName 10.124.33.250

ProxyCommand ssh -q terremarkjump ncat %h 22

I can ssh to it normally eg. ssh tmdb02

I've updated a test playbook to use the new server name syntax eg.

- hosts: "{{server_name}}"

user: prod_deploy

sudo: yes

tasks:

# Store file into /tmp/fetched/host.example.com/tmp/somefile

- name: get INVENTORY

action: fetch src=/etc/my.cnf dest=~

I try to run it:

ansible-playbook /home/prod_deploy/simontest/fetch_mysql_config2.yml --extra-vars="server_name=jruby03"

and this works: jruby03 is listed in /etc/ansible/hosts

If I try a name that is listed in my ssh config, it fails as previously indicated. So, I need to add an entry or two to /etc/ansible/hosts. The server I'm trying to hit is on the other side of a jump server. How do i set this up in /etc/ansible/hosts?

--
You received this message because you are subscribed to a topic in the Google Groups "Ansible Project" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ansible-project/1sYMJ1uTAws/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ansible-proje...@googlegroups.com.

To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CAGYSzJ_nuP9rUS224TWUhmNMssr_TeSbq6fOGobcOXdP%2BpgehQ%40mail.gmail.com.

Henry Finucane

unread,

Jul 9, 2014, 11:03:54 AM7/9/14

to ansible...@googlegroups.com

Ansible only wants to talk to hosts in its inventory. If you want your inventory to be your ssh config file, there is a dynamic inventory script called 'ssh_config.py' that ships with ansible that will do that for you.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CADiFY2fYb%2BJa5TS92po4A0eNhC23EMo%2BpCsqC4PmrAy11yEJZg%40mail.gmail.com.

Brian Coca

unread,

Jul 9, 2014, 12:18:18 PM7/9/14

to ansible...@googlegroups.com

also, remove "user: prod_deploy" from your playbook, using a 'user' will avoid using your .ssh/config file, since you are already setting it there, it is redundant.

Simon Tyler

unread,

Jul 9, 2014, 1:56:19 PM7/9/14

to ansible...@googlegroups.com

Hi Henry,

I am most definitely stupider than normal; I blame a recent pigeon infestation and the resulting lack of sleep.

I did not find that script on my system, so I downloaded it from here https://github.com/ansible/ansible/blob/release1.6.2/plugins/inventory/ssh_config.py. It appears to be throwing a python error:

[prod_deploy@srvtoolbox ansible]$ ./ssh_config.py --list

Traceback (most recent call last):

File "./ssh_config.py", line 111, in <module>

main(sys.argv[1:])

File "./ssh_config.py", line 105, in main

print_list()

File "./ssh_config.py", line 72, in print_list

cfg = get_config()

File "./ssh_config.py", line 67, in get_config

ret_dict[host] = _copy['config']

KeyError: 'config'

I think you wrote it? Perhaps it would be better if I went away and hunted some pigeons, and got some sleep, and then come back. I feel like I'm getting closer to a solution, even if I'm not getting any smarter.

Have you got a suggestion for next steps? Thank you for your patience,

Henry Finucane

unread,

Jul 9, 2014, 2:55:02 PM7/9/14

to ansible...@googlegroups.com

On Wed, Jul 9, 2014 at 10:56 AM, Simon Tyler
<simon.syst...@gmail.com> wrote:
> Hi Henry,
>
> I am most definitely stupider than normal; I blame a recent pigeon
> infestation and the resulting lack of sleep.
>
> I did not find that script on my system, so I downloaded it from here
> https://github.com/ansible/ansible/blob/release1.6.2/plugins/inventory/ssh_config.py.
> It appears to be throwing a python error:
> [prod_deploy@srvtoolbox ansible]$ ./ssh_config.py --list
> Traceback (most recent call last):
> File "./ssh_config.py", line 111, in <module>
> main(sys.argv[1:])
> File "./ssh_config.py", line 105, in main
> print_list()
> File "./ssh_config.py", line 72, in print_list
> cfg = get_config()
> File "./ssh_config.py", line 67, in get_config
> ret_dict[host] = _copy['config']
> KeyError: 'config'
>
> I think you wrote it? Perhaps it would be better if I went away and hunted
> some pigeons, and got some sleep, and then come back. I feel like I'm
> getting closer to a solution, even if I'm not getting any smarter.

I did- or at least the line in question. I may have relied too closely
on my ability to read the paramiko source code and infer its api
contract.

I wrote a stupid patch:
https://github.com/hfinucane/ansible/tree/feature/paramiko-safety

let me know if that works out for you, or if more things are broken.

> https://groups.google.com/d/msgid/ansible-project/ee6d7ecd-28b7-4a7c-a934-f1b91d5f7902%40googlegroups.com.

>
> For more options, visit https://groups.google.com/d/optout.

--

-----------------------
| Henry Finucane
| "I hear aphorisms are popular"
-----------------------

Simon Tyler

unread,

Jul 10, 2014, 11:26:50 AM7/10/14

to ansible...@googlegroups.com

Hi Henry,

Thank you so much for such quick responses!

I have not used git much, here is what I did:

-browsed to https://github.com/hfinucane/ansible/blob/feature/paramiko-safety/plugins/inventory/ssh_config.py

-cut and paste the script into /usr/lib/python2.6/site-packages/ansible/runner/connection_plugins/

-since all of the ansible files are owned by root, I put a copy of ssh_config.py in my user directory so I could run it directly, and made it executable

here are results of test

[prod_deploy@srvtoolbox simontest]$ ./ssh_config.py --list

{"ssh_config": [], "_meta": {"hostvars": {}}}

and

[prod_deploy@srvtoolbox simontest]$ ansible tmdb02 -m ping

No hosts matched

I hope that I've gone about this in an acceptable way; thank you again for your patience,

Simon

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CAGYSzJ-w%3DFhh4YN7WkA1gidv-1is_UQkUHawd4OxsijvHftrAw%40mail.gmail.com.

Henry Finucane

unread,

Jul 10, 2014, 1:12:48 PM7/10/14

to ansible...@googlegroups.com

I talked to Simon a little bit out of band, it looks like the problem
is his version of Paramiko- it predates both the initial and the
working ProxyCommand implementation. You need Paramiko >= 1.10 for
parsing support, and if you want to use Paramiko as a transport you
want >= 1.10.6 .

On Thu, Jul 10, 2014 at 8:26 AM, Simon Tyler

> https://groups.google.com/d/msgid/ansible-project/CADiFY2cn3HRnUccyMZCgpDaGBZDMVE0n2py3-wVrYRRCx%3DPyEg%40mail.gmail.com.

Simon Tyler

unread,

Jul 11, 2014, 2:01:51 PM7/11/14

to ansible...@googlegroups.com

Hello again, we have updated Paramiko but it appears I still have issues.

ssh_config.py works when I don't have to hop through a bastion host. When I have to go through a bastion host, it appears to ignore the Proxy variable. I created an entry in ssh config file for jruby1. (The name in the ansible hosts file is jruby01):

[prod_deploy@srvtoolbox simontest]$ ansible-playbook -i /home/prod_deploy/simontest/ssh_config.py fetch_mysql_config2.yml --extra-vars="server_name=jruby1"

prod_deploy@srvtoolbox simontest]$ ssh -V

OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013

[prod_deploy@srvtoolbox log]$ ANSIBLE_SSH_ARGS="" ansible tmdb02 -i /home/prod_deploy/simontest/ssh_config.py -m ping -c ssh -vvvv

<10.124.33.250> ESTABLISH CONNECTION FOR USER: prod_deploy

<10.124.33.250> REMOTE_MODULE ping

<10.124.33.250> EXEC ['ssh', '-C', '-tt', '-vvv', '-o', 'Port=22', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'ConnectTimeout=10', u'10.124.xx.xxx', "/bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1405085298.07-20022175021754 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1405085298.07-20022175021754 && echo $HOME/.ansible/tmp/ansible-tmp-1405085298.07-20022175021754'"]

tmdb02 | FAILED => SSH encountered an unknown error. The output was:

OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013

debug1: Reading configuration data /home/prod_deploy/.ssh/config

debug1: Reading configuration data /etc/ssh/ssh_config

debug1: Applying options for *

debug2: ssh_connect: needpriv 0

debug1: Connecting to 10.124.xx.xxx [10.124.xx.xxx] port 22.

debug2: fd 4 setting O_NONBLOCK

debug1: connect to address 10.124.xx.xxx port 22: No route to host

ssh: connect to host 10.124.xx.xxx port 22: No route to host

[root@srvtoolbox tmp]# pip show pycrypto

---

Name: pycrypto

Version: 2.0.1

Location: /usr/lib64/python2.6/site-packages

Requires:

[root@srvtoolbox tmp]# pip show paramiko

---

Name: paramiko

Version: 1.14.0

Location: /usr/lib/python2.6/site-packages

Requires: pycrypto, ecdsa

Again, I'll put my /home/prod_deploy/.ssh/config file here, i x'ed out some of the ips:

[prod_deploy@srvtoolbox log]$ cat /home/prod_deploy/.ssh/config

Host terremarkjump

HostName 208.39.xxx.xx

User prod_deploy

Host solr4-dih

User prod_deploy

HostName 10.124.xx.xxx

ProxyCommand ssh -q terremarkjump ncat %h 22

Host sprt_liferay_10

User prod_deploy

HostName ec2-54-202-xxx-xx.us-west-2.compute.amazonaws.com

Host tmdb02

User prod_deploy

HostName 10.124.xx.xxx

ProxyCommand ssh -q terremarkjump ncat %h 22

Host jruby1

HostName ec2-54-245-xxx-xxx.us-west-2.compute.amazonaws.com

User prod_deploy

and for a comparison, here is the same user, connecting to the same host tmdb02, using the same config file, but directly using ssh instead of using ansible:

[prod_deploy@srvtoolbox log]$ ssh tmdb02 -vvvv

OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013

debug1: Reading configuration data /home/prod_deploy/.ssh/config

debug1: Applying options for tmdb02

debug1: Reading configuration data /etc/ssh/ssh_config

debug1: Applying options for *

debug2: ssh_connect: needpriv 0

debug1: Executing proxy command: exec ssh -q terremarkjump ncat 10.124.xx.xxx 22

debug1: permanently_drop_suid: 508

debug1: identity file /home/prod_deploy/.ssh/identity type -1

debug1: identity file /home/prod_deploy/.ssh/identity-cert type -1

debug3: Not a RSA1 key file /home/prod_deploy/.ssh/id_rsa.

debug2: key_type_from_name: unknown key type '-----BEGIN'

debug3: key_read: missing keytype

debug3: key_read: missing whitespace

debug2: key_type_from_name: unknown key type '-----END'

debug3: key_read: missing keytype

debug1: identity file /home/prod_deploy/.ssh/id_rsa type 1

debug1: identity file /home/prod_deploy/.ssh/id_rsa-cert type -1

debug1: identity file /home/prod_deploy/.ssh/id_dsa type -1

debug1: identity file /home/prod_deploy/.ssh/id_dsa-cert type -1

debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3

debug1: match: OpenSSH_5.3 pat OpenSSH*

debug1: Enabling compatibility mode for protocol 2.0

debug1: Local version string SSH-2.0-OpenSSH_5.3

debug2: fd 5 setting O_NONBLOCK

debug2: fd 4 setting O_NONBLOCK

debug1: SSH2_MSG_KEXINIT sent

debug3: Wrote 960 bytes for a total of 981

debug1: SSH2_MSG_KEXINIT received

debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1

debug2: kex_parse_kexinit: ssh-rsa-...@openssh.com,ssh-dss-...@openssh.com,ssh-rsa-c...@openssh.com,ssh-dss-c...@openssh.com,ssh-rsa,ssh-dss

debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-c...@lysator.liu.se

debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ri...@openssh.com,hmac-sha1-96,hmac-md5-96

debug2: kex_parse_kexinit: none,zl...@openssh.com,zlib

debug2: kex_parse_kexinit:

debug2: kex_parse_kexinit: first_kex_follows 0

debug2: kex_parse_kexinit: reserved 0

debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1

debug2: kex_parse_kexinit: ssh-rsa,ssh-dss

debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr

debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-rip...@openssh.com,hmac-sha1-96,hmac-md5-96

debug2: kex_parse_kexinit: none,zl...@openssh.com

debug2: kex_parse_kexinit:

debug2: kex_parse_kexinit: first_kex_follows 0

debug2: kex_parse_kexinit: reserved 0

debug2: mac_setup: found hmac-md5

debug1: kex: server->client aes128-ctr hmac-md5 none

debug2: mac_setup: found hmac-md5

debug1: kex: client->server aes128-ctr hmac-md5 none

debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent

debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP

debug3: Wrote 24 bytes for a total of 1005

debug2: dh_gen_key: priv key bits set: 141/256

debug2: bits set: 511/1024

debug1: SSH2_MSG_KEX_DH_GEX_INIT sent

debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY

debug3: Wrote 144 bytes for a total of 1149

debug3: check_host_in_hostfile: host 10.124.xx.xxx filename /home/prod_deploy/.ssh/known_hosts

debug3: check_host_in_hostfile: match line 111

debug1: Host '10.124.33.250' is known and matches the RSA host key.

debug1: Found key in /home/prod_deploy/.ssh/known_hosts:111

debug2: bits set: 553/1024

debug1: ssh_rsa_verify: signature correct

debug2: kex_derive_keys

debug2: set_newkeys: mode 1

debug1: SSH2_MSG_NEWKEYS sent

debug1: expecting SSH2_MSG_NEWKEYS

debug3: Wrote 16 bytes for a total of 1165

debug2: set_newkeys: mode 0

debug1: SSH2_MSG_NEWKEYS received

debug1: SSH2_MSG_SERVICE_REQUEST sent

debug3: Wrote 48 bytes for a total of 1213

debug2: service_accept: ssh-userauth

debug1: SSH2_MSG_SERVICE_ACCEPT received

debug2: key: /home/prod_deploy/.ssh/identity ((nil))

debug2: key: /home/prod_deploy/.ssh/id_rsa (0x7f330dd7cc30)

debug2: key: /home/prod_deploy/.ssh/id_dsa ((nil))

debug3: Wrote 80 bytes for a total of 1293

debug3: input_userauth_banner

******************************************************************************

* * * * * * * * * * * * * * * * * * NOTICE * * * * * * * * * * * * * * * * *

* *

* This system is restricted to authorized users for business purposes only. *

* *

* Unauthorized access or use is a violation of company policy and the law. *

* *

* This system may be monitored for administrative and security reasons. *

* *

* By proceeding, you acknowledge that: *

* (1) you have read and understand this notice, and *

* (2) consent to system monitoring. *

* *

* * * * * * * * * * * * * * * * * * NOTICE * * * * * * * * * * * * * * * * *

******************************************************************************

debug1: Authentications that can continue: password

debug3: start over, passed a different list password

debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password

debug3: authmethod_lookup password

debug3: remaining preferred: ,gssapi-with-mic,publickey,keyboard-interactive,password

debug3: authmethod_is_enabled password

debug1: Next authentication method: password

It appears to me that ansible is not executing the Proxy command; it's trying to connect to the host directly, which of course fails, because there is no connection to that network except via the jump server.

Does anyone have any suggestions? At this point I am considering a clean build of a new vm and fresh install of ansible,

Reply all

Reply to author

Forward