Unable to reach to EC2 instance using dynamic inventory with a single region

[Nirav Radia]

Hi all,

I am pretty new to Ansible. I am using ec2.py to connect to my EC2 instances and run my ansible scripts on them. Previously, it was working fine when I was using "all" regions. But now when I change the region to a specific one region (any of the available ones), in ec2.ini, it gives me below error for the first time and after second time onward, it works well.

UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh.", "unreachable": true}

What I do is use ec2 module to create a instance and then wait_for port 22 on each host to be up. It wait successfully and moves forward but in the immediate next step when I try to connect to the instance, it gives above error first time. Here is my wait_for task: (ec2_server is the variable registered from ec2 module)

name: wait for ssh server to be running

    wait_for: host={{ item.public_dns_name }} port=22

    with_items: "{{ec2_server.instances | default([])}}"

    when: item.state == 'running'

I ping in between to the instance using private ip and the ping succeeds. I suspect it is some timing issue or DNS name resolution issue with AWS. Has anyone faced such problem before?

Any help would be appreciated. Thanks !

[Alexander H. Laughlin]

Would you mind to post your ec2.ini with the credentials taken out? Also, what is the output of ec2.py when you run it alone?

[Nirav Radia]

Thanks @Alexander for your reply. Here is ec2.ini

# Ansible EC2 external inventory script settings
#


[ec2]


# to talk to a private eucalyptus instance uncomment these lines
# and edit edit eucalyptus_host to be the host name of your cloud controller
#eucalyptus = True
#eucalyptus_host = clc.cloud.domain.org


# AWS regions to make calls to. Set this to 'all' to make request to all regions
# in AWS and merge the results together. Alternatively, set this to a comma
# separated list of regions. E.g. 'us-east-1,us-west-1,us-west-2'
regions = all
regions_exclude = us-gov-west-1,cn-north-1


# When generating inventory, Ansible needs to know how to address a server.
# Each EC2 instance has a lot of variables associated with it. Here is the list:
#   http://docs.pythonboto.org/en/latest/ref/ec2.html#module-boto.ec2.instance
# Below are 2 variables that are used as the address of a server:
#   - destination_variable
#   - vpc_destination_variable


# This is the normal destination variable to use. If you are running Ansible
# from outside EC2, then 'public_dns_name' makes the most sense. If you are
# running Ansible from within EC2, then perhaps you want to use the internal
# address, and should set this to 'private_dns_name'. The key of an EC2 tag
# may optionally be used; however the boto instance variables hold precedence
# in the event of a collision.
destination_variable = public_dns_name


# For server inside a VPC, using DNS names may not make sense. When an instance
# has 'subnet_id' set, this variable is used. If the subnet is public, setting
# this to 'ip_address' will return the public IP address. For instances in a
# private subnet, this should be set to 'private_ip_address', and Ansible must
# be run from with EC2. The key of an EC2 tag may optionally be used; however
# the boto instance variables hold precedence in the event of a collision.
vpc_destination_variable = private_ip_address


# To tag instances on EC2 with the resource records that point to them from
# Route53, uncomment and set 'route53' to True.
route53 = False


# To exclude RDS instances from the inventory, uncomment and set to False.
rds = False


# Additionally, you can specify the list of zones to exclude looking up in
# 'route53_excluded_zones' as a comma-separated list.
# route53_excluded_zones = samplezone1.com, samplezone2.com


# By default, only EC2 instances in the 'running' state are returned. Set
# 'all_instances' to True to return all instances regardless of state.
all_instances = False


# By default, only RDS instances in the 'available' state are returned.  Set
# 'all_rds_instances' to True return all RDS instances regardless of state.
all_rds_instances = False


# API calls to EC2 are slow. For this reason, we cache the results of an API
# call. Set this to the path you want cache files to be written to. Two files
# will be written to this directory:
#   - ansible-ec2.cache
#   - ansible-ec2.index
cache_path = ~/.ansible/tmp


# The number of seconds a cache file is considered valid. After this many
# seconds, a new API call will be made, and the cache file will be updated.
# To disable the cache, set this value to 0
cache_max_age = 0


# Organize groups into a nested/hierarchy instead of a flat namespace.
nested_groups = False


# The EC2 inventory output can become very large. To manage its size,
# configure which groups should be created.
group_by_instance_id = False
group_by_region = True
group_by_availability_zone = False
group_by_ami_id = False
group_by_instance_type = False
group_by_key_pair = False
group_by_vpc_id = False
group_by_security_group = False
group_by_tag_keys = True
group_by_tag_none = False
group_by_route53_names = False
group_by_rds_engine = False
group_by_rds_parameter_group = False


# If you only want to include hosts that match a certain regular expression
# pattern_include = stage-*


# If you want to exclude any hosts that match a certain regular expression
pattern_exclude = datafactory*


# Instance filters can be used to control which instances are retrieved for
# inventory. For the full list of possible filters, please read the EC2 API
# docs: http://docs.aws.amazon.com/AWSEC2/latest/APIReference/ApiReference-query-DescribeInstances.html#query-DescribeInstances-filters
# Filters are key/value pairs separated by '=', to list multiple filters use
# a list separated by commas. See examples below.


# Retrieve only instances with (key=value) env=stage tag
#instance_filters = tag:cmx_env=dev


# Retrieve only instances with role=webservers OR role=dbservers tag
# instance_filters = tag:role=webservers,tag:role=dbservers


# Retrieve only t1.micro instances OR instances with tag env=stage
# instance_filters = instance-type=t1.micro,tag:env=stage


# You can use wildcards in filter values also. Below will list instances which
# tag Name value matches webservers1*
# (ex. webservers15, webservers1a, webservers123 etc)
# instance_filters = tag:Name=webservers1*


elasticache = False
expand_csv_tags = True

In my ec2.py file, I have modified the Regions section a little bit.

...
# Regions

        self.regions = []

        configRegions = os.getenv('AWS_DEFAULT_REGION', config.get('ec2', 'regions'))

        configRegions_exclude = config.get('ec2', 'regions_exclude')

        if (configRegions == 'all'):

            if self.eucalyptus_host:

                self.regions.append(boto.connect_euca(host=self.eucalyptus_host).region.name, **self.credentials)

            else:

                for regionInfo in ec2.regions():

                    if regionInfo.name not in configRegions_exclude:

                        self.regions.append(regionInfo.name)

        else:

            self.regions = configRegions.split(",")
...

I export environment variable AWS_DEFAULT_REGION just before the playbook call. When I run ec2.py, it returns me the host on which I am expecting to run the playbook.

PS: On a different note, I have added "search_regex=OpenSSH" in wait_for before the playbook call, but it didn't help either.

[Alexander H. Laughlin]

Hi Nirav,

I have a vague recollection of at some point having had a similar issue with dynamic inventory that was solved by placing the availability zone in the region field instead of the region. My memory is far from a reliable source of information so if you decide to give that a shot don't be surprised if it doesn't work. 

With that said, I just re-read your original post and your statement that this is only a problem on the first run stuck out because the script uses caching to avoid hitting the AWS api all the time. The related docs are here:

http://docs.ansible.com/ansible/intro_dynamic_inventory.html#example-aws-ec2-external-inventory-script

Though I'm sure you've read through those several times already, the bit I believe affects your specific situation is right at the bottom of that section:

Note that the AWS inventory script will cache results to avoid repeated API calls, and this cache setting is configurable in ec2.ini. To explicitly clear the cache, you can run the ec2.py script with the --refresh-cache parameter:

# ./ec2.py --refresh-cache

Have you tried adding the `--refresh-cache` option to your first run of the ec2.py script? If that works, then bully! but if it doesn't, would you mind posting the related results from the first run of your playbook with the error and then also the related portions of the second one where it works?

Thanks,

Alex

[Nirav Radia]

Hi Alex,

1) I am not sure I understood "placing the availability zone in the region field instead of the region" correctly. Because when I exported "us-west-2a" as AWS_DEFAULT_REGION (which my ec2.py is using to filter region), it gave me error like this:

ERROR! The file inventory/ec2.py is marked as executable, but failed to execute correctly. If this is not supposed to be an executable script, correct this with `chmod -x inventory/ec2.py`.

Inventory script (inventory/ec2.py) had an execution error: region name: us-west-2a likely not supported, or AWS is down.  connection to region failed.

inventory/ec2.py:3: Error parsing host definition ''''': No closing quotation

Let me know where can I place AZ instead of region.

2) Yes, I tried executing 

./inventory/ec2.py --refresh-cache

between instance creation playbook and ping command, but with no luck ! And IMHO it doesn't matter to refresh cache as I have "cache_max_age = 0" in ec2.ini. (Correct me if I am wrong.)

For the first time when I ping doesn't succeed, verbose and console output is:

[Private_IP1 and Private_IP2 are the instances already running and matching the filter in ec2.py, Private_IP3 is the IP of the instance just launched and waited for port 22 successfully]

Using /etc/ansible/ansible.cfg as config file

Loaded callback minimal of type stdout, v2.0

<Private_IP1> ESTABLISH SSH CONNECTION FOR USER: centos

<Private_IP3> ESTABLISH SSH CONNECTION FOR USER: centos

<Private_IP3> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)

<Private_IP3> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)

<Private_IP1> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)

<Private_IP1> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)

<Private_IP1> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile="/home/ubuntu/.ssh/MyKey.pem")

<Private_IP3> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile="/home/ubuntu/.ssh/MyKey.pem")

<Private_IP3> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)

<Private_IP3> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=centos)

<Private_IP1> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)

<Private_IP1> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=centos)

<Private_IP1> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)

<Private_IP3> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)

<Private_IP3> SSH: PlayContext set ssh_common_args: ()

<Private_IP3> SSH: PlayContext set ssh_extra_args: ()

<Private_IP1> SSH: PlayContext set ssh_common_args: ()

<Private_IP1> SSH: PlayContext set ssh_extra_args: ()

<Private_IP1> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r)

<Private_IP3> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r)

<Private_IP3> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/ubuntu/.ssh/MyKey.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=centos -o ConnectTimeout=10 -o ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r Private_IP3 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1474528176.32-147953530428586 `" && echo ansible-tmp-1474528176.32-147953530428586="` echo $HOME/.ansible/tmp/ansible-tmp-1474528176.32-147953530428586 `" ) && sleep 0'"'"''

<Private_IP1> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/ubuntu/.ssh/MyKey.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=centos -o ConnectTimeout=10 -o ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r Private_IP1 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1474528176.32-242806970585719 `" && echo ansible-tmp-1474528176.32-242806970585719="` echo $HOME/.ansible/tmp/ansible-tmp-1474528176.32-242806970585719 `" ) && sleep 0'"'"''

<Private_IP2> ESTABLISH SSH CONNECTION FOR USER: centos

<Private_IP2> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)

<Private_IP2> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)

<Private_IP2> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile="/home/ubuntu/.ssh/MyKey.pem")

<Private_IP2> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)

<Private_IP2> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=centos)

<Private_IP2> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)

<Private_IP2> SSH: PlayContext set ssh_common_args: ()

<Private_IP2> SSH: PlayContext set ssh_extra_args: ()

<Private_IP2> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r)

<Private_IP2> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/ubuntu/.ssh/MyKey.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=centos -o ConnectTimeout=10 -o ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r Private_IP2 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1474528176.32-96061508357666 `" && echo ansible-tmp-1474528176.32-96061508357666="` echo $HOME/.ansible/tmp/ansible-tmp-1474528176.32-96061508357666 `" ) && sleep 0'"'"''

<Private_IP1> PUT /tmp/tmplSwpxr TO /home/centos/.ansible/tmp/ansible-tmp-1474528176.32-242806970585719/ping

<Private_IP1> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)

<Private_IP1> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)

<Private_IP1> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile="/home/ubuntu/.ssh/MyKey.pem")

<Private_IP1> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)

<Private_IP1> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=centos)

<Private_IP1> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)

<Private_IP1> SSH: PlayContext set ssh_common_args: ()

<Private_IP1> SSH: PlayContext set sftp_extra_args: ()

<Private_IP1> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r)

<Private_IP1> SSH: EXEC sftp -b - -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/ubuntu/.ssh/MyKey.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=centos -o ConnectTimeout=10 -o ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r '[Private_IP1]'

<Private_IP2> PUT /tmp/tmptkiK31 TO /home/centos/.ansible/tmp/ansible-tmp-1474528176.32-96061508357666/ping

<Private_IP2> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)

<Private_IP2> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)

<Private_IP2> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile="/home/ubuntu/.ssh/MyKey.pem")

<Private_IP2> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)

<Private_IP2> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=centos)

<Private_IP2> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)

<Private_IP2> SSH: PlayContext set ssh_common_args: ()

<Private_IP2> SSH: PlayContext set sftp_extra_args: ()

<Private_IP2> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r)

<Private_IP2> SSH: EXEC sftp -b - -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/ubuntu/.ssh/MyKey.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=centos -o ConnectTimeout=10 -o ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r '[Private_IP2]'

<Private_IP1> ESTABLISH SSH CONNECTION FOR USER: centos

<Private_IP1> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)

<Private_IP1> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)

<Private_IP1> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile="/home/ubuntu/.ssh/MyKey.pem")

<Private_IP1> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)

<Private_IP1> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=centos)

<Private_IP1> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)

<Private_IP1> SSH: PlayContext set ssh_common_args: ()

<Private_IP1> SSH: PlayContext set ssh_extra_args: ()

<Private_IP1> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r)

<Private_IP1> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/ubuntu/.ssh/MyKey.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=centos -o ConnectTimeout=10 -o ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r -tt Private_IP1 '/bin/sh -c '"'"'LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python /home/centos/.ansible/tmp/ansible-tmp-1474528176.32-242806970585719/ping; rm -rf "/home/centos/.ansible/tmp/ansible-tmp-1474528176.32-242806970585719/" > /dev/null 2>&1 && sleep 0'"'"''

<Private_IP2> ESTABLISH SSH CONNECTION FOR USER: centos

<Private_IP2> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)

<Private_IP2> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)

<Private_IP2> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile="/home/ubuntu/.ssh/MyKey.pem")

<Private_IP2> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)

<Private_IP2> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=centos)

<Private_IP2> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)

<Private_IP2> SSH: PlayContext set ssh_common_args: ()

<Private_IP2> SSH: PlayContext set ssh_extra_args: ()

<Private_IP2> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r)

<Private_IP2> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/ubuntu/.ssh/MyKey.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=centos -o ConnectTimeout=10 -o ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r -tt Private_IP2 '/bin/sh -c '"'"'LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python /home/centos/.ansible/tmp/ansible-tmp-1474528176.32-96061508357666/ping; rm -rf "/home/centos/.ansible/tmp/ansible-tmp-1474528176.32-96061508357666/" > /dev/null 2>&1 && sleep 0'"'"''

Private_IP1 | SUCCESS => {

    "changed": false,

    "invocation": {

        "module_args": {

            "data": null

        },

        "module_name": "ping"

    },

    "ping": "pong"

}

Private_IP2 | SUCCESS => {

    "changed": false,

    "invocation": {

        "module_args": {

            "data": null

        },

        "module_name": "ping"

    },

    "ping": "pong"

}

Private_IP3 | UNREACHABLE! => {

    "changed": false,

    "msg": "Failed to connect to the host via ssh.",

    "unreachable": true

}

On the second time (after half a min max) when it succeeds, the logs are: 

Using /etc/ansible/ansible.cfg as config file

Loaded callback minimal of type stdout, v2.0

<Private_IP3> ESTABLISH SSH CONNECTION FOR USER: centos

<Private_IP3> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)

<Private_IP1> ESTABLISH SSH CONNECTION FOR USER: centos

<Private_IP1> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)

<Private_IP3> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)

<Private_IP3> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile="/home/ubuntu/.ssh/MyKey.pem")

<Private_IP1> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)

<Private_IP1> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile="/home/ubuntu/.ssh/MyKey.pem")

<Private_IP1> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)

<Private_IP3> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)

<Private_IP3> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=centos)

<Private_IP3> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)

<Private_IP1> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=centos)

<Private_IP1> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)

<Private_IP1> SSH: PlayContext set ssh_common_args: ()

<Private_IP3> SSH: PlayContext set ssh_common_args: ()

<Private_IP3> SSH: PlayContext set ssh_extra_args: ()

<Private_IP1> SSH: PlayContext set ssh_extra_args: ()

<Private_IP1> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r)

<Private_IP3> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r)

<Private_IP1> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/ubuntu/.ssh/MyKey.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=centos -o ConnectTimeout=10 -o ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r Private_IP1 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1474528970.94-42038548997439 `" && echo ansible-tmp-1474528970.94-42038548997439="` echo $HOME/.ansible/tmp/ansible-tmp-1474528970.94-42038548997439 `" ) && sleep 0'"'"''

<Private_IP3> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/ubuntu/.ssh/MyKey.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=centos -o ConnectTimeout=10 -o ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r Private_IP3 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1474528970.94-11790883317121 `" && echo ansible-tmp-1474528970.94-11790883317121="` echo $HOME/.ansible/tmp/ansible-tmp-1474528970.94-11790883317121 `" ) && sleep 0'"'"''

<Private_IP2> ESTABLISH SSH CONNECTION FOR USER: centos

<Private_IP2> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)

<Private_IP2> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)

<Private_IP2> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile="/home/ubuntu/.ssh/MyKey.pem")

<Private_IP2> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)

<Private_IP2> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=centos)

<Private_IP2> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)

<Private_IP2> SSH: PlayContext set ssh_common_args: ()

<Private_IP2> SSH: PlayContext set ssh_extra_args: ()

<Private_IP2> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r)

<Private_IP2> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/ubuntu/.ssh/MyKey.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=centos -o ConnectTimeout=10 -o ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r Private_IP2 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1474528970.94-115245318606115 `" && echo ansible-tmp-1474528970.94-115245318606115="` echo $HOME/.ansible/tmp/ansible-tmp-1474528970.94-115245318606115 `" ) && sleep 0'"'"''

<Private_IP1> PUT /tmp/tmpIzmsRg TO /home/centos/.ansible/tmp/ansible-tmp-1474528970.94-42038548997439/ping

<Private_IP1> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)

<Private_IP1> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)

<Private_IP1> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile="/home/ubuntu/.ssh/MyKey.pem")

<Private_IP1> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)

<Private_IP1> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=centos)

<Private_IP1> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)

<Private_IP1> SSH: PlayContext set ssh_common_args: ()

<Private_IP1> SSH: PlayContext set sftp_extra_args: ()

<Private_IP1> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r)

<Private_IP1> SSH: EXEC sftp -b - -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/ubuntu/.ssh/MyKey.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=centos -o ConnectTimeout=10 -o ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r '[Private_IP1]'

<Private_IP2> PUT /tmp/tmppBD_UL TO /home/centos/.ansible/tmp/ansible-tmp-1474528970.94-115245318606115/ping

<Private_IP2> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)

<Private_IP2> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)

<Private_IP2> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile="/home/ubuntu/.ssh/MyKey.pem")

<Private_IP2> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)

<Private_IP2> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=centos)

<Private_IP2> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)

<Private_IP2> SSH: PlayContext set ssh_common_args: ()

<Private_IP2> SSH: PlayContext set sftp_extra_args: ()

<Private_IP2> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r)

<Private_IP2> SSH: EXEC sftp -b - -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/ubuntu/.ssh/MyKey.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=centos -o ConnectTimeout=10 -o ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r '[Private_IP2]'

<Private_IP1> ESTABLISH SSH CONNECTION FOR USER: centos

<Private_IP1> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)

<Private_IP1> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)

<Private_IP1> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile="/home/ubuntu/.ssh/MyKey.pem")

<Private_IP1> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)

<Private_IP1> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=centos)

<Private_IP1> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)

<Private_IP1> SSH: PlayContext set ssh_common_args: ()

<Private_IP1> SSH: PlayContext set ssh_extra_args: ()

<Private_IP1> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r)

<Private_IP1> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/ubuntu/.ssh/MyKey.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=centos -o ConnectTimeout=10 -o ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r -tt Private_IP1 '/bin/sh -c '"'"'LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python /home/centos/.ansible/tmp/ansible-tmp-1474528970.94-42038548997439/ping; rm -rf "/home/centos/.ansible/tmp/ansible-tmp-1474528970.94-42038548997439/" > /dev/null 2>&1 && sleep 0'"'"''

<Private_IP2> ESTABLISH SSH CONNECTION FOR USER: centos

<Private_IP2> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)

<Private_IP2> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)

<Private_IP2> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile="/home/ubuntu/.ssh/MyKey.pem")

<Private_IP2> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)

<Private_IP2> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=centos)

<Private_IP2> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)

<Private_IP2> SSH: PlayContext set ssh_common_args: ()

<Private_IP2> SSH: PlayContext set ssh_extra_args: ()

<Private_IP2> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r)

<Private_IP2> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/ubuntu/.ssh/MyKey.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=centos -o ConnectTimeout=10 -o ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r -tt Private_IP2 '/bin/sh -c '"'"'LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python /home/centos/.ansible/tmp/ansible-tmp-1474528970.94-115245318606115/ping; rm -rf "/home/centos/.ansible/tmp/ansible-tmp-1474528970.94-115245318606115/" > /dev/null 2>&1 && sleep 0'"'"''

Private_IP1 | SUCCESS => {

    "changed": false,

    "invocation": {

        "module_args": {

            "data": null

        },

        "module_name": "ping"

    },

    "ping": "pong"

}

Private_IP2 | SUCCESS => {

    "changed": false,

    "invocation": {

        "module_args": {

            "data": null

        },

        "module_name": "ping"

    },

    "ping": "pong"

}

<Private_IP3> PUT /tmp/tmpQBmgof TO /home/centos/.ansible/tmp/ansible-tmp-1474528970.94-11790883317121/ping

<Private_IP3> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)

<Private_IP3> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)

<Private_IP3> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile="/home/ubuntu/.ssh/MyKey.pem")

<Private_IP3> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)

<Private_IP3> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=centos)

<Private_IP3> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)

<Private_IP3> SSH: PlayContext set ssh_common_args: ()

<Private_IP3> SSH: PlayContext set sftp_extra_args: ()

<Private_IP3> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r)

<Private_IP3> SSH: EXEC sftp -b - -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/ubuntu/.ssh/MyKey.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=centos -o ConnectTimeout=10 -o ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r '[Private_IP3]'

<Private_IP3> ESTABLISH SSH CONNECTION FOR USER: centos

<Private_IP3> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)

<Private_IP3> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)

<Private_IP3> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile="/home/ubuntu/.ssh/MyKey.pem")

<Private_IP3> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)

<Private_IP3> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=centos)

<Private_IP3> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)

<Private_IP3> SSH: PlayContext set ssh_common_args: ()

<Private_IP3> SSH: PlayContext set ssh_extra_args: ()

<Private_IP3> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r)

<Private_IP3> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/ubuntu/.ssh/MyKey.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=centos -o ConnectTimeout=10 -o ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r -tt Private_IP3 '/bin/sh -c '"'"'LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python /home/centos/.ansible/tmp/ansible-tmp-1474528970.94-11790883317121/ping; rm -rf "/home/centos/.ansible/tmp/ansible-tmp-1474528970.94-11790883317121/" > /dev/null 2>&1 && sleep 0'"'"''

Private_IP3 | SUCCESS => {

    "changed": false,

    "invocation": {

        "module_args": {

            "data": null

        },

        "module_name": "ping"

    },

    "ping": "pong"

}

Let me know if you find something.

Thanks again,

Nirav

[Alexander H. Laughlin]

Hi Nirav,

Sorry about the lack of clarity in my suggestion. I was referring to your ec2.ini, specifically:

# AWS regions to make calls to. Set this to 'all' to make request to all regions
# in AWS and merge the results together. Alternatively, set this to a comma
# separated list of regions. E.g. 'us-east-1,us-west-1,us-west-2'
regions = all
regions_exclude = us-gov-west-1,cn-north-1

would become:

# AWS regions to make calls to. Set this to 'all' to make request to all regions
# in AWS and merge the results together. Alternatively, set this to a comma
# separated list of regions. E.g. 'us-east-1,us-west-1,us-west-2'

regions = us-west-2a
regions_exclude = us-gov-west-1,cn-north-1

If you were using that particular availability zone. I'd be surprised if that worked, however, what happens when you unset the environment variable and set the region in the ec2.ini file? 

Alex

[Nirav Radia]

Hi Alex,

Replacing availability zone in ec2.ini also gave me the same error:

ERROR! The file inventory/ec2.py is marked as executable, but failed to execute correctly. If this is not supposed to be an executable script, correct this with `chmod -x inventory/ec2.py`.

Inventory script (inventory/ec2.py) had an execution error: region name: us-west-2a likely not supported, or AWS is down.  connection to region failed.

inventory/ec2.py:3: Error parsing host definition ''''': No closing quotation

Unsetting the environment variable and setting region in ec2.ini does not help and gives the same Unreachable error. :(

Thanks,

Nirav

[Alexander H. Laughlin]

Hi Nirav,

This is a tough nut to crack. Which version of Ansible are you using?

Which region is it specifically that is failing? Or is it just any region that is specified over 'any' region?

Alex

...

[Nirav Radia]

Hi Alex

Sorry for late reply.

I am using Ansible 2.1.1.0.

It happens for any available region over AWS.

One more finding:

* If I remove port in wait_for, it works ok.

[Nirav Radia]

Hi Alex / everyone,

Another finding:

* This also works fine:

wait_for: host={{ item.public_dns_name }} port=22 delay=60 timeout=320 state=started

I want to understand why 

wait_for: host={{ item.public_dns_name }} port=22

does not work when I query for a particular region? Does it return before port 22 is actually open or before machine is actually SSH connect-able?

Just FYI, I am using a custom AMI built on top of Official CentOS 7 AMI (If this helps or if it is an OS specific issue).

Thanks,

Nirav

[Nirav Radia]

Hi,

While browsing the integration tests of Ansible modules, I cam across this:

https://github.com/ansible/ansible/blob/7f8e8ddca9cf103c05bf39d68ebb2e2ded4067f2/test/utils/ansible-playbook_integration_runner/ec2.yml

It has a task named "Wait a little longer for centos". So anyone know why is it there? My target hosts run CentOS also, if this is the issue for the behavior mentioned in my question.

Thanks,

Nirav

[Nirav Radia]

I could not find the real reason why the SSH on EC2 instance running CentOS was not able to connect. The workaround I applied was to write a module which try to connect host over SSH for given number of times and return success when it is successful so that further plays won't fail because of unreachable error.

wait_for_SSH.py

#!/usr/bin/python

'''

module: wait_for_SSH

short_description: Waits for a host to be SSH connectable.

description:

     - Tries to Ansible ping(not ICMP ping) to host as per the passed parameters

options:

  host:

    description:

      - A resolvable hostname or IP address to ansible ping

    required: true

  retries:

    description:

      - maximum number of times to retry

    required: false

    default: 10

  delay:

    description:

      - number of seconds to wait between two consecutive pings

    required: false

    default: 5

'''

from ansible.module_utils.basic import *

from subprocess import call

import time

def validate_params(module, retries, delay):

  if retries < 0:

    module.fail_json(msg="retries should be greater than 0")

  if delay < 0:

    module.fail_json(msg="delay should be greater than 0")   

  return

def main():

    fields = {

      "host": {"required": True, "type": "str"},

      "retries": {"required": False, "type": "int", "default": 10},

      "delay": {"required": False, "type": "int", "default": 5}

    }

    module = AnsibleModule(argument_spec=fields)

    host = module.params['host']

    retries = module.params['retries']

    delay = module.params['delay']

    count = 0

    output = 1

    validate_params(module, retries, delay)

    while (count < retries) and (output != 0):

      if delay:

        time.sleep(delay)

      output = call(["ansible", "all", "-i", ","+host, "-m", "ping"])

      count += 1

    response = {"output" : output}

    module.exit_json(changed=False, output=response)

    

if __name__ == '__main__':

    main()

And executed it after wait_for port 22.

...  
  # Wait only for running instances because 'ec2_server' might contain terminated instances to fulfil exact_count condition

  - name: wait for ssh server to be running

    wait_for: host={{ item.public_dns_name }} port=22 search_regex=OpenSSH

    with_items: "{{ec2_server.instances | default([])}}"

    when: item.state == 'running'

  

  # wait_for_SSH is our custom module which tries to Ansible ping on created instances until it is successful as per retries specified(bug AD-3)

  - name: Ensure SSH is running

    wait_for_SSH:

      host: "{{item.private_ip}}"

    register: moduleoutput

    with_items: "{{ec2_server.instances | default([])}}"

    when: item.state == 'running'
..<<further plays>>...

Thanks,

Nirav