very very simple test with ansible.builtin.uri

[dmastrop]

hi all

I am running a very very simple test with the ansible.builtin.uri and it is failing.

This is in a development setup and the terraform infra and the apps are all made to be up and running before this simple test.

I verify that the apps are running on EC2 with a browser hit to each of the ip addresses (I know is its redirecting because url changes in browser)

There are 2 EC2 nodes up each running an ansible installed Granfana and prometheus app on a terraform infra.

I also verified from my IDE (Cloud9 instance) that a wget on each of the Urls is returning the 302.

The access rules on the EC2 nodes are pretty much open

When I run the ansible playbook I am seeing these errors (I replaced public IPs with XXXX and YYYY)

I am running the ansible playbook from the same Cloud9 IDE where I tested the wget that works with a returned 302.

The url looks to be correctly formed.  The correct ip addresses are pulled from inventory

My simple playbook that I wanted to append to the main playbook is below.

I am not getting any syntax errors.

warm regards

Dave

==========

dave:~/environment/mtc-terraform-ansible-jenkins (Jenkins_development) $ ansible-playbook -i aws_hosts --key-file /home/ubuntu/.ssh/mtckey playbooks/node-test.yml --limit @/home/ubuntu/environment/mtc-terraform-ansible-jenkins/.ansible-retry/node-test.retry

PLAY [Test for Granana and Prometheus access] ***************************************************************************************************

TASK [Grafana] **********************************************************************************************************************************
fatal: [XXXX]: FAILED! => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python3"}, "changed": false, "content": "", "elapsed": 30, "msg": "Status code was -1 and not [302]: Request failed: <urlopen error timed out>", "redirected": false, "status": -1, "url": "http://XXXX:3000"}
fatal: [YYYY]: FAILED! => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python3"}, "changed": false, "content": "", "elapsed": 30, "msg": "Status code was -1 and not [302]: Request failed: <urlopen error timed out>", "redirected": false, "status": -1, "url": "http://YYYY:3000"}
        to retry, use: --limit @/home/ubuntu/environment/mtc-terraform-ansible-jenkins/.ansible-retry/node-test.retry

PLAY RECAP **************************************************************************************************************************************
XXXX               : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0  
YYYY             : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

=======

playbook for this test:

---
- name: Test for Granana and Prometheus access

  hosts: main
  gather_facts: no
  tasks:
  # https://docs.ansible.com/ansible/latest/collections/ansible/builtin/uri_module.html
  - name: Grafana
    ansible.builtin.uri:
      url: http://{{ inventory_hostname }}:3000
      follow_redirects: none
      status_code: [302]
     
  - name: Prometheus
    ansible.builtin.uri:
      url: http://{{ inventory_hostname }}:9090
      follow_redirects: none
      status_code: [302]

[Dick Visser]

I see a mix of terraform, "ip addresses", browsers, EC2 nodes, "apps", Granfana, Granana, prometheus, Cloud9, "pretty much open" access rules, and a little Jenkins sprinkled on top as well.

From your story, I get that you are to be able to access publicly accessible URLs from the internet.

And the ansible playbook fails. The uri tasks appear to be running on the host that runs the service.

So you cannot access the URLs from the server itself, a timeout means that there is something blocking the HTTP request and/or response.

That can be caused by any of the mentioned infrastructure bits...

Essentially, the uri module just reports that it can't reach a URL from the host that you're targeting, which isn't an ansible problem, but rather a network/filtering/etc problem. 

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/afa91f36-4e80-43b6-9610-136eb002a25en%40googlegroups.com.

[dave.ma...@gmail.com]

sorry one correction in the above

The rule that worked is source ip= public ip address of the service node (looking at the tcpdump you can understand why; pubic ip is the 18 address)

The EC2 node has one virtual interface and that is the private ip address so the source = private ip address is not needed.

Only the source ip=public ip address is needed.

Edit is below in red:

"Yet with the ansible traffic (being generated from my Cloud9 IDE where Jenkins is installed) in the tcpdump I see the traffic from ansible.builtin.uri coming in to the private ip address and I assume the ubuntu internally loopbacks that to the public ip address (?). I don't really know, but I do know that this access list resolved the issue of ansible.builtin.uri being sent to the public ip address of the service nodes:

Source = public ip address of the service node

Allow

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CAF8BbLZs7T5KgmsPM2Ex7TH4pQAUPocWpXLhFceimnPwsB4-Sg%40mail.gmail.com.

[dave.ma...@gmail.com]

Here is the complete message 

warm regards

======

hi Dick

one correction

" The uri tasks appear to be running on the host that runs the service."

The uri tasks are running from Jenkins which is co- located on the Cloud9 IDE EC2 instance.   

There are 2 separate EC2 service nodes (separate from the Cloud9 IDE EC2 instance) that are part of the terraform/ansible jenkins deployment.

So there is a total of 3 EC2 instances with Jenkins running from Cloud9 IDE EC2 instance.

• The wget from the terminal on Cloud9 to the public ip of the service nodes works fine and the traffic from PC to public ip service nodes works fine.
• There are access lists on amazon security group for the service nodes for both of these traffic flows and they work great.
• I would have assumed that ansible running on the Jenkins Cloud9 IDE would also work as well (ansible is sending traffic to the public ip as well based upon  url: http://{{ inventory_hostname }} and the inventory_hostname list of ip addresses are the public ip addresses and the url looks to be constructed properly in the Jenkins logs )

I ended up running a tcpdump on the EC2 nodes and saw something that was very odd  and added the relevant access list to the security group on amazon for the nodes and it is working now.   I had originally opened up everything as a test with the ansible uri traffic, and that worked, so I knew that it was an access list issue. 

But why with just the ansible.builtin.uri? the access lists that I had to create based upon the tcpdump do not really make sense to me at all and it is only necessary with the ansible generated request.

• As stated in original post a wget (also tried curl as well) from the Cloud9 IDE works fine and hitting the url from the browser of the PC works fine.  Perhaps there is a default setting in the ansible.builtin.uri that is causing this (something to do with proxy address embedded  header?)
• A tcpdump on the EC2 service node for PC to EC2 service node traffic shows the original ip address of my PC as source and the destination as the public ip address of the node. (this makes perfect sense).  
• I had an access list rule to allow this originally so this explains why this worked fine.

Yet with the ansible traffic (being generated from my Cloud9 IDE where Jenkins is installed) in the tcpdump I see the traffic from ansible.builtin.uri coming in to the private ip address and I assume the ubuntu internally loopbacks that to the public ip address (?). I don't really know, but I do know that this access list resolved the issue of ansible.builtin.uri being sent to the public ip address of the service nodes:

Source = public ip address of the service node

Allow

Why the traffic internally on amazon is being routed to the private ip address is a mystery.

I don't have a tcpdump decoder on the ubuntu EC2 instance or Cloud9 IDE so I can't look at the headers in the ansible request, but perhaps it is using an embedded proxy ip of the private ip address in the request and amazon internally routes the request as such when it comes in. I don't know.

I did run curl as well from the Cloud9 IDE and that worked as expected and did not exhibit this strange behavior.

It is working now but not sure why the traffic is routed as such for ansible.builtin.uri traffic only.

The tcpdump is below

(sorry for long response but this is a strange issue)

warm regards

Dave

=====

tcpdump of ansible.builtin.uri traffice (Cloud9 IDE) to service node public ip address (the 18 address)

The SYN is from private ip and is loopbacked to the the public IP. I don't see this with the curl, wget, or browser PC to service node traffic. Each of those have src ip as expected.

Note: this service node has been decommissioned. 

19:03:08.473078 IP ip-10-123-1-110.us-west-1.compute.internal.60390 > ec2-18-144-45-220.us-west-1.compute.amazonaws.com.3000: Flags [S], seq 1020145439, win 62727, options [mss 8961,sackOK,TS val 3617442296 ecr 0,nop,wscale 7], length 0

19:03:08.473291 IP ec2-18-144-45-220.us-west-1.compute.amazonaws.com.60390 > ip-10-123-1-110.us-west-1.compute.internal.3000: Flags [S], seq 1020145439, win 62727, options [mss 1460,sackOK,TS val 3617442296 ecr 0,nop,wscale 7], length 0

19:03:08.473315 IP ip-10-123-1-110.us-west-1.compute.internal.3000 > ec2-18-144-45-220.us-west-1.compute.amazonaws.com.60390: Flags [S.], seq 864838021, ack 1020145440, win 62643, options [mss 8961,sackOK,TS val 3617442297 ecr 3617442296,nop,wscale 7], length 0

19:03:08.473497 IP ec2-18-144-45-220.us-west-1.compute.amazonaws.com.3000 > ip-10-123-1-110.us-west-1.compute.internal.60390: Flags [S.], seq 864838021, ack 1020145440, win 62643, options [mss 1460,sackOK,TS val 3617442297 ecr 3617442296,nop,wscale 7], length 0

19:03:08.473514 IP ip-10-123-1-110.us-west-1.compute.internal.60390 > ec2-18-144-45-220.us-west-1.compute.amazonaws.com.3000: Flags [.], ack 1, win 491, options [nop,nop,TS val 3617442297 ecr 3617442297], length 0

19:03:08.473641 IP ip-10-123-1-110.us-west-1.compute.internal.60390 > ec2-18-144-45-220.us-west-1.compute.amazonaws.com.3000: Flags [P.], seq 1:120, ack 1, win 491, options [nop,nop,TS val 3617442297 ecr 3617442297], length 119

19:03:08.473769 IP ec2-18-144-45-220.us-west-1.compute.amazonaws.com.60390 > ip-10-123-1-110.us-west-1.compute.internal.3000: Flags [.], ack 1, win 491, options [nop,nop,TS val 3617442297 ecr 3617442297], length 0

19:03:08.473844 IP ec2-18-144-45-220.us-west-1.compute.amazonaws.com.60390 > ip-10-123-1-110.us-west-1.compute.internal.3000: Flags [P.], seq 1:120, ack 1, win 491, options [nop,nop,TS val 3617442297 ecr 3617442297], length 119

19:03:08.473852 IP ip-10-123-1-110.us-west-1.compute.internal.3000 > ec2-18-144-45-220.us-west-1.compute.amazonaws.com.60390: Flags [.], ack 120, win 489, options [nop,nop,TS val 3617442297 ecr 3617442297], length 0

19:03:08.474021 IP ec2-18-144-45-220.us-west-1.compute.amazonaws.com.3000 > ip-10-123-1-110.us-west-1.compute.internal.60390: Flags [.], ack 120, win 489, options [nop,nop,TS val 3617442297 ecr 3617442297], length 0

19:03:08.474769 IP ip-10-123-1-110.us-west-1.compute.internal.3000 > ec2-18-144-45-220.us-west-1.compute.amazonaws.com.60390: Flags [P.], seq 1:300, ack 120, win 489, options [nop,nop,TS val 3617442298 ecr 3617442297], length 299

19:03:08.474832 IP ip-10-123-1-110.us-west-1.compute.internal.3000 > ec2-18-144-45-220.us-west-1.compute.amazonaws.com.60390: Flags [F.], seq 300, ack 120, win 489, options [nop,nop,TS val 3617442298 ecr 3617442297], length 0

19:03:08.474989 IP ec2-18-144-45-220.us-west-1.compute.amazonaws.com.3000 > ip-10-123-1-110.us-west-1.compute.internal.60390: Flags [P.], seq 1:300, ack 120, win 489, options [nop,nop,TS val 3617442298 ecr 3617442297], length 299

19:03:08.474996 IP ip-10-123-1-110.us-west-1.compute.internal.60390 > ec2-18-144-45-220.us-west-1.compute.amazonaws.com.3000: Flags [.], ack 300, win 489, options [nop,nop,TS val 3617442298 ecr 3617442298], length 0

19:03:08.475038 IP ec2-18-144-45-220.us-west-1.compute.amazonaws.com.3000 > ip-10-123-1-110.us-west-1.compute.internal.60390: Flags [F.], seq 300, ack 120, win 489, options [nop,nop,TS val 3617442298 ecr 3617442297], length 0

19:03:08.475179 IP ec2-18-144-45-220.us-west-1.compute.amazonaws.com.60390 > ip-10-123-1-110.us-west-1.compute.internal.3000: Flags [.], ack 300, win 489, options [nop,nop,TS val 3617442298 ecr 3617442298], length 0

19:03:08.476038 IP ip-10-123-1-110.us-west-1.compute.internal.60390 > ec2-18-144-45-220.us-west-1.compute.amazonaws.com.3000: Flags [F.], seq 120, ack 301, win 489, options [nop,nop,TS val 3617442299 ecr 3617442298], length 0

19:03:08.476204 IP ec2-18-144-45-220.us-west-1.compute.amazonaws.com.60390 > ip-10-123-1-110.us-west-1.compute.internal.3000: Flags [F.], seq 120, ack 301, win 489, options [nop,nop,TS val 3617442299 ecr 3617442298], length 0

19:03:08.476216 IP ip-10-123-1-110.us-west-1.compute.internal.3000 > ec2-18-144-45-220.us-west-1.compute.amazonaws.com.60390: Flags [.], ack 121, win 489, options [nop,nop,TS val 3617442299 ecr 3617442299], length 0

19:03:08.476430 IP ec2-18-144-45-220.us-west-1.compute.amazonaws.com.3000 > ip-10-123-1-110.us-west-1.compute.internal.60390: Flags [.], ack 121, win 489, options [nop,nop,TS val 3617442299 ecr 3617442299], length 0

[dmastrop]

hi Dick

one correction

" The uri tasks appear to be running on the host that runs the service."

The uri tasks are running from Jenkins which is co- located on the Cloud9 IDE EC2 instance.   

There are 2 separate EC2 service nodes (separate from the Cloud9 IDE EC2 instance) that are part of the terraform/ansible jenkins deployment.

So there is a total of 3 EC2 instances with Jenkins running from Cloud9 IDE EC2 instance.

This is with the edit (in red below):