Unable to connect to a specific subnet from AWX tower plus DNS resolution not working.

[kunal kalyanpur]

Hi,

We recently had an AWX Operator deployed in Kubernetes Cluster. Please find the details below:

AWX Version installed: 21.12

AWX operator version: 1.2.0

Ansible version: 2.9.14

However, I have run into a strange problem. Hosts in specific subnets are not pingable whereas other subnets ping. I am unable to connect to my LDAP server as it resides in the subnet that's inaccessible. Could you kindly let me know what steps I need to follow in order to overcome this issue.  I am new to Kubernetes. I could see a whole lot of iptables rules created post deployment, but I am hesitant to touch them as that could break the cluster communication.

Secondly DNS name resolution doesn't work. If I provide hostnames in Host Inventory of AWX it simply fails to connect however with ip address, it works. Once again for specific subnets only. 

I am unable to figure out which parameters I need to add in order for DNS resolution to work. 

This is what I see currently in my awx-web container:

#kubectl exec -it awx-5bb7bdb785-zn6pw -n awx -c awx-web -- bash
bash-5.1$ cat /etc/resolv.conf
search awx.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.43.0.10
options ndots:5

Looking forward to your guidance and inputs to resolve these 2 outstanding issues. I need to get the testing done before I demo this to my management. 

Rgds,

Kunal

[kunal kalyanpur]

Can someone please guide here!!

Rgds,

Kunal

[Rilindo Foster]

Hi Kunal,

 

Sounds like the issue in question is an issue with Kubernetes networking in conflict with existing host or on-premise networks. I would suggest reaching out to a forum specific to Kubernetes (Slack, mailing list, etc) to see if that is case or something else.

 

• Rilindo

--
You received this message because you are subscribed to the Google Groups "AWX Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to awx-project...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/cfddf210-36d2-4f5f-b66b-5725651ac454n%40googlegroups.com.

[Michael Kelly]

Hi Kunal,

this looks very like a problem that I have, can you please answer the following.

Are you able to connect to those subnets from the container's host?

If you are not then the problem is not related to AWX/Kubernetes.

If you can, can you connect using the LDAP server's IP address?

Regards,

Michael.

To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/MN2PR12MB3151B555E618C4F3948D6C7FF1B99%40MN2PR12MB3151.namprd12.prod.outlook.com.

[kunal kalyanpur]

Hi Michael,

Thanks for reaching out firstly!! Please find my replies in line:

Are you able to connect to those subnets from the container's host?

There is no ping command on the awx-web container firstly. When I login to the container I am logged in as an awx user. I cannot seem to sudo root as it prompts for the password and I don't know what that password is.

I ran a curl https://www.google.com and it connects

I ran a curl https://github.com and it connects

From the container host, I am able to ping all subnets just fine. However from AWX Web UI when I run a run a ping playbook to the host ip's they fail

If you are not then the problem is not related to AWX/Kubernetes.

I am unsure where the problem lies as out of 10 subnets, only 2 subnets are accessible. Rest 8 subnets are inaccessible from AWX web UI

If you can, can you connect using the LDAP server's IP address?

From the container, I ran the following command: echo | openssl s_client -connect <LDAP_IP_ADDRESS>:636  and it shows me the certificate contents

Since name resolution is not working from AWX Web UI, I tried filling in LDAP settings details with ip address. However from container logs it shows cannot to LDAP server

=================================================================

2023-03-13 13:39:22,536 WARNING  [756ffb56079c479996959a581bb46e93] django_auth_ldap Caught LDAPError while authenticating ksureshkalyanpur: SERVER_DOWN({'result': -1, 'desc': "Can't contact LDAP server", 'ctrls': [], 'info': 'error:0A000086:SSL routines::certificate verify failed (CA signature digest algorithm too weak)'})

=================================================================

When I try pinging the LDAP ip from AWX WEB UI, it fails

Looking forward to any leads or pointers on the above issues.

Rgds,

Kunal

 

You received this message because you are subscribed to a topic in the Google Groups "AWX Project" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/awx-project/1cfbLzTv-uc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to awx-project...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/CADee-3PJ4UZ-kBPV7ZegdA6zRv21E%3Dv6sZ3T8qaRe1qSGY21yQ%40mail.gmail.com.

[Michael Kelly]

Hi Kunal,

is it the case that the fqdns AWX cannot resolve, are in a domain that cannot be resolved from outside your company?

To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/CAA_Lw8TDGHQ9OpEak7e_9-8XxKtWjvGU2XuvYQmOUoPStApOEQ%40mail.gmail.com.

[kunal kalyanpur]

Hi Michael,

The fqdns are in the company domain. The AWX controller resides on a Kuberenetes cluster which is in the company n/w and domain. So it's all internal. From the host, all subnets are accessible however Kubernetes setup has created so many firewall rules that I myself cant figure out what is blocking and what its not. There is no firewalld or selinux running on the host machine. So I am unsure where the problem lies here. 

Is there a core DNS or N/W Config file that Kubernetes creates in yaml form or some form that can be checked? There is nothing under AWX Operator yaml at least.

The service_type selected is nodeport.

Rgds,

Kunal

To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/CADee-3NCrixTmSEa_RAHXuZh4JLfk27%2BiS5FnxKHdPJ%2B_pzRiw%40mail.gmail.com.

[Michael Kelly]

Hi Kunal,

go to https://www.dnsqueries.com/en/dns_lookup.php and see if you can resolve any of the fqdns that are failing.

To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/CAA_Lw8Tqcjq6RcE73GXo0wFK5RX9J2MnpnsRfq6ia%3DaSToV-2Q%40mail.gmail.com.

[kunal kalyanpur]

Thanks for the update Michael. Will check!!

Rgds,

Kunal

To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/CADee-3NHgana3V-P5LMzpJEJ2ZNj4SnYGx264ano7SOk3ZaADQ%40mail.gmail.com.