Open Source version of Automation Mesh?

[JC PiCoDoN]

Hi List!

I have deployed the latest AWX using the Operator on a k8s cluster.

It is simple (for now) with one replica of both Operator and AWX instance, the PG DB is external.

AWX is exposed via https using Traefik (IngressRoute) for the WebUI/API part.

I would like to implement Execution Node(s) on remote site(s) but so far haven't been able to make it work. I have tried (many iterations of):

- creating a new instance via the UI and deploy this package on my remote execution node

- deploy receptor and ansible-runner manually on the exec. node

As Stefan has been discussing in other posts (I have been following his adventures and trying to replicate), I understand we need to expose the receptor outside the k8s cluster so that execution nodes can reach it (ie. create a load-balancer service which maps to the receptor tcp port of the controller node ), but by default the "tcp-listener" part of the receptor config isn't present in the configmap which is used for /etc/receptor/receptor.conf (on awx_ee container I think) so I imagine it isn't listening?

What is the proper way to go?

1. enable receptor in the control node, using an extra configmap or the operator?

2. create a k8s service pointing to the default (27199) or custom (configmap) port for receptor (since receptor traffic isn't on the web/api port)

3. create an install package from the UI for the new exec. node and deploy it (using the listener port configured above)

4. wait for the exec. node to call home and register in the UI (via the exposed k8s service)

5. schedule jobs on the new exec. node

Bonus question: can the exec. node be installed manually (no UI install package) and run fully in containers (receptor and ansible-runner)?

Thank you in advance for any help (or link) that would clarify the steps! :)

[kurokobo]

Hi,

Refer to this documentation: https://github.com/ansible/awx/blob/devel/docs/execution_nodes.md

> I understand we need to expose the receptor outside the k8s cluster so that execution nodes can reach it

Only the connection from awx-ee container to execution node is required, not the other way around.

Therefore, the awx-ee container does not need to have a tcp-listener, nor does it need to expose the port of the awx-ee container to the outside of the cluster.

Ensure the port 27199/tcp (default) on execution node is reachable from awx-ee container, e.g. configure firewalld on execution node to allow inbound connection to 27199.

Perhaps there is a different reason why your execution node is not working properly.

Can you provide screenshots or logs that you've faced? I think the logs of the awx-ee container would be helpful.

--

@kurokobo

------- Original Message -------

--
You received this message because you are subscribed to the Google Groups "AWX Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to awx-project...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/e5188ee0-b745-41f6-b2d2-fa51e94ce5can%40googlegroups.com.

[JC PiCoDoN]

Thanks for the answer! It's good news that I don't need to expose the K8s side :)

We have strict firewall rules and it take ages to open a new port, can I use a custom port that I know is already opened (like 8080) for the exec. node(s)?

If so, do I specify the custom port when building the install package (when adding a new instance in AWX UI)?

Do you know if this method is the only one or if we can deploy receptor manually and have it call AWX to register?

Thank you for the help!!

[kurokobo]

Hi,

> can I use a custom port that I know is already opened (like 8080) for the exec. node(s)?

You can specify custom port for Receptor on execution node when you add new instance through Web UI.

Once you specify the port, the group_vars/all.yaml file in the generated install bundle contains your custom port as variable "receptor_port". This variable will be used to generate configuration file for Receptor on the execution node.

> Do you know if this method is the only one or if we can deploy receptor manually and have it call AWX to register?

Using install bundle is the simplest way since execution node has to have valid certificates for Receptor that generated by AWX. These certificates are generated by AWX, and included in the generated install bundle. The playbook in the install bundle places these certs on the correct path on the execution node, and generate configuration file for Receptor to use these certs, during installation.

Technically you can generate these certs manually by invoking some commands inside the awx-ee container and also you can deploy/configure Receptor on execution node manually, but it's a complicated way.

If you want to deploy them manually, you should know what tasks are contained in the playbook inside the install bundle, and how to generate certs for Receotor,.

Regards,

--

@kurokobo

------- Original Message -------

To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/0651a1e5-3447-4a9f-99a8-4f9386ab8cfbn%40googlegroups.com.

[JC PiCoDoN]

Yeah the bundle sounds like a better option! :)

I tried deploying the bundle with port 8080, it worked but when I look for open ports on the receptor host only tcp6 is open for 8080, awx_ee cannot reach it... is there somewhere I need to enable tcp4?

[kurokobo]

Hi,

> when I look for open ports on the receptor host only tcp6 is open for 8080

"IPv4-mapped IPv6" may look like that.

In such case, even if you only see tcp6, you should still be able to connect with IPv4.

You should test the TCP connection to 8080 port for IPv4 by nc or telnet on the execution node.

I think your IPv4 is reachable but if you want to use specific IPv4 address to listen on,

you can add "bindaddr: <IP Address>" under "tcp-listener" and restart receptor.

> - tcp-listener:

>     port: 27199

>     tls: tls_server

>     bindaddr: x.x.x.x   👈👈👈

As I mentioned before, have you checked the awx-ee logs?

Other than that, you can find various hints by looking at the configuration file (/etc/receptor/receptor.conf) in awx-ee container

or looking at the logs of the Receptor on the execution node side.

Regards,

--

@kurokobo

------- Original Message -------

To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/d2e97628-85f8-431f-9720-a83c74575d8fn%40googlegroups.com.

[JC PiCoDoN]

Hi,

I have tried a fresh install of the exec. node using the bundle on a Rocky 9 machine... the playbook fails but after adding pip on the host the playbook installs properly, receptor runs but the port shows as tcp6 only (via netstat -tlpn):

receptor.service - Receptor
     Loaded: loaded (/usr/lib/systemd/system/receptor.service; enabled; vend>
    Drop-In: /etc/systemd/system/receptor.service.d
             └─override.conf
     Active: active (running) since Thu 2023-03-23 14:26:05 EDT; 6s ago
   Main PID: 15563 (receptor)
      Tasks: 7 (limit: 100428)
     Memory: 9.2M
        CPU: 97ms
     CGroup: /system.slice/receptor.service
             └─15563 /usr/bin/receptor -c /etc/receptor/receptor.conf

netstat -tlpn
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
tcp6       0      0 :::8080                 :::*                    LISTEN      -
tcp6       0      0 :::22                   :::*                    LISTEN      -

If it try to connect to port 8080 it fails:

nc -vz host.domain.local 8080
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: No route to host.

I have also edited the receptor.conf file and added the bind address (bindaddr) but no luck.

On another host, I have deployed receptor 1.3.0 using docker-compose and this one seems to work on port 8080:

nc -vz host2.domain.local 8080
host2.domain.local (xx.xx.xx.xx:8080) open

Could it be the bundle version the issue?

Thank for your help again!!

[JC PiCoDoN]

Reg. the awx_ee logs, the only thing relevant is this, coming over and over:

WARNING 2023/03/23 18:41:34 Backend connection failed (will retry): dial tcp xx.xx.xx.xx:8080: connect: no route to host

WARNING 2023/03/23 18:41:37 Backend connection failed (will retry): tls: first record does not look like a TLS handshake

[kurokobo]

Hi,

> but the port shows as tcp6 only

Again, I think this is not an issue. As I mentioned before this is usually "IPv4-mapped IPv6" address and can be connected via IPv4.

I've deployed Rocky 9, add intance on AWX with port 8080, and run installer for it. I got following output after installation.

This node can be connected via IPv4 and works well.

> [root@exec01 ~]# netstat -tlpn

> Active Internet connections (only servers)

> Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    

> tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      909/sshd: /usr/sbin

> tcp6       0      0 :::8080                 :::*                    LISTEN      6561/receptor      

> tcp6       0      0 :::22                   :::*                    LISTEN      909/sshd: /usr/sbin

> Could it be the bundle version the issue?

I don't think so, "no route to host" is simply network-related issue, not receptor issue.

Installer do nothing to the host network e.g. iptables, nftables, firewalld.

Could you please double check the firewalld on your exec node, or some firewalls in your network configured correctly.

> sudo firewall-cmd --add-port=8080/tcp --permanent
> sudo firewall-cmd --reload

Regards,

--

@kurokobo

------- Original Message -------

To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/454a9698-4165-49bb-8bc1-5b4d68c9a296n%40googlegroups.com.

[JC PiCoDoN]

Oops, I feel so bad I didn't check the host's firewall (I thought it was disabled)... it did fix the issue and I can now see my Exec. Node! Thank you so much for the support!

On another note, I followed your excellent use case on Github to try to play with my receptor running with docker-compose...

I managed to create the necessary files (hostname.crt and hostname.key) with the awx_ee container and add them to my receptor, as well as the mesh CA.

Since the signature is enabled for the worker, I need to provide the public key (work_public_key.pem): where can I find this one or generate it?

Thank you!!

[kurokobo]

Hi,

I am glad to hear that your exec node works properly :)

If anything, I want installer to configure firewalld automatically. It may be good topic to create new issue as an RFE.

> On another note, I followed your excellent use case on Github to try to play with my receptor running with docker-compose...

> I managed to create the necessary files (hostname.crt and hostname.key) with the awx_ee container and add them to my receptor, as well as the mesh CA.

> Since the signature is enabled for the worker, I need to provide the public key (work_public_key.pem): where can I find this one or generate it?

Hope you enjoy my repo :)

As you know "work_public_key.pem" is contained install bundle,

and you can dig how install bundle is generated inside the AWX here: https://github.com/ansible/awx/blob/devel/awx/api/views/instance_install_bundle.py

This code works on awx-web container, and you can see "work-public-key.pem" is copy of "/etc/receptor/signing/work-public-key.pem" (Line 66).

So, this is the file you want.

> $ kubectl -n awx exec -it deployment/awx -c awx-web -- ls -l /etc/receptor/signing/work-public-key.pem

> -rw-r--r--. 1 root 1000 799 Mar 22 05:13 /etc/receptor/signing/work-public-key.pem

As a side note, this file is generated by AWX Operator and is stored as a Secret resource on K8s,

AWX Operator makes awx-web container in AWX pod to mount that Secret as "/etc/receptor/signing/work-public-key.pem".

There are two keys; public key is for awx-web container, and private key is for awx-ee and awx-task container.

> $ kubectl -n awx describe secret awx-receptor-work-signing

> Name:         awx-receptor-work-signing

> ...

> Data

> ====

> work-private-key.pem:  3242 bytes

> work-public-key.pem:   799 bytes

> $ kubectl -n awx describe deployment/awx

> ...

>    awx-web:

>     ...

>     Mounts:

>       ...

>       /etc/receptor/signing/work-public-key.pem from awx-receptor-work-signing (ro,path="work-public-key.pem")

Have fun with Receptor ;)

Regards,

--

@kurokobo

------- Original Message -------

To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/752d00f6-473e-494f-8ba6-22ed13da5f8cn%40googlegroups.com.

[JC PiCoDoN]

Thanks for the public_key location and all the help!!

Have a great weekend :)