AWX HA

[Tom K.]

Hello All,

Apologies if this should go under a different mailing list. 

Wanted to know when setting up AWX, is it possible to:

1) Specify the path where AWX will install too? 

2) Point to an external RabbitMQ Cluster?  We have our own and it would be much simpler if we can just connect to one. 

3) Point to an external PostgreSQL Cluster?  Same reason as above. 

Cheers,

TK

[Matthew Jones]

Since AWX is installed as a container #1 isn't super relevant?

2) No

3) Yes see: https://github.com/ansible/awx/blob/devel/installer/inventory#L63-L69

--
You received this message because you are subscribed to the Google Groups "AWX Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to awx-project...@googlegroups.com.
To post to this group, send email to awx-p...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/a261f703-5ddc-4b40-9512-d5181ec59380%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tom K.]

Hey Matthew,

So here's some background.  We're attempting to set AWX up w/ HA.  So instead of internally hosted single MQ or Postgres Docker images, we want to connect them to Highly Available PostgreSQL and RabbitMQ clusters we have in containers or otherwise.  What this means we only have to take care of the Docker/AWX GUI container and not the single RabbitMQ and single PostgreSQL containers, should issues arise within it.

Having said that, I would only need to cater to the AWX container and any number of instances of it on separate hosts for redundancy.

In regards to keeping everything in the Docker containers, how can I achieve HA via 3 redundant AWX instances assuming everything is kept in the Docker containers?  Assume OpenShift, Kubernetes are not available and I'm using VM's w/ Docker  ( Assuming KVM and VMWare for example ) .  Using the various documentation pages available, I'm still having trouble visualizing how the 3  x RabbitMQ instances and 3 x PostgreSQL instances are clustered and talk to each other in the event 1-2 nodes go offline.  What replication is used on the 3 instances of PostgreSQL and the 3 instances of RabbitMQ, for example, if everything is kept in Docker Containers.

This all feeds into the supportability realm of things.  

Cheers,

TK

[Matthew Jones]

Yeah you are pretty far outside of our supported deployment scenario. You'll need to sort out the postgres and rabbitmq bits on your own, that's a deep rabbit hole that I solved in a particular way for the Docker deployment and deal with in a different way on K8s/openshift. If those platforms aren't available to yhou, then you'll need to roll your own solution. Note that each AWX worker needs to be added to the cluster individually and they'll need their own queues/exchanges to negotiate work between each other. That only works when all AWX cluster members know about each other and know how to route messages to each other individually, to say nothing of actually configuring rabbitmq's own clustering You've got a good bit of work to do here.

I'll also caution that any random AWX upgrade is probably going to break you in strange ways that we won't be able to help with. If you're okay doing the work to make AWX work and then maintaining that across upgrades then good luck.

postgresql high availability is another matter altogether, and I'll point out that the AWX installer does not set this up (it only configured a single postgres instance, or requires that you provide connectivity to an external postgres that you maintain)... it's not really a big deal to set up streaming replication on your own. There are other off-the-shelf solutions for automatic failover and problem detection but that's out of scope of this mailing list.

To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/dcb8ad11-53ad-4538-8bf3-bc1ebeba56d6%40googlegroups.com.

[Tom K.]

Hey Matthew,

Ok, noted all that.

I've got most of all these bits except the AWX piece actually.  Here's what it looks like so far.

RabbitMQ Cluster is ready and tested.  

PostgreSQL 10 w/ Patroni is also up and running.   

The two above clusters were relatively easy.  The single instance of AWX I have running connected fine to the PostgreSQL 10 Cluster via the VIP I've setup.  It created tables without issues.  What remains:

1) Point AWX to the RabbitMQ Cluster.  
2) Confirm that subsequent AWX container's / instances can use the same PostgreSQL DB.  

Now you've mentioned "Note that each AWX worker needs to be added to the cluster individually and they'll need their own queues/exchanges to negotiate work between each other."  By 'their own' do you mean a shared queue/exchange account on the RabbitMQ instance / cluster?

Where does AWX keep the RabbitMQ connection details after install?  I'd like to just change that to point to the RabbitMQ cluster and see how that works.

My future upgrades would be to just install net-new instances of AWX from the latest git branch, migrate the DB to another on the same cluster and start things up again.  I'll usually stick a VIP via Haproxy / Keepalived on top of the AWX instances and set a friendly name via IPA over that.  

Cheers,

Tom

PS: FWIW, here's the cluster setups I've used: http://mdevsys.com/wp/postgres-sql-ha-cluster-quick-start-guide/ , http://mdevsys.com/wp/install-rabbitmq-in-high-availability/

[Tom K.]

I notice the AWX task container points to the AWX RabbitMQ docker instance already like so:

RABBITMQ_ENV_RABBITMQ_DEFAULT_PASS=guest

RABBITMQ_ENV_RABBITMQ_DEFAULT_USER=guest

RABBITMQ_ENV_RABBITMQ_DEFAULT_VHOST=awx

RABBITMQ_ENV_RABBITMQ_ERLANG_COOKIE=cookiemonster

RABBITMQ_ENV_RABBITMQ_GITHUB_TAG=v3.7.4

RABBITMQ_ENV_RABBITMQ_GPG_KEY=0A9AF2115F4687BD29803A206B73A36E6026DFCA

RABBITMQ_ENV_RABBITMQ_HOME=/opt/rabbitmq

RABBITMQ_ENV_RABBITMQ_LOGS=-

RABBITMQ_ENV_RABBITMQ_SASL_LOGS=-

RABBITMQ_ENV_RABBITMQ_VERSION=3.7.4

RABBITMQ_HOST=rabbitmq

RABBITMQ_NAME=/awx_task/rabbitmq

RABBITMQ_PASSWORD=guest

RABBITMQ_PORT=5672

RABBITMQ_PORT_15671_TCP=tcp://172.17.0.2:15671

RABBITMQ_PORT_15671_TCP_ADDR=172.17.0.2

RABBITMQ_PORT_15671_TCP_PORT=15671

RABBITMQ_PORT_15671_TCP_PROTO=tcp

RABBITMQ_PORT_15672_TCP=tcp://172.17.0.2:15672

RABBITMQ_PORT_15672_TCP_ADDR=172.17.0.2

RABBITMQ_PORT_15672_TCP_PORT=15672

RABBITMQ_PORT_15672_TCP_PROTO=tcp

RABBITMQ_PORT_25672_TCP=tcp://172.17.0.2:25672

RABBITMQ_PORT_25672_TCP_ADDR=172.17.0.2

RABBITMQ_PORT_25672_TCP_PORT=25672

RABBITMQ_PORT_25672_TCP_PROTO=tcp

RABBITMQ_PORT_4369_TCP=tcp://172.17.0.2:4369

RABBITMQ_PORT_4369_TCP_ADDR=172.17.0.2

RABBITMQ_PORT_4369_TCP_PORT=4369

RABBITMQ_PORT_4369_TCP_PROTO=tcp

RABBITMQ_PORT_5671_TCP=tcp://172.17.0.2:5671

RABBITMQ_PORT_5671_TCP_ADDR=172.17.0.2

RABBITMQ_PORT_5671_TCP_PORT=5671

RABBITMQ_PORT_5671_TCP_PROTO=tcp

RABBITMQ_PORT_5672_TCP=tcp://172.17.0.2:5672

RABBITMQ_PORT_5672_TCP_ADDR=172.17.0.2

RABBITMQ_PORT_5672_TCP_PORT=5672

RABBITMQ_PORT_5672_TCP_PROTO=tcp

RABBITMQ_USER=guest

RABBITMQ_VHOST=awx

And the host is set using the 'rabbitmq' variable: 

../tools/docker-compose.yml:      RABBITMQ_HOST: rabbitmq

Seems like changing this to to my external host might do the trick?  ;)  Haven't had the time to look where rabbitmq is set from.

Cheers,
TK

[Tom K.]

Managed to disable the Docker RabbitMQ and point it to my RabbitMQ Cluster by making these changes:

1) Delete all previous containers (ie cleanup environment)

2) Remove all images.

3) Stop docker.

4) Add parameter rabbitmq_host=<RabbitMQ Cluster VIP>  .  In my case rmq-c01 . 

5) Execute:   ansible-playbook -i inventory install.yml

6) Start the GUI: http://awx-m01.nix.mds.xyz/  

NOTE: I did not clear the DB.   I wanted to see if it will pick up the earlier one the installer created.

7) Need to test further.

[root@awx-m01 installer]# cat inventory |grep -v "#"
localhost ansible_connection=local ansible_python_interpreter="/usr/bin/env python"

[all:vars]

dockerhub_base=ansible

awx_task_hostname=awx
awx_web_hostname=awxweb
postgres_data_dir=/tmp/pgdocker
host_port=80

docker_compose_dir=/var/lib/awx

pg_hostname=psql-c01
pg_username=awx
pg_password=awxpass
pg_database=awx
pg_port=5432

rabbitmq_host=rmq-c01
rabbitmq_port=5672
rabbitmq_vhost=tower
rabbitmq_username=tower
rabbitmq_password='password'
rabbitmq_cookie=rabbitmqcookie

admin_user=admin
admin_password=password

create_preload_data=True

secret_key=awxsecret
[root@awx-m01 installer]#

[root@awx-m01 awx]# git status
# On branch devel
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#       modified:   installer/inventory
#       modified:   installer/roles/local_docker/tasks/standalone.yml
#       modified:   installer/roles/local_docker/templates/docker-compose.yml.j2
#
no changes added to commit (use "git add" and/or "git commit -a")
[root@awx-m01 awx]#
[root@awx-m01 awx]# git diff
diff --git a/installer/inventory b/installer/inventory
index 847ba60..980b674 100644
--- a/installer/inventory
+++ b/installer/inventory
@@ -62,15 +62,22 @@ docker_compose_dir=/var/lib/awx


 # Set pg_hostname if you have an external postgres server, otherwise
 # a new postgres service will be created
-# pg_hostname=postgresql
+pg_hostname=psql-c01
 pg_username=awx
 pg_password=awxpass
 pg_database=awx
 pg_port=5432


 # RabbitMQ Configuration
-rabbitmq_password=awxpass
-rabbitmq_erlang_cookie=cookiemonster
+# rabbitmq_password=awxpass
+# rabbitmq_erlang_cookie=cookiemonster
+
+rabbitmq_host=rmq-c01
+rabbitmq_port=5672
+rabbitmq_vhost=tower
+rabbitmq_username=tower
+rabbitmq_password='password'
+rabbitmq_cookie=rabbitmqcookie


 # Use a local distribution build container image for building the AWX package
 # This is helpful if you don't want to bother installing the build-time dependencies as
diff --git a/installer/roles/local_docker/tasks/standalone.yml b/installer/roles/local_docker/tasks/standalone.yml
index ad7a064..9f05872 100644
--- a/installer/roles/local_docker/tasks/standalone.yml
+++ b/installer/roles/local_docker/tasks/standalone.yml
@@ -26,6 +26,7 @@
       RABBITMQ_ERLANG_COOKIE: "{{ rabbitmq_erlang_cookie }}"
       RABBITMQ_DEFAULT_USER: "{{ rabbitmq_default_username }}"
       RABBITMQ_DEFAULT_PASS: "{{ rabbitmq_default_password }}"
+  when: rabbitmq_host is not defined or rabbitmq_host == ''
   register: rabbitmq_container_activate


 - name: Activate memcached container
@@ -40,41 +41,72 @@
     seconds: 15
   when: postgres_container_activate.changed or rabbitmq_container_activate.changed


+
+
+
+- name: Set properties without postgres and rabbitmq for awx_web
+  set_fact:
+    pg_hostname_actual: "{{ pg_hostname }}"
+    rabbitmq_host_actual: "{{ rabbitmq_host }}"
+    awx_web_container_links:
+      - memcached
+  when: pg_hostname is defined and rabbitmq_host is defined
+
 - name: Set properties without postgres for awx_web
   set_fact:
     pg_hostname_actual: "{{ pg_hostname }}"
+    rabbitmq_host_actual: rabbitmq
     awx_web_container_links:
       - rabbitmq
       - memcached
-  when: pg_hostname is defined
+  when: pg_hostname is defined and ( rabbitmq_host is not defined or rabbitmq_host == "" )


 - name: Set properties with postgres for awx_web
   set_fact:
     pg_hostname_actual: postgres
+    rabbitmq_host_actual: rabbitmq
     awx_web_container_links:
       - rabbitmq
       - memcached
       - postgres
-  when: pg_hostname is not defined or pg_hostname == ''
+  when: ( pg_hostname is not defined or pg_hostname == '' ) and ( rabbitmq_host is not defined or rabbitmq_host == "" )
+
+
+
+
+
+- name: Set properties without postgres and rabbitmq for awx_task
+  set_fact:
+    pg_hostname_actual: "{{ pg_hostname }}"
+    rabbitmq_host_actual: "{{ rabbitmq_host }}"
+    awx_task_container_links:
+      - memcached
+      - awx_web:awxweb
+  when: pg_hostname is defined and rabbitmq_host is defined


 - name: Set properties without postgres for awx_task
   set_fact:
     pg_hostname_actual: "{{ pg_hostname }}"
+    rabbitmq_host_actual: rabbitmq
     awx_task_container_links:
       - rabbitmq
       - memcached
       - awx_web:awxweb
-  when: pg_hostname is defined
+  when: pg_hostname is defined and ( rabbitmq_host is not defined or rabbitmq_host == "" )


 - name: Set properties with postgres for awx_task
   set_fact:
     pg_hostname_actual: postgres
+    rabbitmq_host_actual: rabbitmq
     awx_task_container_links:
       - rabbitmq
       - memcached
       - awx_web:awxweb
       - postgres
-  when: pg_hostname is not defined or pg_hostname == ''
+  when: pg_hostname is not defined or pg_hostname == '' and ( rabbitmq_host is not defined or rabbitmq_host == "" )
+
+
+


 - name: Activate AWX Web Container
   docker_container:
@@ -106,7 +138,7 @@
       DATABASE_HOST: "{{ pg_hostname_actual }}"
       RABBITMQ_USER: "{{ rabbitmq_default_username }}"
       RABBITMQ_PASSWORD: "{{ rabbitmq_default_password }}"
-      RABBITMQ_HOST: "rabbitmq"
+      RABBITMQ_HOST: "{{ rabbitmq_host_actual }}"
       RABBITMQ_PORT: "{{ rabbitmq_port }}"
       RABBITMQ_VHOST: "{{ rabbitmq_default_vhost }}"
       MEMCACHED_HOST: "memcached"
@@ -147,7 +179,7 @@
       DATABASE_PORT: "{{ pg_port }}"
       RABBITMQ_USER: "{{ rabbitmq_default_username }}"
       RABBITMQ_PASSWORD: "{{ rabbitmq_default_password }}"
-      RABBITMQ_HOST: "rabbitmq"
+      RABBITMQ_HOST: "{{ rabbitmq_host_actual }}"
       RABBITMQ_PORT: "{{ rabbitmq_port }}"
       RABBITMQ_VHOST: "{{ rabbitmq_default_vhost }}"
       MEMCACHED_HOST: "memcached"
diff --git a/installer/roles/local_docker/templates/docker-compose.yml.j2 b/installer/roles/local_docker/templates/docker-compose.yml.j2
index 5e227bf..d2a7e31 100644
--- a/installer/roles/local_docker/templates/docker-compose.yml.j2
+++ b/installer/roles/local_docker/templates/docker-compose.yml.j2
@@ -5,9 +5,11 @@ services:
   web:
     image: {{ awx_web_docker_actual_image }}
     depends_on:
+      {% if pg_hostname is not defined %}
       - rabbitmq
+      {% endif %}
       - memcached
-      {% if pg_hostname is not defined %}
+      {% if rabbitmq_host is not defined %}
       - postgres
       {% endif %}
     ports:
[root@awx-m01 awx]#
[root@awx-m01 awx]#

[Tom K.]

I tested the queue connectivity by running a receiver on AWX WEB and a sender on AWX TASK.  Results:

[root@awx ~]# ./r-rmq.py
 [*] Waiting for messages. To exit press CTRL+C
 [x] Received 'Blah! Blah! World!'
 [x] Received 'Blah! Blah! World!'
 [x] Received 'Blah! Blah! World!'
 [x] Received 'Blah! Blah! World!'
 [x] Received 'Blah! Blah! World!'

[root@awxweb awx]# ./s-mq.py
 [x] Sent 'Blah! Blah! World!'
[root@awxweb awx]# ./s-mq.py
 [x] Sent 'Blah! Blah! World!'
[root@awxweb awx]# ./s-mq.py
 [x] Sent 'Blah! Blah! World!'
[root@awxweb awx]#

Still need more testing to be sure that it's 100%.

+  when: ( pg_hostname is not defined </spa

[Adrian Sebastian Dutu]

Hi Tom,

I am trying to do get clustering working with docker also, just that I don't need an external RabbitMQ. I am trying to get the rabbitmq containers on each node in the same cluster. I managed to get the awx_web and awx_task in the cluster. I am wondering how did you install it. In my docker compose file I do not see that many variables. I see just 

      RABBITMQ_USER: guest

      RABBITMQ_PASSWORD: guest

      RABBITMQ_HOST: rabbitmq

      RABBITMQ_PORT: 5672

      RABBITMQ_VHOST: awx

[CV]

Hi Tom,

i'm trying to replicate what you did as a test. Just connecting 2 containers to a single rabbitmq host (i'll set up a clustered rabbitmq for production later). When spinning up the AWX web container I get an error: 

fatal: [localhost]: FAILED! => {"changed": false, "msg": "Error creating container: 500 Server Error: Internal Server Error (\"{\"message\":\"Could not get container for rabbitmq\"}\")"}

I believe I modified all of the files as specified above, do you have any idea what might be causing this error?

[Sujith A R]

I am soon writing up a playbook for building HA in latest AWX version, since there are so many threads in here related with this and most of the informations are scattered across. This is what all we need if are looking for HA in latest AWX version ( V 3.0.1) in docker on linux ,

* Make the DB centralized either as a Docker container / PaSS / external .

* Build a RabbitMQ cluster across all nodes and get rid of the rabbitmq container deployment.

* Update the necessary user permission and policy to the rabbitmq cluster.

* Build a custom image with the appropriate changes in the awx_web & awx_task container and run it with docker and point this to the cluster running on their respective node. There are no changes in memcache container.

PS: We don't have to make any changes for celery worker in latest AWX since it is able to pickup and execute jobs as per the node we defined under instance group. 

By referring Ansible tower latest installation roles could help to understand how these all are works in Enterprise version and not in AWX. It is a hidden gem and we need to explore our own.

These all are the links helps if someone wish to explore, 

https://docs.ansible.com/ansible-tower/latest/html/administration/clustering.html#job-runtime-behavior 

https://releases.ansible.com/ansible-tower/setup/?extIdCarryOver=true&sc_cid=701f2000001OH7YAAW

https://groups.google.com/forum/m/#!topic/awx-project/-YABJ1hA3XI 

Stay tuned.

To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/c6830218-bc6c-478a-ac09-d055fe1f93f0%40googlegroups.com.