Missing awx instance

[Elia Curti]

Hello,

I've deployed in my aks cluster AWX with awx-operator 0.28.0

I have set 2 replicas for AWX.

Now are both running but as you can see from RESTARTS they change status in "CrashLoopBackOff". 

Then automatically we come back to running... is it normal?

Thank you very much.

Elia

[Manikandan Balasubramanian]

I see a similar issue in AKS cluster. For me, it goes to CrashLoopBackOff and stays that way when I spin 4 AWX nodes. My base AKS cluster has 8 nodes, 32CPUs, and there is no reason why this could happen because of the load in the AKS cluster itself. Any pointers on why this happens would help greatly. I am running: AWX version: 21.3.0

Thanks and Regards,

Mani

[Elia Curti]

I have an AKS cluster dedicated to AWX with 3 nodes and globally 12 CPU... It's a staging enviroment there is no load on the cluster. Now I'm testing it running manually some templates. The reason must be another... could not be the load on cluster.

From yesterday (23h) pods are running without restarts... I don't understand what it causing this issue.

Thanks for you reply,

Elia

[AWX Project]

In this case we suspect the awx-operator to have some useful logs around this,

You might change you AWX CR (custom resource) with "kubectl edit awx"  to set

no_log: "false"
so that you can get more details about pod failures when viewing awx-operator logs after the crashloopbackoff

kubectl logs deployments/awx-operator-controller-manager -c awx-manager -f

you may grep the logs for "Failed=1" and look through those to find relevant error messages.

AWX Team

[Elia Curti]

Hello,

I've done a new clean installation with awx-operator 0.28.0  and now I'm having the issue so it couldn't be due to load on aks cluster.

I've found this in awx-manager container:

localhost                  : ok=48   changed=1    unreachable=0    failed=1    skipped=39   rescued=0    ignored=0
{"level":"error","ts":1663658311.580133,"msg":"Reconciler error","controller":"awx-controller","object":{"name":"awx","namespace":"awx"},"namespace":"awx","name":"awx","reconcileID":"1b0a8f47-4283-49e6-abd1-264fefb86f4a","error":"event runner on failed","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controlle...@v0.12.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controlle...@v0.12.1/pkg/internal/controller/controller.go:234"}

Thank you for your help.

[Elia Curti]

Is there any information from the log above to help you understand the issue?

Thank you,

Elia

[Elia Curti]

Hello,

here we are again:

The "CrashLoopBackOff" issue is related to "awx-web" container. 

What could be the problem? 

Thank you very much.

I'm watching container logs but I find no errors:

[wait-for-migrations] Waiting for database migrations...
[wait-for-migrations] Attempt 1 of 30
2022-10-13 07:29:47,921 INFO RPC interface 'supervisor' initialized
2022-10-13 07:29:47,921 INFO RPC interface 'supervisor' initialized
2022-10-13 07:29:47,921 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2022-10-13 07:29:47,921 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2022-10-13 07:29:47,921 INFO supervisord started with pid 7
2022-10-13 07:29:47,921 INFO supervisord started with pid 7
2022-10-13 07:29:48,923 INFO spawned: 'superwatcher' with pid 26
2022-10-13 07:29:48,923 INFO spawned: 'superwatcher' with pid 26
2022-10-13 07:29:48,925 INFO spawned: 'nginx' with pid 27
2022-10-13 07:29:48,925 INFO spawned: 'nginx' with pid 27
2022-10-13 07:29:48,927 INFO spawned: 'uwsgi' with pid 28
2022-10-13 07:29:48,927 INFO spawned: 'uwsgi' with pid 28
2022-10-13 07:29:48,930 INFO spawned: 'daphne' with pid 29
[uWSGI] getting INI configuration from /etc/tower/uwsgi.ini
*** Starting uWSGI 2.0.18 (64bit) on [Thu Oct 13 07:29:48 2022] ***
compiled with version: 11.3.1 20220421 (Red Hat 11.3.1-2) on 16 September 2022 16:25:19
os: Linux-5.4.0-1083-azure #87~18.04.1-Ubuntu SMP Fri Jun 3 13:19:07 UTC 2022
nodename: awx-7566768857-pm6wn
machine: x86_64
clock source: unix
detected number of CPU cores: 4
current working directory: /var/lib/awx
detected binary path: /var/lib/awx/venv/awx/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 127.0.0.1:8050 fd 3
Python version: 3.9.13 (main, Jul 25 2022, 00:00:00)  [GCC 11.3.1 20220421 (Red Hat 11.3.1-2)]
2022-10-13 07:29:48,930 INFO spawned: 'daphne' with pid 29
2022-10-13 07:29:48,932 INFO spawned: 'wsbroadcast' with pid 30
2022-10-13 07:29:48,932 INFO spawned: 'wsbroadcast' with pid 30
2022-10-13 07:29:48,935 INFO spawned: 'awx-rsyslogd' with pid 32
2022-10-13 07:29:48,935 INFO spawned: 'awx-rsyslogd' with pid 32
READY
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x7bb480
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 609552 bytes (595 KB) for 5 cores
*** Operational MODE: preforking ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 28)
spawned uWSGI worker 1 (pid: 33, cores: 1)
spawned uWSGI worker 2 (pid: 34, cores: 1)
spawned uWSGI worker 3 (pid: 35, cores: 1)
spawned uWSGI worker 4 (pid: 36, cores: 1)
spawned uWSGI worker 5 (pid: 37, cores: 1)
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
2022-10-13 07:29:49,981 INFO success: superwatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-13 07:29:49,981 INFO success: superwatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
WSGI app 0 (mountpoint='/') ready in 3 seconds on interpreter 0x7bb480 pid: 33 (default app)
WSGI app 0 (mountpoint='/') ready in 3 seconds on interpreter 0x7bb480 pid: 37 (default app)
WSGI app 0 (mountpoint='/') ready in 3 seconds on interpreter 0x7bb480 pid: 34 (default app)
WSGI app 0 (mountpoint='/') ready in 3 seconds on interpreter 0x7bb480 pid: 36 (default app)
WSGI app 0 (mountpoint='/') ready in 3 seconds on interpreter 0x7bb480 pid: 35 (default app)
2022-10-13 07:29:51,432 INFO     [-] daphne.cli Starting server at tcp:port=8051:interface=127.0.0.1
2022-10-13 07:29:51,432 INFO     Starting server at tcp:port=8051:interface=127.0.0.1
2022-10-13 07:29:51,434 INFO     [-] daphne.server HTTP/2 support not enabled (install the http2 and tls Twisted extras)
2022-10-13 07:29:51,434 INFO     HTTP/2 support not enabled (install the http2 and tls Twisted extras)
2022-10-13 07:29:51,434 INFO     [-] daphne.server Configuring endpoint tcp:port=8051:interface=127.0.0.1
2022-10-13 07:29:51,434 INFO     Configuring endpoint tcp:port=8051:interface=127.0.0.1
2022-10-13 07:29:51,435 INFO     [-] daphne.server Listening on TCP address 127.0.0.1:8051
2022-10-13 07:29:51,435 INFO     Listening on TCP address 127.0.0.1:8051
2022-10-13 07:29:51,965 INFO     [-] awx.main.wsbroadcast Unable to return currently active instance: No instance found with the current cluster host id, retry in 5s...
2022-10-13 07:29:57,428 INFO exited: wsbroadcast (exit status 0; not expected)
2022-10-13 07:29:57,428 INFO exited: wsbroadcast (exit status 0; not expected)
2022-10-13 07:29:58,431 INFO spawned: 'wsbroadcast' with pid 52
2022-10-13 07:29:58,431 INFO spawned: 'wsbroadcast' with pid 52
2022-10-13 07:30:00,437 INFO     [-] awx.main.wsbroadcast Unable to return currently active instance: No instance found with the current cluster host id, retry in 5s...
2022-10-13 07:30:05,830 INFO exited: wsbroadcast (exit status 0; not expected)
2022-10-13 07:30:05,830 INFO exited: wsbroadcast (exit status 0; not expected)
2022-10-13 07:30:07,834 INFO spawned: 'wsbroadcast' with pid 55
2022-10-13 07:30:07,834 INFO spawned: 'wsbroadcast' with pid 55
2022-10-13 07:30:09,892 INFO     [-] awx.main.wsbroadcast Unable to return currently active instance: No instance found with the current cluster host id, retry in 5s...
2022-10-13 07:30:15,277 INFO exited: wsbroadcast (exit status 0; not expected)
2022-10-13 07:30:15,277 INFO exited: wsbroadcast (exit status 0; not expected)
2022-10-13 07:30:18,283 INFO spawned: 'wsbroadcast' with pid 58
2022-10-13 07:30:18,283 INFO spawned: 'wsbroadcast' with pid 58
2022-10-13 07:30:19,285 INFO success: nginx entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
2022-10-13 07:30:19,285 INFO success: nginx entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
2022-10-13 07:30:19,285 INFO success: uwsgi entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
2022-10-13 07:30:19,285 INFO success: uwsgi entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
2022-10-13 07:30:19,285 INFO success: daphne entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
2022-10-13 07:30:19,285 INFO success: daphne entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
2022-10-13 07:30:19,285 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
2022-10-13 07:30:19,285 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
2022-10-13 07:30:20,136 INFO     [-] awx.main.wsbroadcast Unable to return currently active instance: No instance found with the current cluster host id, retry in 5s...
2022-10-13 07:30:25,562 INFO exited: wsbroadcast (exit status 0; not expected)
2022-10-13 07:30:25,562 INFO exited: wsbroadcast (exit status 0; not expected)
2022-10-13 07:30:26,564 INFO gave up: wsbroadcast entered FATAL state, too many start retries too quickly
2022-10-13 07:30:26,564 INFO gave up: wsbroadcast entered FATAL state, too many start retries too quickly
2022-10-13 07:30:27,566 WARN received SIGQUIT indicating exit request
2022-10-13 07:30:27,566 WARN received SIGQUIT indicating exit request
Processing Event: ver:3.0 server:supervisor serial:0 pool:superwatcher poolserial:0 eventname:PROCESS_STATE_FATAL len:68
2022-10-13 07:30:27,577 INFO waiting for superwatcher, nginx, uwsgi, daphne, awx-rsyslogd to die
...brutally killing workers...
2022-10-13 07:30:27,566 INFO     [-] daphne.server Killed 0 pending application instances
2022-10-13 07:30:27,566 INFO     Killed 0 pending application instances
2022-10-13 07:30:27,577 INFO waiting for superwatcher, nginx, uwsgi, daphne, awx-rsyslogd to die
2022-10-13 07:30:27,578 INFO stopped: awx-rsyslogd (exit status 0)
2022-10-13 07:30:27,578 INFO stopped: awx-rsyslogd (exit status 0)
2022-10-13 07:30:27,610 INFO stopped: nginx (exit status 0)
2022-10-13 07:30:27,610 INFO stopped: nginx (exit status 0)
2022-10-13 07:30:27,965 INFO stopped: daphne (exit status 0)
2022-10-13 07:30:27,965 INFO stopped: daphne (exit status 0)
worker 1 buried after 1 seconds
worker 2 buried after 1 seconds
worker 3 buried after 1 seconds
worker 4 buried after 1 seconds
worker 5 buried after 1 seconds
binary reloading uWSGI...
chdir() to /var/lib/awx
closing all non-uwsgi socket fds > 2 (max_fd = 1048576)...
found fd 3 mapped to socket 0 (127.0.0.1:8050)
running /var/lib/awx/venv/awx/bin/uwsgi
[uWSGI] getting INI configuration from /etc/tower/uwsgi.ini
*** Starting uWSGI 2.0.18 (64bit) on [Thu Oct 13 07:30:29 2022] ***
compiled with version: 11.3.1 20220421 (Red Hat 11.3.1-2) on 16 September 2022 16:25:19
os: Linux-5.4.0-1083-azure #87~18.04.1-Ubuntu SMP Fri Jun 3 13:19:07 UTC 2022
nodename: awx-7566768857-pm6wn
machine: x86_64
clock source: unix
detected number of CPU cores: 4
current working directory: /var/lib/awx
detected binary path: /var/lib/awx/venv/awx/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 inherited INET address 127.0.0.1:8050 fd 3
Python version: 3.9.13 (main, Jul 25 2022, 00:00:00)  [GCC 11.3.1 20220421 (Red Hat 11.3.1-2)]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x10e5d90
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
2022-10-13 07:30:30,582 INFO waiting for superwatcher, uwsgi to die
2022-10-13 07:30:30,582 INFO waiting for superwatcher, uwsgi to die
mapped 609552 bytes (595 KB) for 5 cores
*** Operational MODE: preforking ***
*** uWSGI is running in multiple interpreter mode ***
gracefully (RE)spawned uWSGI master process (pid: 28)
spawned uWSGI worker 1 (pid: 61, cores: 1)
spawned uWSGI worker 2 (pid: 62, cores: 1)
spawned uWSGI worker 3 (pid: 63, cores: 1)
spawned uWSGI worker 4 (pid: 64, cores: 1)
spawned uWSGI worker 5 (pid: 65, cores: 1)
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
WSGI app 0 (mountpoint='/') ready in 2 seconds on interpreter 0x10e5d90 pid: 61 (default app)
WSGI app 0 (mountpoint='/') ready in 2 seconds on interpreter 0x10e5d90 pid: 63 (default app)
WSGI app 0 (mountpoint='/') ready in 2 seconds on interpreter 0x10e5d90 pid: 65 (default app)
WSGI app 0 (mountpoint='/') ready in 2 seconds on interpreter 0x10e5d90 pid: 64 (default app)
WSGI app 0 (mountpoint='/') ready in 2 seconds on interpreter 0x10e5d90 pid: 62 (default app)
2022-10-13 07:30:34,483 INFO waiting for superwatcher, uwsgi to die
2022-10-13 07:30:34,483 INFO waiting for superwatcher, uwsgi to die
2022-10-13 07:30:37,487 INFO waiting for superwatcher, uwsgi to die
2022-10-13 07:30:37,487 INFO waiting for superwatcher, uwsgi to die
2022-10-13 07:30:38,489 WARN killing 'uwsgi' (28) with SIGKILL
2022-10-13 07:30:38,489 WARN killing 'uwsgi' (28) with SIGKILL
2022-10-13 07:30:38,502 INFO stopped: uwsgi (terminated by SIGKILL)
2022-10-13 07:30:38,502 INFO stopped: uwsgi (terminated by SIGKILL)
2022-10-13 07:30:38,503 INFO stopped: superwatcher (terminated by SIGTERM)
2022-10-13 07:30:38,503 INFO stopped: superwatcher (terminated by SIGTERM)

g

[Elia Curti]

Edited awx and set "no_log: false".

For instance I've found this:

onTime\": \"2022-10-13T08:00:55Z\", \"lastUpdateTime\": \"2022-10-13T08:00:55Z\", \"message\": \"Deployment does not have minimum availability.\", \"reason\": \"MinimumReplicasUnavailable\", \"status\": \"False\", \"type\": \"Available\"}], \"observedGeneration\": 3, \"readyReplicas\": 1, \"replicas\": 2, \"unavailableReplicas\": 1, \"updatedReplicas\": 2}}}\n\r\nPLAY RECAP *********************************************************************\r\nlocalhost                  : ok=50   changed=0    unreachable=0    failed=1    skipped=51   rescued=0    ignored=0   \r\n\n","job":"8583884555945897733","name":"awx","namespace":"awx","error":"exit status 2"}
{"level":"error","ts":1665648202.4457707,"msg":"Reconciler error","controller":"awx-controller","object":{"name":"awx","namespace":"awx"},"namespace":"awx","name":"awx","reconcileID":"be6d797d-28f7-48bf-ad55-c3816633a639","error":"event runner on failed","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controlle...@v0.12.2/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controlle...@v0.12.2/pkg/internal/controller/controller.go:234"}
{"level":"error","ts":1665648347.4962242,"logger":"logging_event_handler","msg":"","name":"awx","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWX","event_type":"runner_on_failed","job":"114496043156805132","EventData.Task":"Apply deployment resources","EventData.TaskArgs":"","EventData.FailedTaskPath":"/opt/ansible/roles/installer/tasks/resources_configuration.yml:164","error":"[playbook task failed]"}
{"level":"error","ts":1665648347.7489324,"logger":"runner","msg":"ansible-playbook 2.9.27\r\n  config file = /etc/ansible/ansible.cfg\r\n  configured module search path = ['/usr/share/ansible/openshift']\r\n  ansible python module location = /usr/local/lib/python3.8/site-packages/ansible\r\n  executable location = /usr/local/bin/ansible-playbook\r\n  python version = 3.8.12 (default, Sep 16 2021, 10:46:05) [GCC 8.5.0 20210514 (Red Hat 8.5.0-3)]\r\nUsing /etc/ansible/ansible.cfg as config file\r\nSkipping callback 'actionable', as we already have a stdout callback.\nSkipping callback 'awx_display', as we already have a stdout callback.\nSkipping callback 'counter_enabled', as we already have a stdout callback.\nSkipping callback 'debug', as we already have a stdout callback.\nSkipping callback 'dense', as we already have a stdout callback.\nSkipping callback 'dense', as we already have a stdout callback.\nSkipping callback 'full_skip', as we already have a stdout callback.\nSkipping callback 'json', as we already have a stdout callback.\nSkipping callback 'minimal', as we already have a stdout callback.\nSkipping callback 'null', as we already have a stdout callback.\nSkipping callback 'oneline', as we already have a stdout callback.\nSkipping callback 'selective', as we already have a stdout callback.\nSkipping callback 'skippy', as we already have a stdout callback.\nSkipping callback 'stderr', as we already have a stdout callback.\nSkipping callback 'unixy', as we already have a stdout callback.\nSkipping callback 'yaml', as we already have a stdout callback.\n\r\nPLAYBOOK: awx.yml *********************************************************

Could help?

Thanks.

Il giorno venerdì 16 settembre 2022 alle 19:40:03 UTC+2 AWX Project ha scritto:

[Elia Curti]

Or this:

localhost                  : ok=50   changed=0    unreachable=0    failed=1    skipped=51   rescued=0    ignored=0

"creationTimestamp\": \"2022-06-29T15:11:08Z\", \"labels\": {\"app.kubernetes.io/component\": \"awx\", \"app.kubernetes.io/managed-by\": \"awx-operator\", \"app.kubernetes.io/name\": \"awx\", \"app.kubernetes.io/operator-version\": \"0.29.0\", \"app.kubernetes.io/part-of\": \"awx\"}, \"managedFields\": [{\"apiVersion\": \"v1\", \"fieldsType\": \"FieldsV1\", \"fieldsV1\": {\"f:data\": {\".\": {}, \"f:credentials.py\": {}, \"f:execution_environments.py\": {}, \"f:ldap.py\": {}}, \"f:metadata\": {\"f:annotations\": {\".\": {}, \"f:kubectl.kubernetes.io/last-applied-configuration\": {}}, \"f:labels\": {\".\": {}, \"f:app.kubernetes.io/component\": {}, \"f:app.kubernetes.io/managed-by\": {}, \"f:app.kubernetes.io/name\": {}, \"f:app.kubernetes.io/operator-version\": {}, \"f:app.kubernetes.io/part-of\": {}}, \"f:ownerReferences\": {\".\": {}, \"k:{\\\"uid\\\":\\\"093d7650-2976-4827-9751-e30d4640124d\\\"}\": {}}}, \"f:type\": {}}, \"manager\": \"OpenAPI-Generator\", \"operation\": \"Update\", \"time\": \"2022-06-29T15:11:08Z\"}], \"name\": \"awx-app-credentials\", \"namespace\": \"awx\", \"ownerReferences\": [{\"apiVersion\": \"awx.ansible.com/v1beta1\", \"kind\": \"AWX\", \"name\": \"awx\", \"uid\": \"093d7650-2976-4827-9751-e30d4640124d\"}], \"resourceVersion\": \"34937463\", \"uid\": \"ac79ebed-c1b1-4ebd-ad27-3d39a80bbaf3\"}, \"type\": \"Opaque\"}}\nok: [localhost] => (item=rbac/service_account) => {\"ansible_loop_var\": \"item\", \"changed\": false, \"item\": \"rbac/service_account\", \"result\": {\"results\": [{\"changed\": false, \"duration\": 0, \"method\": \"apply\", \"result\": {\"apiVersion\": \"v1\", \"kind\": \"ServiceAccount\", \"metadata\": {\"annotations\": {\"kubectl.kubernetes.io/last-applied-configuration\": \"{\\\"apiVersion\\\":\\\"v1\\\",\\\"kind\\\":\\\"ServiceAccount\\\",\\\"metadata\\\":{\\\"labels\\\":{\\\"app.kubernetes.io/component\\\":\\\"awx\\\",\\\"app.kubernetes.io/managed-by\\\":\\\"awx-operator\\\",\\\"app.kubernetes.io/name\\\":\\\"awx\\\",\\\"app.kubernetes.io/operator-version\\\":\\\"0.29.0\\\",\\\"app.kubernetes.io/part-of\\\":\\\"awx\\\"},\\\"name\\\":\\\"awx\\\",\\\"namespace\\\":\\\"awx\\\"}}\"}, \"creationTimestamp\": \"2022-06-29T15:11:09Z\", \"labels\": {\"app.kubernetes.io/component\": \"awx\", \"app.kubernetes.io/managed-by\": \"awx-operator\", \"app.kubernetes.io/name\": \"awx\", \"app.kubernetes.io/operator-version\": \"0.29.0\", \"app.kubernetes.io/part-of\": \"awx\"}, \"managedFields\": [{\"apiVersion\": \"v1\", \"fieldsType\": \"FieldsV1\", \"fieldsV1\": {\"f:metadata\": {\"f:annotations\": {\".\": {}, \"f:kubectl.kubernetes.io/last-applied-configuration\": {}}, \"f:labels\": {\".\": {}, \"f:app.kubernetes.io/component\": {}, \"f:app.kubernetes.io/managed-by\": {}, \"f:app.kubernetes.io/name\": {}, \"f:app.kubernetes.io/operator-version\": {}, \"f:app.kubernetes.io/part-of\": {}}, \"f:ownerReferences\": {\".\": {}, \"k:{\\\"uid\\\":\\\"093d7650-2976-4827-9751-e30d4640124d\\\"}\": {}}}}, \"manager\": \"OpenAPI-Generator\", \"operation\": \"Update\", \"time\": \"2022-06-29T15:11:09Z\"}, {\"apiVersion\": \"v1\", \"fieldsType\": \"FieldsV1\", \"fieldsV1\": {\"f:secrets\": {\".\": {}, \"k:{\\\"name\\\":\\\"awx-token-rxrvj\\\"}\": {}}}, \"manager\": \"kube-controller-manager\", \"operation\": \"Update\", \"time\": \"2022-06-29T15:11:09Z\"}], \"name\": \"awx\", \"namespace\": \"awx\", \"ownerReferences\": [{\"apiVersion\": \"awx.ansible.com/v1beta1\", \"kind\": \"AWX\", \"name\": \"awx\", \"uid\": \"093d7650-2976-4827-9751-e30d4640124d\"}], \"resourceVersion\": \"34937465\", \"uid\": \"5d87ef47-b612-48d6-b802-795bfb346d66\"}, \"secrets\": [{\"name\": \"awx-token-rxrvj\"}]}}, {\"changed\": false, \"duration\": 0, \"method\": \"apply\", \"result\": {\"apiVersion\": \"rbac.authorization.k8s.io/v1\", \"kind\": \"Role\", \"metadata\": {\"annotations\": {\"kubectl.kubernetes.io/last-applied-configuration\": \"{\\\"apiVersion\\\":\\\"rbac.authorization.k8s.io/v1\\\",\\\"kind\\\":\\\"Role\\\",\\\"metadata\\\":{\\\"name\\\":\\\"awx\\\",\\\"namespace\\\":\\\"awx\\\"},\\\"rules\\\":[{\\\"apiGroups\\\":[\\\"\\\"],\\\"resources\\\":[\\\"pods\\\"],\\\"verbs\\\":[\\\"get\\\",\\\"list\\\",\\\"watch\\\",\\\"create\\\",\\\"update\\\",\\\"patch\\\",\\\"delete\\\"]},{\\\"apiGroups\\\":[\\\"\\\"],\\\"resources\\\":[\\\"pods/log\\\"],\\\"verbs\\\":[\\\"get\\\"]},{\\\"apiGroups\\\":[\\\"\\\"],\\\"resources\\\":[\\\"pods/attach\\\"],\\\"verbs\\\":[\\\"create\\\"]},{\\\"apiGroups\\\":[\\\"\\\"],\\\"resources\\\":[\\\"secrets\\\"],\\\"verbs\\\":[\\\"get\\\",\\\"create\\\",\\\"delete\\\"]}]}\"}, \"creationTimestamp\": \"2022-06-29T15:11:09Z\", \"managedFields\": [{\"apiVersion\": \"rbac.authorization.k8s.io/v1\", \"fieldsType\": \"FieldsV1\", \"fieldsV1\": {\"f:metadata\": {\"f:annotations\": {\".\": {}, \"f:kubectl.kubernetes.io/last-applied-configuration\": {}}, \"f:ownerReferences\": {\".\": {}, \"k:{\\\"uid\\\":\\\"093d7650-2976-4827-9751-e30d4640124d\\\"}\": {}}}, \"f:rules\": {}}, \"manager\": \"OpenAPI-Generator\", \"operation\": \"Update\", \"time\": \"2022-06-29T15:11:09Z\"}], \"name\": \"awx\", \"namespace\": \"awx\", \"ownerReferences\": [{\"apiVersion\": \"awx.ansible.com/v1beta1\", \"kind\": \"AWX\", \"name\": \"awx\", \"uid\": \"093d7650-2976-4827-9751-e30d4640124d\"}], \"resourceVersion\": \"27644\", \"uid\": \"48d00219-b8c8-4fb6-90b0-b2de5a775945\"}, \"rules\": [{\"apiGroups\": [\"\"], \"resources\": [\"pods\"], \"verbs\": [\"get\", \"list\", \"watch\", \"create\", \"update\", \"patch\", \"delete\"]}, {\"apiGroups\": [\"\"], \"resources\": [\"pods/log\"], \"verbs\": [\"get\"]}, {\"apiGroups\": [\"\"], \"resources\": [\"pods/attach\"], \"verbs\": [\"create\"]}, {\"apiGroups\": [\"\"], \"resources\": [\"secrets\"], \"verbs\": [\"get\", \"create\", \"delete\"]}]}}, {\"changed\": false, \"duration\": 0, \"method\": \"apply\", \"result\": {\"apiVersion\": \"rbac.authorization.k8s.io/v1\", \"kind\": \"RoleBinding\", \"metadata\": {\"annotations\": {\"kubectl.kubernetes.io/last-applied-configuration\": \"{\\\"apiVersion\\\":\\\"rbac.authorization.k8s.io/v1\\\",\\\"kind\\\":\\\"RoleBinding\\\",\\\"metadata\\\":{\\\"name\\\":\\\"awx\\\",\\\"namespace\\\":\\\"awx\\\"},\\\"roleRef\\\":{\\\"apiGroup\\\":\\\"rbac.authorization.k8s.io\\\",\\\"kind\\\":\\\"Role\\\",\\\"name\\\":\\\"awx\\\"},\\\"subjects\\\":[{\\\"kind\\\":\\\"ServiceAccount\\\",\\\"name\\\":\\\"awx\\\"}]}\"}, \"creationTimestamp\": \"2022-06-29T15:11:10Z\", \"managedFields\": [{\"apiVersion\": \"rbac.authorization.k8s.io/v1\", \"fieldsType\": \"FieldsV1\", \"fieldsV1\": {\"f:metadata\": {\"f:annotations\": {\".\": {}, \"f:kubectl.kubernetes.io/last-applied-configuration\": {}}, \"f:ownerReferences\": {\".\": {}, \"k:{\\\"uid\\\":\\\"093d7650-2976-4827-9751-e30d4640124d\\\"}\": {}}}, \"f:roleRef\": {}, \"f:subjects\": {}}, \"manager\": \"OpenAPI-Generator\", \"operation\": \"Update\", \"time\": \"2022-06-29T15:11:09Z\"}], \"name\": \"awx\", \"namespace\": \"awx\", \"ownerReferences\": [{\"apiVersion\": \"awx.ansible.com/v1beta1\", \"kind\": \"AWX\", \"name\": \"awx\", \"uid\": \"093d7650-2976-4827-9751-e30d4640124d\"}], \"resourceVersion\": \"27645\", \"uid\": \"da54729b-20b1-4bce-87fe-efffb9fca7af\"}, \"roleRef\": {\"apiGroup\": \"rbac.authorization.k8s.io\", \"kind\": \"Role\", \"name\": \"awx\"}, \"subjects\": [{\"kind\": \"ServiceAccount\", \"name\": \"awx\"}]}}]}}\nok: [localhost] => (item=storage/persistent) => {\"ansible_loop_var\": \"item\", \"changed\": false, \"item\": \"storage/persistent\", \"result\": {\"results\": []}}\nok: [localhost] => (item=networking/service) => {\"ansible_loop_var\": \"item\", \"changed\": false, \"duration\": 0, \"item\": \"networking/service\", \"method\": \"apply\", \"result\": {\"apiVersion\": \"v1\", \"kind\": \"Service\", \"metadata\": {\"annotations\": {\"kubectl.kubernetes.io/last-applied-configuration\": \"{\\\"apiVersion\\\":\\\"v1\\\",\\\"kind\\\":\\\"Service\\\",\\\"metadata\\\":{\\\"labels\\\":{\\\"app.kubernetes.io/component\\\":\\\"awx\\\",\\\"app.kubernetes.io/managed-by\\\":\\\"awx-operator\\\",\\\"app.kubernetes.io/name\\\":\\\"awx\\\",\\\"app.kubernetes.io/operator-version\\\":\\\"0.29.0\\\",\\\"app.kubernetes.io/part-of\\\":\\\"awx\\\"},\\\"name\\\":\\\"awx-service\\\",\\\"namespace\\\":\\\"awx\\\"},\\\"spec\\\":{\\\"ports\\\":[{\\\"name\\\":\\\"http\\\",\\\"nodePort\\\":30080,\\\"port\\\":80,\\\"protocol\\\":\\\"TCP\\\",\\\"targetPort\\\":8052}],\\\"selector\\\":{\\\"app.kubernetes.io/component\\\":\\\"awx\\\",\\\"app.kubernetes.io/managed-by\\\":\\\"awx-operator\\\",\\\"app.kubernetes.io/name\\\":\\\"awx\\\"},\\\"type\\\":\\\"NodePort\\\"}}\"}, \"creationTimestamp\": \"2022-06-29T15:11:11Z\", \"labels\": {\"app.kubernetes.io/component\": \"awx\", \"app.kubernetes.io/managed-by\": \"awx-operator\", \"app.kubernetes.io/name\": \"awx\", \"app.kubernetes.io/operator-version\": \"0.29.0\", \"app.kubernetes.io/part-of\": \"awx\"}, \"managedFields\": [{\"apiVersion\": \"v1\", \"fieldsType\": \"FieldsV1\", \"fieldsV1\": {\"f:metadata\": {\"f:annotations\": {\".\": {}, \"f:kubectl.kubernetes.io/last-applied-configuration\": {}}, \"f:labels\": {\".\": {}, \"f:app.kubernetes.io/component\": {}, \"f:app.kubernetes.io/managed-by\": {}, \"f:app.kubernetes.io/name\": {}, \"f:app.kubernetes.io/operator-version\": {}, \"f:app.kubernetes.io/part-of\": {}}, \"f:ownerReferences\": {\".\": {}, \"k:{\\\"uid\\\":\\\"093d7650-2976-4827-9751-e30d4640124d\\\"}\": {}}}, \"f:spec\": {\"f:externalTrafficPolicy\": {}, \"f:internalTrafficPolicy\": {}, \"f:ports\": {\".\": {}, \"k:{\\\"port\\\":80,\\\"protocol\\\":\\\"TCP\\\"}\": {\".\": {}, \"f:name\": {}, \"f:nodePort\": {}, \"f:port\": {}, \"f:protocol\": {}, \"f:targetPort\": {}}}, \"f:selector\": {}, \"f:sessionAffinity\": {}, \"f:type\": {}}}, \"manager\": \"OpenAPI-Generator\", \"operation\": \"Update\", \"time\": \"2022-06-29T15:11:11Z\"}], \"name\": \"awx-service\", \"namespace\": \"awx\", \"ownerReferences\": [{\"apiVersion\": \"awx.ansible.com/v1beta1\", \"kind\": \"AWX\", \"name\": \"awx\", \"uid\": \"093d7650-2976-4827-9751-e30d4640124d\"}], \"resourceVersion\": \"34937475\", \"uid\": \"bc967b69-6cb9-453c-9ff2-5c7b4ae67e7e\"}, \"spec\": {\"clusterIP\": \"10.0.155.91\", \"clusterIPs\": [\"10.0.155.91\"], \"externalTrafficPolicy\": \"Cluster\", \"internalTrafficPolicy\": \"Cluster\", \"ipFamilies\": [\"IPv4\"], \"ipFamilyPolicy\": \"SingleStack\", \"ports\": [{\"name\": \"http\", \"nodePort\": 30080, \"port\": 80, \"protocol\": \"TCP\", \"targetPort\": 8052}], \"selector\": {\"app.kubernetes.io/component\": \"awx\", \"app.kubernetes.io/managed-by\": \"awx-operator\", \"app.kubernetes.io/name\": \"awx\"}, \"sessionAffinity\": \"None\", \"type\": \"NodePort\"}, \"status\": {\"loadBalancer\": {}}}}\nok: [localhost] => (item=networking/ingress) => {\"ansible_loop_var\": \"item\", \"changed\": false, \"item\": \"networking/ingress\", \"result\": {\"results\": []}}\n\r\nTASK [installer : Set default awx app image] ***********************************\r\ntask path: /opt/ansible/roles/installer/tasks/resources_configuration.yml:134\nok: [localhost] => {\"ansible_facts\": {\"_default_image\": \"quay.io/ansible/awx:21.6.0\"}, \"changed\": false}\n\r\nTASK [installer : Set user provided awx app image] *****************************\r\ntask path: /opt/ansible/roles/installer/tasks/resources_configuration.yml:138\nskipping: [localhost] => {\"changed\": false, \"skip_reason\": \"Conditional result was False\"}\n\r\nTASK [installer : Set AWX app image URL] ***************************************\r\ntask path: /opt/ansible/roles/installer/tasks/resources_configuration.yml:145\nok: [localhost] => {\"ansible_facts\": {\"_image\": \"quay.io/ansible/awx:21.6.0\"}, \"changed\": false}\n\r\nTASK [installer : Set default redis image] *************************************\r\ntask path: /opt/ansible/roles/installer/tasks/resources_configuration.yml:149\nok: [localhost] => {\"ansible_facts\": {\"_default_redis_image\": \"docker.io/redis:7\"}, \"changed\": false}\n\r\nTASK [installer : Set user provided redis image] *******************************\r\ntask path: /opt/ansible/roles/installer/tasks/resources_configuration.yml:153\nskipping: [localhost] => {\"changed\": false, \"skip_reason\": \"Conditional result was False\"}\n\r\nTASK [installer : Set Redis image URL] *****************************************\r\ntask path: /opt/ansible/roles/installer/tasks/resources_configuration.yml:160\nok: [localhost] => {\"ansible_facts\": {\"_redis_image\": \"docker.io/redis:7\"}, \"changed\": false}\n\r\nTASK [installer : Apply deployment resources] **********************************\r\ntask path: /opt/ansible/roles/installer/tasks/resources_configuration.yml:164\nfatal: [localhost]: FAILED! => {\"changed\": false, \"duration\": 120, \"method\": \"apply\", \"msg\": \"Deployment awx: Resource apply timed out\", \"result\": {\"apiVersion\": \"apps/v1\", \"kind\": \"Deployment\", \"metadata\": {\"annotations\": {\"deployment.kubernetes.io/revision\": \"3\", \"kubectl.kubernetes.io/last-applied-configuration\": \"{\\\"apiVersion\\\":\\\"apps/v1\\\",\\\"kind\\\":\\\"Deployment\\\",\\\"metadata\\\":{\\\"labels\\\":{\\\"app.kubernetes.io/component\\\":\\\"awx\\\",\\\"app.kubernetes.io/managed-by\\\":\\\"awx-operator\\\",\\\"app.kubernetes.io/name\\\":\\\"awx\\\",\\\"app.kubernetes.io/operator-version\\\":\\\"0.29.0\\\",\\\"app.kubernetes.io/part-of\\\":\\\"awx\\\",\\\"app.kubernetes.io/version\\\":\\\"21.6.0\\\"},\\\"name\\\":\\\"awx\\\",\\\"namespace\\\":\\\"awx\\\"},\\\"spec\\\":{\\\"replicas\\\":2,\\\"selector\\\":{\\\"matchLabels\\\":{\\\"app.kubernetes.io/component\\\":\\\"awx\\\",\\\"app.kubernetes.io/managed-by\\\":\\\"awx-operator\\\",\\\"app.kubernetes.io/name\\\":\\\"awx\\\"}},\\\"template\\\":{\\\"metadata\\\":{\\\"labels\\\":{\\\"app.kubernetes.io/component\\\":\\\"awx\\\",\\\"app.kubernetes.io/managed-by\\\":\\\"awx-operator\\\",\\\"app.kubernetes.io/name\\\":\\\"awx\\\",\\\"app.kubernetes.io/operator-version\\\":\\\"0.29.0\\\",\\\"app.kubernetes.io/part-of\\\":\\\"awx\\\",\\\"app.kubernetes.io/version\\\":\\\"21.6.0\\\"}},\\\"spec\\\":{\\\"containers\\\":[{\\\"args\\\":[\\\"redis-server\\\",\\\"/etc/redis.conf\\\"],\\\"image\\\":\\\"docker.io/redis:7\\\",\\\"imagePullPolicy\\\":\\\"IfNotPresent\\\",\\\"name\\\":\\\"redis\\\",\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"50m\\\",\\\"memory\\\":\\\"64Mi\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/etc/redis.conf\\\",\\\"name\\\":\\\"awx-redis-config\\\",\\\"readOnly\\\":true,\\\"subPath\\\":\\\"redis.conf\\\"},{\\\"mountPath\\\":\\\"/var/run/redis\\\",\\\"name\\\":\\\"awx-redis-socket\\\"},{\\\"mountPath\\\":\\\"/data\\\",\\\"name\\\":\\\"awx-redis-data\\\"}]},{\\\"args\\\":[\\\"/usr/bin/launch_awx.sh\\\"],\\\"env\\\":[{\\\"name\\\":\\\"MY_POD_NAMESPACE\\\",\\\"valueFrom\\\":{\\\"fieldRef\\\":{\\\"fieldPath\\\":\\\"metadata.namespace\\\"}}},{\\\"name\\\":\\\"UWSGI_MOUNT_PATH\\\",\\\"value\\\":\\\"/\\\"}],\\\"image\\\":\\\"quay.io/ansible/awx:21.6.0\\\",\\\"imagePullPolicy\\\":\\\"IfNotPresent\\\",\\\"name\\\":\\\"awx-web\\\",\\\"ports\\\":[{\\\"containerPort\\\":8052}],\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"128Mi\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/etc/tower/conf.d/execution_environments.py\\\",\\\"name\\\":\\\"awx-application-credentials\\\",\\\"readOnly\\\":true,\\\"subPath\\\":\\\"execution_environments.py\\\"},{\\\"mountPath\\\":\\\"/etc/tower/conf.d/credentials.py\\\",\\\"name\\\":\\\"awx-application-credentials\\\",\\\"readOnly\\\":true,\\\"subPath\\\":\\\"credentials.py\\\"},{\\\"mountPath\\\":\\\"/etc/tower/conf.d/ldap.py\\\",\\\"name\\\":\\\"awx-application-credentials\\\",\\\"readOnly\\\":true,\\\"subPath\\\":\\\"ldap.py\\\"},{\\\"mountPath\\\":\\\"/etc/tower/SECRET_KEY\\\",\\\"name\\\":\\\"awx-secret-key\\\",\\\"readOnly\\\":true,\\\"subPath\\\":\\\"SECRET_KEY\\\"},{\\\"mountPath\\\":\\\"/etc/tower/settings.py\\\",\\\"name\\\":\\\"awx-settings\\\",\\\"readOnly\\\":true,\\\"subPath\\\":\\\"settings.py\\\"},{\\\"mountPath\\\":\\\"/etc/nginx/nginx.conf\\\",\\\"name\\\":\\\"awx-nginx-conf\\\",\\\"readOnly\\\":true,\\\"subPath\\\":\\\"nginx.conf\\\"},{\\\"mountPath\\\":\\\"/var/run/redis\\\",\\\"name\\\":\\\"awx-redis-socket\\\"},{\\\"mountPath\\\":\\\"/var/run/supervisor\\\",\\\"name\\\":\\\"supervisor-socket\\\"},{\\\"mountPath\\\":\\\"/var/run/awx-rsyslog\\\",\\\"name\\\":\\\"rsyslog-socket\\\"},{\\\"mountPath\\\":\\\"/var/lib/awx/rsyslog\\\",\\\"name\\\":\\\"rsyslog-dir\\\"},{\\\"mountPath\\\":\\\"/var/lib/awx/projects\\\",\\\"name\\\":\\\"awx-projects\\\"},{\\\"mountPath\\\":\\\"/etc/receptor/signing/work-public-key.pem\\\",\\\"name\\\":\\\"awx-receptor-work-signing\\\",\\\"readOnly\\\":true,\\\"subPath\\\":\\\"work-public-key.pem\\\"},{\\\"mountPath\\\":\\\"/etc/receptor/tls/ca\\\",\\\"name\\\":\\\"awx-receptor-ca\\\",\\\"readOnly\\\":true}]},{\\\"args\\\":[\\\"/usr/bin/launch_awx_task.sh\\\"],\\\"env\\\":[{\\\"name\\\":\\\"SUPERVISOR_WEB_CONFIG_PATH\\\",\\\"value\\\":\\\"/etc/supervisord.conf\\\"},{\\\"name\\\":\\\"AWX_SKIP_MIGRATIONS\\\",\\\"value\\\":\\\"1\\\"},{\\\"name\\\":\\\"MY_POD_UID\\\",\\\"valueFrom\\\":{\\\"fieldRef\\\":{\\\"fieldPath\\\":\\\"metadata.uid\\\"}}},{\\\"name\\\":\\\"MY_POD_IP\\\",\\\"valueFrom\\\":{\\\"fieldRef\\\":{\\\"fieldPath\\\":\\\"status.podIP\\\"}}},{\\\"name\\\":\\\"MY_POD_NAMESPACE\\\",\\\"valueFrom\\\":{\\\"fieldRef\\\":{\\\"fieldPath\\\":\\\"metadata.namespace\\\"}}}],\\\"image\\\":\\\"quay.io/ansible/awx:21.6.0\\\",\\\"imagePullPolicy\\\":\\\"IfNotPresent\\\",\\\"name\\\":\\\"awx-task\\\",\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"128Mi\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/etc/tower/conf.d/execution_environments.py\\\",\\\"name\\\":\\\"awx-application-credentials\\\",\\\"readOnly\\\":true,\\\"subPath\\\":\\\"execution_environments.py\\\"},{\\\"mountPath\\\":\\\"/etc/tower/conf.d/credentials.py\\\",\\\"name\\\":\\\"awx-application-credentials\\\",\\\"readOnly\\\":true,\\\"subPath\\\":\\\"credentials.py\\\"},{\\\"mountPath\\\":\\\"/etc/tower/conf.d/ldap.py\\\",\\\"name\\\":\\\"awx-application-credentials\\\",\\\"readOnly\\\":true,\\\"subPath\\\":\\\"ldap.py\\\"},{\\\"mountPath\\\":\\\"/etc/tower/SECRET_KEY\\\",\\\"name\\\":\\\"awx-secret-key\\\",\\\"readOnly\\\":true,\\\"subPath\\\":\\\"SECRET_KEY\\\"},{\\\"mountPath\\\":\\\"/etc/tower/settings.py\\\",\\\"name\\\":\\\"awx-settings\\\",\\\"readOnly\\\":true,\\\"subPath\\\":\\\"settings.py\\\"},{\\\"mountPath\\\":\\\"/var/run/redis\\\",\\\"name\\\":\\\"awx-redis-socket\\\"},{\\\"mountPath\\\":\\\"/var/run/supervisor\\\",\\\"name\\\":\\\"supervisor-socket\\\"},{\\\"mountPath\\\":\\\"/var/run/awx-rsyslog\\\",\\\"name\\\":\\\"rsyslog-socket\\\"},{\\\"mountPath\\\":\\\"/var/lib/awx/rsyslog\\\",\\\"name\\\":\\\"rsyslog-dir\\\"},{\\\"mountPath\\\":\\\"/etc/receptor/\\\",\\\"name\\\":\\\"awx-receptor-config\\\"},{\\\"mountPath\\\":\\\"/etc/receptor/signing/work-private-key.pem\\\",\\\"name\\\":\\\"awx-receptor-work-signing\\\",\\\"readOnly\\\":true,\\\"subPath\\\":\\\"work-private-key.pem\\\"},{\\\"mountPath\\\":\\\"/var/run/receptor\\\",\\\"name\\\":\\\"receptor-socket\\\"},{\\\"mountPath\\\":\\\"/var/lib/awx/projects\\\",\\\"name\\\":\\\"awx-projects\\\"}]},{\\\"args\\\":[\\\"/bin/sh\\\",\\\"-c\\\",\\\"if [ ! -f /etc/receptor/receptor.conf ]; then\\\\n  cp /etc/receptor/receptor-default.conf /etc/receptor/receptor.conf\\\\n  sed -i \\\\\\\"s/HOSTNAME/$HOSTNAME/g\\\\\\\" /etc/receptor/receptor.conf\\\\nfi\\\\nexec receptor --config /etc/receptor/receptor.conf\\\\n\\\"],\\\"env\\\":null,\\\"image\\\":\\\"quay.io/ansible/awx-ee:latest\\\",\\\"imagePullPolicy\\\":\\\"IfNotPresent\\\",\\\"name\\\":\\\"awx-ee\\\",\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"64Mi\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/etc/receptor/receptor-default.conf\\\",\\\"name\\\":\\\"awx-default-receptor-config\\\",\\\"subPath\\\":\\\"receptor.conf\\\"},{\\\"mountPath\\\":\\\"/etc/receptor/\\\",\\\"name\\\":\\\"awx-receptor-config\\\"},{\\\"mountPath\\\":\\\"/etc/receptor/tls/ca/receptor-ca.crt\\\",\\\"name\\\":\\\"awx-receptor-ca\\\",\\\"readOnly\\\":true,\\\"subPath\\\":\\\"receptor-ca.crt\\\"},{\\\"mountPath\\\":\\\"/etc/receptor/signing/work-private-key.pem\\\",\\\"name\\\":\\\"awx-receptor-work-signing\\\",\\\"readOnly\\\":true,\\\"subPath\\\":\\\"work-private-key.pem\\\"},{\\\"mountPath\\\":\\\"/etc/receptor/tls/\\\",\\\"name\\\":\\\"awx-receptor-tls\\\"},{\\\"mountPath\\\":\\\"/var/run/receptor\\\",\\\"name\\\":\\\"receptor-socket\\\"},{\\\"mountPath\\\":\\\"/var/lib/awx/projects\\\",\\\"name\\\":\\\"awx-projects\\\"}]}],\\\"initContainers\\\":[{\\\"command\\\":[\\\"/bin/sh\\\",\\\"-c\\\",\\\"hostname=$MY_POD_NAME\\\\nreceptor --cert-makereq bits=2048 commonname=$hostname dnsname=$hostname nodeid=$hostname outreq=/etc/receptor/tls/receptor.req outkey=/etc/receptor/tls/receptor.key\\\\nreceptor --cert-signreq req=/etc/receptor/tls/receptor.req cacert=/etc/receptor/tls/ca/receptor-ca.crt cakey=/etc/receptor/tls/ca/receptor-ca.key outcert=/etc/receptor/tls/receptor.crt verify=yes\\\\n\\\"],\\\"env\\\":[{\\\"name\\\":\\\"MY_POD_NAME\\\",\\\"valueFrom\\\":{\\\"fieldRef\\\":{\\\"fieldPath\\\":\\\"metadata.name\\\"}}}],\\\"image\\\":\\\"quay.io/ansible/awx-ee:latest\\\",\\\"imagePullPolicy\\\":\\\"IfNotPresent\\\",\\\"name\\\":\\\"init\\\",\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/etc/receptor/tls/ca\\\",\\\"name\\\":\\\"awx-receptor-ca\\\",\\\"readOnly\\\":true},{\\\"mountPath\\\":\\\"/etc/receptor/tls/\\\",\\\"name\\\":\\\"awx-receptor-tls\\\"}]}],\\\"priorityClassName\\\":\\\"\\\",\\\"serviceAccountName\\\":\\\"awx\\\",\\\"volumes\\\":[{\\\"name\\\":\\\"awx-application-credentials\\\",\\\"secret\\\":{\\\"items\\\":[{\\\"key\\\":\\\"credentials.py\\\",\\\"path\\\":\\\"credentials.py\\\"},{\\\"key\\\":\\\"ldap.py\\\",\\\"path\\\":\\\"ldap.py\\\"},{\\\"key\\\":\\\"execution_environments.py\\\",\\\"path\\\":\\\"execution_environments.py\\\"}],\\\"secretName\\\":\\\"awx-app-credentials\\\"}},{\\\"emptyDir\\\":{},\\\"name\\\":\\\"awx-receptor-tls\\\"},{\\\"name\\\":\\\"awx-receptor-ca\\\",\\\"secret\\\":{\\\"secretName\\\":\\\"awx-receptor-ca\\\"}},{\\\"name\\\":\\\"awx-receptor-work-signing\\\",\\\"secret\\\":{\\\"secretName\\\":\\\"awx-receptor-work-signing\\\"}},{\\\"name\\\":\\\"awx-secret-key\\\",\\\"secret\\\":{\\\"items\\\":[{\\\"key\\\":\\\"secret_key\\\",\\\"path\\\":\\\"SECRET_KEY\\\"}],\\\"secretName\\\":\\\"awx-secret-key\\\"}},{\\\"configMap\\\":{\\\"items\\\":[{\\\"key\\\":\\\"settings\\\",\\\"path\\\":\\\"settings.py\\\"}],\\\"name\\\":\\\"awx-awx-configmap\\\"},\\\"name\\\":\\\"awx-settings\\\"},{\\\"configMap\\\":{\\\"items\\\":[{\\\"key\\\":\\\"nginx_conf\\\",\\\"path\\\":\\\"nginx.conf\\\"}],\\\"name\\\":\\\"awx-awx-configmap\\\"},\\\"name\\\":\\\"awx-nginx-conf\\\"},{\\\"configMap\\\":{\\\"items\\\":[{\\\"key\\\":\\\"redis_conf\\\",\\\"path\\\":\\\"redis.conf\\\"}],\\\"name\\\":\\\"awx-awx-configmap\\\"},\\\"name\\\":\\\"awx-redis-config\\\"},{\\\"emptyDir\\\":{},\\\"name\\\":\\\"awx-redis-socket\\\"},{\\\"emptyDir\\\":{},\\\"name\\\":\\\"awx-redis-data\\\"},{\\\"emptyDir\\\":{},\\\"name\\\":\\\"supervisor-socket\\\"},{\\\"emptyDir\\\":{},\\\"name\\\":\\\"rsyslog-socket\\\"},{\\\"emptyDir\\\":{},\\\"name\\\":\\\"receptor-socket\\\"},{\\\"emptyDir\\\":{},\\\"name\\\":\\\"rsyslog-dir\\\"},{\\\"emptyDir\\\":{},\\\"name\\\":\\\"awx-receptor-config\\\"},{\\\"configMap\\\":{\\\"items\\\":[{\\\"key\\\":\\\"receptor_conf\\\",\\\"path\\\":\\\"receptor.conf\\\"}],\\\"name\\\":\\\"awx-awx-configmap\\\"},\\\"name\\\":\\\"awx-default-receptor-config\\\"},{\\\"emptyDir\\\":{},\\\"name\\\":\\\"awx-projects\\\"}]}}}}\"}, \"creationTimestamp\": \"2022-06-29T15:11:13Z\", \"generation\": 3, \"labels\": {\"app.kubernetes.io/component\": \"awx\", \"app.kubernetes.io/managed-by\": \"awx-operator\", \"app.kubernetes.io/name\": \"awx\", \"app.kubernetes.io/operator-version\": \"0.29.0\", \"app.kubernetes.io/part-of\": \"awx\", \"app.kubernetes.io/version\": \"21.6.0\"}, \"managedFields\": [{\"apiVersion\": \"apps/v1\", \"fieldsType\": \"FieldsV1\", \"fieldsV1\": {\"f:metadata\": {\"f:annotations\": {\".\": {}, \"f:kubectl.kubernetes.io/last-applied-configuration\": {}}, \"f:labels\": {\".\": {}, \"f:app.kubernetes.io/component\": {}, \"f:app.kubernetes.io/managed-by\": {}, \"f:app.kubernetes.io/name\": {}, \"f:app.kubernetes.io/operator-version\": {}, \"f:app.kubernetes.io/part-of\": {}, \"f:app.kubernetes.io/version\": {}}, \"f:ownerReferences\": {\".\": {}, \"k:{\\\"uid\\\":\\\"093d7650-2976-4827-9751-e30d4640124d\\\"}\": {}}}, \"f:spec\": {\"f:progressDeadlineSeconds\": {}, \"f:replicas\": {}, \"f:revisionHistoryLimit\": {}, \"f:selector\": {}, \"f:strategy\": {\"f:rollingUpdate\": {\".\": {}, \"f:maxSurge\": {}, \"f:maxUnavailable\": {}}, \"f:type\": {}}, \"f:template\": {\"f:metadata\": {\"f:labels\": {\".\": {}, \"f:app.kubernetes.io/component\": {}, \"f:app.kubernetes.io/managed-by\": {}, \"f:app.kubernetes.io/name\": {}, \"f:app.kubernetes.io/operator-version\": {}, \"f:app.kubernetes.io/part-of\": {}, \"f:app.kubernetes.io/version\": {}}}, \"f:spec\": {\"f:containers\": {\"k:{\\\"name\\\":\\\"awx-ee\\\"}\": {\".\": {}, \"f:args\": {}, \"f:image\": {}, \"f:imagePullPolicy\": {}, \"f:name\": {}, \"f:resources\": {\".\": {}, \"f:requests\": {\".\": {}, \"f:cpu\": {}, \"f:memory\": {}}}, \"f:terminationMessagePath\": {}, \"f:terminationMessagePolicy\": {}, \"f:volumeMounts\": {\".\": {}, \"k:{\\\"mountPath\\\":\\\"/etc/receptor/\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}}, \"k:{\\\"mountPath\\\":\\\"/etc/receptor/receptor-default.conf\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}, \"f:subPath\": {}}, \"k:{\\\"mountPath\\\":\\\"/etc/receptor/signing/work-private-key.pem\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}, \"f:readOnly\": {}, \"f:subPath\": {}}, \"k:{\\\"mountPath\\\":\\\"/etc/receptor/tls/\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}}, \"k:{\\\"mountPath\\\":\\\"/etc/receptor/tls/ca/receptor-ca.crt\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}, \"f:readOnly\": {}, \"f:subPath\": {}}, \"k:{\\\"mountPath\\\":\\\"/var/lib/awx/projects\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}}, \"k:{\\\"mountPath\\\":\\\"/var/run/receptor\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}}}}, \"k:{\\\"name\\\":\\\"awx-task\\\"}\": {\".\": {}, \"f:args\": {}, \"f:env\": {\".\": {}, \"k:{\\\"name\\\":\\\"AWX_SKIP_MIGRATIONS\\\"}\": {\".\": {}, \"f:name\": {}, \"f:value\": {}}, \"k:{\\\"name\\\":\\\"MY_POD_IP\\\"}\": {\".\": {}, \"f:name\": {}, \"f:valueFrom\": {\".\": {}, \"f:fieldRef\": {}}}, \"k:{\\\"name\\\":\\\"MY_POD_NAMESPACE\\\"}\": {\".\": {}, \"f:name\": {}, \"f:valueFrom\": {\".\": {}, \"f:fieldRef\": {}}}, \"k:{\\\"name\\\":\\\"MY_POD_UID\\\"}\": {\".\": {}, \"f:name\": {}, \"f:valueFrom\": {\".\": {}, \"f:fieldRef\": {}}}, \"k:{\\\"name\\\":\\\"SUPERVISOR_WEB_CONFIG_PATH\\\"}\": {\".\": {}, \"f:name\": {}, \"f:value\": {}}}, \"f:image\": {}, \"f:imagePullPolicy\": {}, \"f:name\": {}, \"f:resources\": {\".\": {}, \"f:requests\": {\".\": {}, \"f:cpu\": {}, \"f:memory\": {}}}, \"f:terminationMessagePath\": {}, \"f:terminationMessagePolicy\": {}, \"f:volumeMounts\": {\".\": {}, \"k:{\\\"mountPath\\\":\\\"/etc/receptor/\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}}, \"k:{\\\"mountPath\\\":\\\"/etc/receptor/signing/work-private-key.pem\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}, \"f:readOnly\": {}, \"f:subPath\": {}}, \"k:{\\\"mountPath\\\":\\\"/etc/tower/SECRET_KEY\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}, \"f:readOnly\": {}, \"f:subPath\": {}}, \"k:{\\\"mountPath\\\":\\\"/etc/tower/conf.d/credentials.py\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}, \"f:readOnly\": {}, \"f:subPath\": {}}, \"k:{\\\"mountPath\\\":\\\"/etc/tower/conf.d/execution_environments.py\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}, \"f:readOnly\": {}, \"f:subPath\": {}}, \"k:{\\\"mountPath\\\":\\\"/etc/tower/conf.d/ldap.py\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}, \"f:readOnly\": {}, \"f:subPath\": {}}, \"k:{\\\"mountPath\\\":\\\"/etc/tower/settings.py\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}, \"f:readOnly\": {}, \"f:subPath\": {}}, \"k:{\\\"mountPath\\\":\\\"/var/lib/awx/projects\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}}, \"k:{\\\"mountPath\\\":\\\"/var/lib/awx/rsyslog\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}}, \"k:{\\\"mountPath\\\":\\\"/var/run/awx-rsyslog\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}}, \"k:{\\\"mountPath\\\":\\\"/var/run/receptor\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}}, \"k:{\\\"mountPath\\\":\\\"/var/run/redis\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}}, \"k:{\\\"mountPath\\\":\\\"/var/run/supervisor\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}}}}, \"k:{\\\"name\\\":\\\"awx-web\\\"}\": {\".\": {}, \"f:args\": {}, \"f:env\": {\".\": {}, \"k:{\\\"name\\\":\\\"MY_POD_NAMESPACE\\\"}\": {\".\": {}, \"f:name\": {}, \"f:valueFrom\": {\".\": {}, \"f:fieldRef\": {}}}, \"k:{\\\"name\\\":\\\"UWSGI_MOUNT_PATH\\\"}\": {\".\": {}, \"f:name\": {}, \"f:value\": {}}}, \"f:image\": {}, \"f:imagePullPolicy\": {}, \"f:name\": {}, \"f:ports\": {\".\": {}, \"k:{\\\"containerPort\\\":8052,\\\"protocol\\\":\\\"TCP\\\"}\": {\".\": {}, \"f:containerPort\": {}, \"f:protocol\": {}}}, \"f:resources\": {\".\": {}, \"f:requests\": {\".\": {}, \"f:cpu\": {}, \"f:memory\": {}}}, \"f:terminationMessagePath\": {}, \"f:terminationMessagePolicy\": {}, \"f:volumeMounts\": {\".\": {}, \"k:{\\\"mountPath\\\":\\\"/etc/nginx/nginx.conf\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}, \"f:readOnly\": {}, \"f:subPath\": {}}, \"k:{\\\"mountPath\\\":\\\"/etc/receptor/signing/work-public-key.pem\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}, \"f:readOnly\": {}, \"f:subPath\": {}}, \"k:{\\\"mountPath\\\":\\\"/etc/receptor/tls/ca\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}, \"f:readOnly\": {}}, \"k:{\\\"mountPath\\\":\\\"/etc/tower/SECRET_KEY\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}, \"f:readOnly\": {}, \"f:subPath\": {}}, \"k:{\\\"mountPath\\\":\\\"/etc/tower/conf.d/credentials.py\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}, \"f:readOnly\": {}, \"f:subPath\": {}}, \"k:{\\\"mountPath\\\":\\\"/etc/tower/conf.d/execution_environments.py\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}, \"f:readOnly\": {}, \"f:subPath\": {}}, \"k:{\\\"mountPath\\\":\\\"/etc/tower/conf.d/ldap.py\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}, \"f:readOnly\": {}, \"f:subPath\": {}}, \"k:{\\\"mountPath\\\":\\\"/etc/tower/settings.py\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}, \"f:readOnly\": {}, \"f:subPath\": {}}, \"k:{\\\"mountPath\\\":\\\"/var/lib/awx/projects\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}}, \"k:{\\\"mountPath\\\":\\\"/var/lib/awx/rsyslog\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}}, \"k:{\\\"mountPath\\\":\\\"/var/run/awx-rsyslog\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}}, \"k:{\\\"mountPath\\\":\\\"/var/run/redis\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}}, \"k:{\\\"mountPath\\\":\\\"/var/run/supervisor\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}}}}, \"k:{\\\"name\\\":\\\"redis\\\"}\": {\".\": {}, \"f:args\": {}, \"f:image\": {}, \"f:imagePullPolicy\": {}, \"f:name\": {}, \"f:resources\": {\".\": {}, \"f:requests\": {\".\": {}, \"f:cpu\": {}, \"f:memory\": {}}}, \"f:terminationMessagePath\": {}, \"f:terminationMessagePolicy\": {}, \"f:volumeMounts\": {\".\": {}, \"k:{\\\"mountPath\\\":\\\"/data\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}}, \"k:{\\\"mountPath\\\":\\\"/etc/redis.conf\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}, \"f:readOnly\": {}, \"f:subPath\": {}}, \"k:{\\\"mountPath\\\":\\\"/var/run/redis\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}}}}}, \"f:dnsPolicy\": {}, \"f:initContainers\": {\".\": {}, \"k:{\\\"name\\\":\\\"init\\\"}\": {\".\": {}, \"f:command\": {}, \"f:env\": {\".\": {}, \"k:{\\\"name\\\":\\\"MY_POD_NAME\\\"}\": {\".\": {}, \"f:name\": {}, \"f:valueFrom\": {\".\": {}, \"f:fieldRef\": {}}}}, \"f:image\": {}, \"f:imagePullPolicy\": {}, \"f:name\": {}, \"f:resources\": {}, \"f:terminationMessagePath\": {}, \"f:terminationMessagePolicy\": {}, \"f:volumeMounts\": {\".\": {}, \"k:{\\\"mountPath\\\":\\\"/etc/receptor/tls/\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}}, \"k:{\\\"mountPath\\\":\\\"/etc/receptor/tls/ca\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}, \"f:readOnly\": {}}}}}, \"f:restartPolicy\": {}, \"f:schedulerName\": {}, \"f:securityContext\": {}, \"f:serviceAccount\": {}, \"f:serviceAccountName\": {}, \"f:terminationGracePeriodSeconds\": {}, \"f:volumes\": {\".\": {}, \"k:{\\\"name\\\":\\\"awx-application-credentials\\\"}\": {\".\": {}, \"f:name\": {}, \"f:secret\": {\".\": {}, \"f:defaultMode\": {}, \"f:items\": {}, \"f:secretName\": {}}}, \"k:{\\\"name\\\":\\\"awx-default-receptor-config\\\"}\": {\".\": {}, \"f:configMap\": {\".\": {}, \"f:defaultMode\": {}, \"f:items\": {}, \"f:name\": {}}, \"f:name\": {}}, \"k:{\\\"name\\\":\\\"awx-nginx-conf\\\"}\": {\".\": {}, \"f:configMap\": {\".\": {}, \"f:defaultMode\": {}, \"f:items\": {}, \"f:name\": {}}, \"f:name\": {}}, \"k:{\\\"name\\\":\\\"awx-projects\\\"}\": {\".\": {}, \"f:emptyDir\": {}, \"f:name\": {}}, \"k:{\\\"name\\\":\\\"awx-receptor-ca\\\"}\": {\".\": {}, \"f:name\": {}, \"f:secret\": {\".\": {}, \"f:defaultMode\": {}, \"f:secretName\": {}}}, \"k:{\\\"name\\\":\\\"awx-receptor-config\\\"}\": {\".\": {}, \"f:emptyDir\": {}, \"f:name\": {}}, \"k:{\\\"name\\\":\\\"awx-receptor-tls\\\"}\": {\".\": {}, \"f:emptyDir\": {}, \"f:name\": {}}, \"k:{\\\"name\\\":\\\"awx-receptor-work-signing\\\"}\": {\".\": {}, \"f:name\": {}, \"f:secret\": {\".\": {}, \"f:defaultMode\": {}, \"f:secretName\": {}}}, \"k:{\\\"name\\\":\\\"awx-redis-config\\\"}\": {\".\": {}, \"f:configMap\": {\".\": {}, \"f:defaultMode\": {}, \"f:items\": {}, \"f:name\": {}}, \"f:name\": {}}, \"k:{\\\"name\\\":\\\"awx-redis-data\\\"}\": {\".\": {}, \"f:emptyDir\": {}, \"f:name\": {}}, \"k:{\\\"name\\\":\\\"awx-redis-socket\\\"}\": {\".\": {}, \"f:emptyDir\": {}, \"f:name\": {}}, \"k:{\\\"name\\\":\\\"awx-secret-key\\\"}\": {\".\": {}, \"f:name\": {}, \"f:secret\": {\".\": {}, \"f:defaultMode\": {}, \"f:items\": {}, \"f:secretName\": {}}}, \"k:{\\\"name\\\":\\\"awx-settings\\\"}\": {\".\": {}, \"f:configMap\": {\".\": {}, \"f:defaultMode\": {}, \"f:items\": {}, \"f:name\": {}}, \"f:name\": {}}, \"k:{\\\"name\\\":\\\"receptor-socket\\\"}\": {\".\": {}, \"f:emptyDir\": {}, \"f:name\": {}}, \"k:{\\\"name\\\":\\\"rsyslog-dir\\\"}\": {\".\": {}, \"f:emptyDir\": {}, \"f:name\": {}}, \"k:{\\\"name\\\":\\\"rsyslog-socket\\\"}\": {\".\": {}, \"f:emptyDir\": {}, \"f:name\": {}}, \"k:{\\\"name\\\":\\\"supervisor-socket\\\"}\": {\".\": {}, \"f:emptyDir\": {}, \"f:name\": {}}}}}}}, \"manager\": \"OpenAPI-Generator\", \"operation\": \"Update\", \"time\": \"2022-09-29T08:05:29Z\"}, {\"apiVersion\": \"apps/v1\", \"fieldsType\": \"FieldsV1\", \"fieldsV1\": {\"f:metadata\": {\"f:annotations\": {\"f:deployment.kubernetes.io/revision\": {}}}, \"f:status\": {\"f:availableReplicas\": {}, \"f:conditions\": {\".\": {}, \"k:{\\\"type\\\":\\\"Available\\\"}\": {\".\": {}, \"f:lastTransitionTime\": {}, \"f:lastUpdateTime\": {}, \"f:message\": {}, \"f:reason\": {}, \"f:status\": {}, \"f:type\": {}}, \"k:{\\\"type\\\":\\\"Progressing\\\"}\": {\".\": {}, \"f:lastTransitionTime\": {}, \"f:lastUpdateTime\": {}, \"f:message\": {}, \"f:reason\": {}, \"f:status\": {}, \"f:type\": {}}}, \"f:observedGeneration\": {}, \"f:readyReplicas\": {}, \"f:replicas\": {}, \"f:unavailableReplicas\": {}, \"f:updatedReplicas\": {}}}, \"manager\": \"kube-controller-manager\", \"operation\": \"Update\", \"subresource\": \"status\", \"time\": \"2022-10-13T08:18:58Z\"}], \"name\": \"awx\", \"namespace\": \"awx\", \"ownerReferences\": [{\"apiVersion\": \"awx.ansible.com/v1beta1\", \"kind\": \"AWX\", \"name\": \"awx\", \"uid\": \"093d7650-2976-4827-9751-e30d4640124d\"}], \"resourceVersion\": \"40763918\", \"uid\": \"a12f3282-dd9e-4bba-80ed-236344f915d3\"}, \"spec\": {\"progressDeadlineSeconds\": 600, \"replicas\": 2, \"revisionHistoryLimit\": 10, \"selector\": {\"matchLabels\": {\"app.kubernetes.io/component\": \"awx\", \"app.kubernetes.io/managed-by\": \"awx-operator\", \"app.kubernetes.io/name\": \"awx\"}}, \"strategy\": {\"rollingUpdate\": {\"maxSurge\": \"25%\", \"maxUnavailable\": \"25%\"}, \"type\": \"RollingUpdate\"}, \"template\": {\"metadata\": {\"creationTimestamp\": null, \"labels\": {\"app.kubernetes.io/component\": \"awx\", \"app.kubernetes.io/managed-by\": \"awx-operator\", \"app.kubernetes.io/name\": \"awx\", \"app.kubernetes.io/operator-version\": \"0.29.0\", \"app.kubernetes.io/part-of\": \"awx\", \"app.kubernetes.io/version\": \"21.6.0\"}}, \"spec\": {\"containers\": [{\"args\": [\"redis-server\", \"/etc/redis.conf\"], \"image\": \"docker.io/redis:7\", \"imagePullPolicy\": \"IfNotPresent\", \"name\": \"redis\", \"resources\": {\"requests\": {\"cpu\": \"50m\", \"memory\": \"64Mi\"}}, \"terminationMessagePath\": \"/dev/termination-log\", \"terminationMessagePolicy\": \"File\", \"volumeMounts\": [{\"mountPath\": \"/etc/redis.conf\", \"name\": \"awx-redis-config\", \"readOnly\": true, \"subPath\": \"redis.conf\"}, {\"mountPath\": \"/var/run/redis\", \"name\": \"awx-redis-socket\"}, {\"mountPath\": \"/data\", \"name\": \"awx-redis-data\"}]}, {\"args\": [\"/usr/bin/launch_awx.sh\"], \"env\": [{\"name\": \"MY_POD_NAMESPACE\", \"valueFrom\": {\"fieldRef\": {\"apiVersion\": \"v1\", \"fieldPath\": \"metadata.namespace\"}}}, {\"name\": \"UWSGI_MOUNT_PATH\", \"value\": \"/\"}], \"image\": \"quay.io/ansible/awx:21.6.0\", \"imagePullPolicy\": \"IfNotPresent\", \"name\": \"awx-web\", \"ports\": [{\"containerPort\": 8052, \"protocol\": \"TCP\"}], \"resources\": {\"requests\": {\"cpu\": \"100m\", \"memory\": \"128Mi\"}}, \"terminationMessagePath\": \"/dev/termination-log\", \"terminationMessagePolicy\": \"File\", \"volumeMounts\": [{\"mountPath\": \"/etc/tower/conf.d/execution_environments.py\", \"name\": \"awx-application-credentials\", \"readOnly\": true, \"subPath\": \"execution_environments.py\"}, {\"mountPath\": \"/etc/tower/conf.d/credentials.py\", \"name\": \"awx-application-credentials\", \"readOnly\": true, \"subPath\": \"credentials.py\"}, {\"mountPath\": \"/etc/tower/conf.d/ldap.py\", \"name\": \"awx-application-credentials\", \"readOnly\": true, \"subPath\": \"ldap.py\"}, {\"mountPath\": \"/etc/tower/SECRET_KEY\", \"name\": \"awx-secret-key\", \"readOnly\": true, \"subPath\": \"SECRET_KEY\"}, {\"mountPath\": \"/etc/tower/settings.py\", \"name\": \"awx-settings\", \"readOnly\": true, \"subPath\": \"settings.py\"}, {\"mountPath\": \"/etc/nginx/nginx.conf\", \"name\": \"awx-nginx-conf\", \"readOnly\": true, \"subPath\": \"nginx.conf\"}, {\"mountPath\": \"/var/run/redis\", \"name\": \"awx-redis-socket\"}, {\"mountPath\": \"/var/run/supervisor\", \"name\": \"supervisor-socket\"}, {\"mountPath\": \"/var/run/awx-rsyslog\", \"name\": \"rsyslog-socket\"}, {\"mountPath\": \"/var/lib/awx/rsyslog\", \"name\": \"rsyslog-dir\"}, {\"mountPath\": \"/var/lib/awx/projects\", \"name\": \"awx-projects\"}, {\"mountPath\": \"/etc/receptor/signing/work-public-key.pem\", \"name\": \"awx-receptor-work-signing\", \"readOnly\": true, \"subPath\": \"work-public-key.pem\"}, {\"mountPath\": \"/etc/receptor/tls/ca\", \"name\": \"awx-receptor-ca\", \"readOnly\": true}]}, {\"args\": [\"/usr/bin/launch_awx_task.sh\"], \"env\": [{\"name\": \"SUPERVISOR_WEB_CONFIG_PATH\", \"value\": \"/etc/supervisord.conf\"}, {\"name\": \"AWX_SKIP_MIGRATIONS\", \"value\": \"1\"}, {\"name\": \"MY_POD_UID\", \"valueFrom\": {\"fieldRef\": {\"apiVersion\": \"v1\", \"fieldPath\": \"metadata.uid\"}}}, {\"name\": \"MY_POD_IP\", \"valueFrom\": {\"fieldRef\": {\"apiVersion\": \"v1\", \"fieldPath\": \"status.podIP\"}}}, {\"name\": \"MY_POD_NAMESPACE\", \"valueFrom\": {\"fieldRef\": {\"apiVersion\": \"v1\", \"fieldPath\": \"metadata.namespace\"}}}], \"image\": \"quay.io/ansible/awx:21.6.0\", \"imagePullPolicy\": \"IfNotPresent\", \"name\": \"awx-task\", \"resources\": {\"requests\": {\"cpu\": \"100m\", \"memory\": \"128Mi\"}}, \"terminationMessagePath\": \"/dev/termination-log\", \"terminationMessagePolicy\": \"File\", \"volumeMounts\": [{\"mountPath\": \"/etc/tower/conf.d/execution_environments.py\", \"name\": \"awx-application-credentials\", \"readOnly\": true, \"subPath\": \"execution_environments.py\"}, {\"mountPath\": \"/etc/tower/conf.d/credentials.py\", \"name\": \"awx-application-credentials\", \"readOnly\": true, \"subPath\": \"credentials.py\"}, {\"mountPath\": \"/etc/tower/conf.d/ldap.py\", \"name\": \"awx-application-credentials\", \"readOnly\": true, \"subPath\": \"ldap.py\"}, {\"mountPath\": \"/etc/tower/SECRET_KEY\", \"name\": \"awx-secret-key\", \"readOnly\": true, \"subPath\": \"SECRET_KEY\"}, {\"mountPath\": \"/etc/tower/settings.py\", \"name\": \"awx-settings\", \"readOnly\": true, \"subPath\": \"settings.py\"}, {\"mountPath\": \"/var/run/redis\", \"name\": \"awx-redis-socket\"}, {\"mountPath\": \"/var/run/supervisor\", \"name\": \"supervisor-socket\"}, {\"mountPath\": \"/var/run/awx-rsyslog\", \"name\": \"rsyslog-socket\"}, {\"mountPath\": \"/var/lib/awx/rsyslog\", \"name\": \"rsyslog-dir\"}, {\"mountPath\": \"/etc/receptor/\", \"name\": \"awx-receptor-config\"}, {\"mountPath\": \"/etc/receptor/signing/work-private-key.pem\", \"name\": \"awx-receptor-work-signing\", \"readOnly\": true, \"subPath\": \"work-private-key.pem\"}, {\"mountPath\": \"/var/run/receptor\", \"name\": \"receptor-socket\"}, {\"mountPath\": \"/var/lib/awx/projects\", \"name\": \"awx-projects\"}]}, {\"args\": [\"/bin/sh\", \"-c\", \"if [ ! -f /etc/receptor/receptor.conf ]; then\\n  cp /etc/receptor/receptor-default.conf /etc/receptor/receptor.conf\\n  sed -i \\\"s/HOSTNAME/$HOSTNAME/g\\\" /etc/receptor/receptor.conf\\nfi\\nexec receptor --config /etc/receptor/receptor.conf\\n\"], \"image\": \"quay.io/ansible/awx-ee:latest\", \"imagePullPolicy\": \"IfNotPresent\", \"name\": \"awx-ee\", \"resources\": {\"requests\": {\"cpu\": \"100m\", \"memory\": \"64Mi\"}}, \"terminationMessagePath\": \"/dev/termination-log\", \"terminationMessagePolicy\": \"File\", \"volumeMounts\": [{\"mountPath\": \"/etc/receptor/receptor-default.conf\", \"name\": \"awx-default-receptor-config\", \"subPath\": \"receptor.conf\"}, {\"mountPath\": \"/etc/receptor/\", \"name\": \"awx-receptor-config\"}, {\"mountPath\": \"/etc/receptor/tls/ca/receptor-ca.crt\", \"name\": \"awx-receptor-ca\", \"readOnly\": true, \"subPath\": \"receptor-ca.crt\"}, {\"mountPath\": \"/etc/receptor/signing/work-private-key.pem\", \"name\": \"awx-receptor-work-signing\", \"readOnly\": true, \"subPath\": \"work-private-key.pem\"}, {\"mountPath\": \"/etc/receptor/tls/\", \"name\": \"awx-receptor-tls\"}, {\"mountPath\": \"/var/run/receptor\", \"name\": \"receptor-socket\"}, {\"mountPath\": \"/var/lib/awx/projects\", \"name\": \"awx-projects\"}]}], \"dnsPolicy\": \"ClusterFirst\", \"initContainers\": [{\"command\": [\"/bin/sh\", \"-c\", \"hostname=$MY_POD_NAME\\nreceptor --cert-makereq bits=2048 commonname=$hostname dnsname=$hostname nodeid=$hostname outreq=/etc/receptor/tls/receptor.req outkey=/etc/receptor/tls/receptor.key\\nreceptor --cert-signreq req=/etc/receptor/tls/receptor.req cacert=/etc/receptor/tls/ca/receptor-ca.crt cakey=/etc/receptor/tls/ca/receptor-ca.key outcert=/etc/receptor/tls/receptor.crt verify=yes\\n\"], \"env\": [{\"name\": \"MY_POD_NAME\", \"valueFrom\": {\"fieldRef\": {\"apiVersion\": \"v1\", \"fieldPath\": \"metadata.name\"}}}], \"image\": \"quay.io/ansible/awx-ee:latest\", \"imagePullPolicy\": \"IfNotPresent\", \"name\": \"init\", \"resources\": {}, \"terminationMessagePath\": \"/dev/termination-log\", \"terminationMessagePolicy\": \"File\", \"volumeMounts\": [{\"mountPath\": \"/etc/receptor/tls/ca\", \"name\": \"awx-receptor-ca\", \"readOnly\": true}, {\"mountPath\": \"/etc/receptor/tls/\", \"name\": \"awx-receptor-tls\"}]}], \"restartPolicy\": \"Always\", \"schedulerName\": \"default-scheduler\", \"securityContext\": {}, \"serviceAccount\": \"awx\", \"serviceAccountName\": \"awx\", \"terminationGracePeriodSeconds\": 30, \"volumes\": [{\"name\": \"awx-application-credentials\", \"secret\": {\"defaultMode\": 420, \"items\": [{\"key\": \"credentials.py\", \"path\": \"credentials.py\"}, {\"key\": \"ldap.py\", \"path\": \"ldap.py\"}, {\"key\": \"execution_environments.py\", \"path\": \"execution_environments.py\"}], \"secretName\": \"awx-app-credentials\"}}, {\"emptyDir\": {}, \"name\": \"awx-receptor-tls\"}, {\"name\": \"awx-receptor-ca\", \"secret\": {\"defaultMode\": 420, \"secretName\": \"awx-receptor-ca\"}}, {\"name\": \"awx-receptor-work-signing\", \"secret\": {\"defaultMode\": 420, \"secretName\": \"awx-receptor-work-signing\"}}, {\"name\": \"awx-secret-key\", \"secret\": {\"defaultMode\": 420, \"items\": [{\"key\": \"secret_key\", \"path\": \"SECRET_KEY\"}], \"secretName\": \"awx-secret-key\"}}, {\"configMap\": {\"defaultMode\": 420, \"items\": [{\"key\": \"settings\", \"path\": \"settings.py\"}], \"name\": \"awx-awx-configmap\"}, \"name\": \"awx-settings\"}, {\"configMap\": {\"defaultMode\": 420, \"items\": [{\"key\": \"nginx_conf\", \"path\": \"nginx.conf\"}], \"name\": \"awx-awx-configmap\"}, \"name\": \"awx-nginx-conf\"}, {\"configMap\": {\"defaultMode\": 420, \"items\": [{\"key\": \"redis_conf\", \"path\": \"redis.conf\"}], \"name\": \"awx-awx-configmap\"}, \"name\": \"awx-redis-config\"}, {\"emptyDir\": {}, \"name\": \"awx-redis-socket\"}, {\"emptyDir\": {}, \"name\": \"awx-redis-data\"}, {\"emptyDir\": {}, \"name\": \"supervisor-socket\"}, {\"emptyDir\": {}, \"name\": \"rsyslog-socket\"}, {\"emptyDir\": {}, \"name\": \"receptor-socket\"}, {\"emptyDir\": {}, \"name\": \"rsyslog-dir\"}, {\"emptyDir\": {}, \"name\": \"awx-receptor-config\"}, {\"configMap\": {\"defaultMode\": 420, \"items\": [{\"key\": \"receptor_conf\", \"path\": \"receptor.conf\"}], \"name\": \"awx-awx-configmap\"}, \"name\": \"awx-default-receptor-config\"}, {\"emptyDir\": {}, \"name\": \"awx-projects\"}]}}}, \"status\": {\"availableReplicas\": 1, \"conditions\": [{\"lastTransitionTime\": \"2022-06-29T15:11:13Z\", \"lastUpdateTime\": \"2022-09-29T08:06:16Z\", \"message\": \"ReplicaSet \\\"awx-7566768857\\\" has successfully progressed.\", \"reason\": \"NewReplicaSetAvailable\", \"status\": \"True\", \"type\": \"Progressing\"}, {\"lastTransitionTime\": \"2022-10-13T08:18:58Z\", \"lastUpdateTime\": \"2022-10-13T08:18:58Z\", \"message\": \"Deployment does not have minimum availability.\", \"reason\": \"MinimumReplicasUnavailable\", \"status\": \"False\", \"type\": \"Available\"}], \"observedGeneration\": 3, \"readyReplicas\": 1, \"replicas\": 2, \"unavailableReplicas\": 1, \"updatedReplicas\": 2}}}\n\r\nPLAY RECAP *********************************************************************\r\nlocalhost                  : ok=50   changed=0    unreachable=0    failed=1    skipped=51   rescued=0    ignored=0   \r\n\n","job":"2815119228376918080","name":"awx","namespace":"awx","error":"exit status 2"}

--
You received this message because you are subscribed to a topic in the Google Groups "AWX Project" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/awx-project/wB0dbH3xUcI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to awx-project...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/1abf685f-0c57-4375-a3a7-ec5fdf673b58n%40googlegroups.com.

[AWX Project]

this looks relevant "Deployment does not have minimum availability" -- how many cpu / memory does the cluster have access to?

Are you setting any custom resource requests/limits for the containers, or just using defaults?

Does your deployment work with replicas set to 1?

[Elia Curti]

The cluster is dedicated to AWX with 3 nodes (4cpu 14GB) and I'm using defaults.

My deployments works with 2 replicas and only one has the issue reported.
The strange thing is that after about 20 restarts it starts working on its own again. 

I don't understand why sometimes an awx pod crashes like this, with no load on the cluster.

Thank you for your help,

Elia

[AWX Project]

Have you made any progress on this? We have not run across this error but having to restart 20 times to get it working sounds really frustrating.

-The AWX Team

[Elia Curti]

No progress but the problem has not occurred since the last report.

However, the pod restarted on its own until it worked again.
I don't know what to think, it doesn't happen often.

Thank you for your help.

Elia