Container AWX-web crash

[JS G.]

Hello, 

awx-operatior : 0.24.0

awx : 19.3.0

plateform : k3s

CPU : 4G

MEM : 8G

I have a problem with awx-web container (Back-off restarting failed container) : 

awx-web container logs : 

[wait-for-migrations] Waiting for database migrations...
[wait-for-migrations] Attempt 1 of 30
2022-07-19 11:57:53,777 INFO RPC interface 'supervisor' initialized
2022-07-19 11:57:53,777 INFO RPC interface 'supervisor' initialized
2022-07-19 11:57:53,777 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2022-07-19 11:57:53,777 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2022-07-19 11:57:53,777 INFO supervisord started with pid 7
2022-07-19 11:57:53,777 INFO supervisord started with pid 7
2022-07-19 11:57:54,778 INFO spawned: 'superwatcher' with pid 26
2022-07-19 11:57:54,778 INFO spawned: 'superwatcher' with pid 26
2022-07-19 11:57:54,781 INFO spawned: 'nginx' with pid 27
2022-07-19 11:57:54,781 INFO spawned: 'nginx' with pid 27
2022-07-19 11:57:54,782 INFO spawned: 'uwsgi' with pid 28
2022-07-19 11:57:54,782 INFO spawned: 'uwsgi' with pid 28
2022-07-19 11:57:54,784 INFO spawned: 'daphne' with pid 29
2022-07-19 11:57:54,784 INFO spawned: 'daphne' with pid 29
2022-07-19 11:57:54,786 INFO spawned: 'wsbroadcast' with pid 30
2022-07-19 11:57:54,786 INFO spawned: 'wsbroadcast' with pid 30
2022-07-19 11:57:54,789 INFO spawned: 'awx-rsyslogd' with pid 31
2022-07-19 11:57:54,789 INFO spawned: 'awx-rsyslogd' with pid 31
READY
[uWSGI] getting INI configuration from /etc/tower/uwsgi.ini
*** Starting uWSGI 2.0.18 (64bit) on [Tue Jul 19 11:57:54 2022] ***
compiled with version: 11.3.1 20220421 (Red Hat 11.3.1-2) on 12 July 2022 16:57:29
os: Linux-4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022
nodename: awx-6cfb9cdcfb-pkddj
machine: x86_64
clock source: unix
detected number of CPU cores: 4
current working directory: /var/lib/awx
detected binary path: /var/lib/awx/venv/awx/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 127.0.0.1:8050 fd 3
Python version: 3.9.13 (main, Jun  9 2022, 00:00:00)  [GCC 11.3.1 20220421 (Red Hat 11.3.1-2)]
2022/07/19 11:57:54 [emerg] 27#27: socket() [::]:8052 failed (97: Address family not supported by protocol)
nginx: [emerg] socket() [::]:8052 failed (97: Address family not supported by protocol)
2022-07-19 11:57:54,795 INFO exited: nginx (exit status 1; not expected)
2022-07-19 11:57:54,795 INFO exited: nginx (exit status 1; not expected)
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x12ac460
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 609552 bytes (595 KB) for 5 cores
*** Operational MODE: preforking ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 28)
spawned uWSGI worker 1 (pid: 32, cores: 1)
spawned uWSGI worker 2 (pid: 33, cores: 1)
spawned uWSGI worker 3 (pid: 34, cores: 1)
spawned uWSGI worker 4 (pid: 35, cores: 1)
spawned uWSGI worker 5 (pid: 36, cores: 1)
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
2022-07-19 11:57:55,855 INFO success: superwatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-07-19 11:57:55,855 INFO success: superwatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-07-19 11:57:55,856 INFO spawned: 'nginx' with pid 37
2022-07-19 11:57:55,856 INFO spawned: 'nginx' with pid 37
2022/07/19 11:57:55 [emerg] 37#37: socket() [::]:8052 failed (97: Address family not supported by protocol)
nginx: [emerg] socket() [::]:8052 failed (97: Address family not supported by protocol)
2022-07-19 11:57:55,917 INFO exited: nginx (exit status 1; not expected)
2022-07-19 11:57:55,917 INFO exited: nginx (exit status 1; not expected)
WSGI app 0 (mountpoint='/') ready in 2 seconds on interpreter 0x12ac460 pid: 33 (default app)
WSGI app 0 (mountpoint='/') ready in 2 seconds on interpreter 0x12ac460 pid: 32 (default app)
WSGI app 0 (mountpoint='/') ready in 3 seconds on interpreter 0x12ac460 pid: 35 (default app)
2022-07-19 11:57:57,273 INFO     [-] daphne.cli Starting server at tcp:port=8051:interface=127.0.0.1
2022-07-19 11:57:57,273 INFO     Starting server at tcp:port=8051:interface=127.0.0.1
2022-07-19 11:57:57,273 INFO     [-] daphne.server HTTP/2 support not enabled (install the http2 and tls Twisted extras)
2022-07-19 11:57:57,273 INFO     HTTP/2 support not enabled (install the http2 and tls Twisted extras)
2022-07-19 11:57:57,273 INFO     [-] daphne.server Configuring endpoint tcp:port=8051:interface=127.0.0.1
2022-07-19 11:57:57,273 INFO     Configuring endpoint tcp:port=8051:interface=127.0.0.1
2022-07-19 11:57:57,274 INFO     [-] daphne.server Listening on TCP address 127.0.0.1:8051
2022-07-19 11:57:57,274 INFO     Listening on TCP address 127.0.0.1:8051
WSGI app 0 (mountpoint='/') ready in 3 seconds on interpreter 0x12ac460 pid: 36 (default app)
WSGI app 0 (mountpoint='/') ready in 3 seconds on interpreter 0x12ac460 pid: 34 (default app)
2022-07-19 11:57:57,937 INFO     [-] awx.main.wsbroadcast Active instance with hostname awx-6cfb9cdcfb-pkddj is registered.
2022-07-19 11:57:57,939 INFO spawned: 'nginx' with pid 52
2022-07-19 11:57:57,939 INFO spawned: 'nginx' with pid 52
2022/07/19 11:57:57 [emerg] 52#52: socket() [::]:8052 failed (97: Address family not supported by protocol)
nginx: [emerg] socket() [::]:8052 failed (97: Address family not supported by protocol)
2022-07-19 11:57:57,959 INFO exited: nginx (exit status 1; not expected)
2022-07-19 11:57:57,959 INFO exited: nginx (exit status 1; not expected)
2022-07-19 11:58:00,963 INFO spawned: 'nginx' with pid 54
2022-07-19 11:58:00,963 INFO spawned: 'nginx' with pid 54
2022/07/19 11:58:00 [emerg] 54#54: socket() [::]:8052 failed (97: Address family not supported by protocol)
nginx: [emerg] socket() [::]:8052 failed (97: Address family not supported by protocol)
2022-07-19 11:58:00,973 INFO exited: nginx (exit status 1; not expected)
2022-07-19 11:58:00,973 INFO exited: nginx (exit status 1; not expected)
2022-07-19 11:58:01,974 INFO gave up: nginx entered FATAL state, too many start retries too quickly
2022-07-19 11:58:01,974 INFO gave up: nginx entered FATAL state, too many start retries too quickly
Processing Event: ver:3.0 server:supervisor serial:0 pool:superwatcher poolserial:0 eventname:PROCESS_STATE_FATAL len:62
2022-07-19 11:58:02,975 WARN received SIGQUIT indicating exit request
2022-07-19 11:58:02,975 WARN received SIGQUIT indicating exit request
2022-07-19 11:58:02,976 INFO waiting for superwatcher, uwsgi, daphne, wsbroadcast, awx-rsyslogd to die
2022-07-19 11:58:02,976 INFO waiting for superwatcher, uwsgi, daphne, wsbroadcast, awx-rsyslogd to die
...brutally killing workers...
2022-07-19 11:58:02,983 INFO     [-] daphne.server Killed 0 pending application instances
2022-07-19 11:58:02,983 INFO     Killed 0 pending application instances
2022-07-19 11:58:03,010 INFO stopped: awx-rsyslogd (exit status 0)
2022-07-19 11:58:03,010 INFO stopped: awx-rsyslogd (exit status 0)
2022-07-19 11:58:03,694 INFO stopped: daphne (exit status 0)
2022-07-19 11:58:03,694 INFO stopped: daphne (exit status 0)
2022-07-19 11:58:03,694 INFO stopped: wsbroadcast (terminated by SIGTERM)
2022-07-19 11:58:03,694 INFO stopped: wsbroadcast (terminated by SIGTERM)
worker 1 buried after 1 seconds
worker 2 buried after 1 seconds
worker 3 buried after 1 seconds
worker 4 buried after 1 seconds
worker 5 buried after 1 seconds
binary reloading uWSGI...
chdir() to /var/lib/awx
closing all non-uwsgi socket fds > 2 (max_fd = 1048576)...
found fd 3 mapped to socket 0 (127.0.0.1:8050)
running /var/lib/awx/venv/awx/bin/uwsgi
[uWSGI] getting INI configuration from /etc/tower/uwsgi.ini
*** Starting uWSGI 2.0.18 (64bit) on [Tue Jul 19 11:58:04 2022] ***
compiled with version: 11.3.1 20220421 (Red Hat 11.3.1-2) on 12 July 2022 16:57:29
os: Linux-4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022
nodename: awx-6cfb9cdcfb-pkddj
machine: x86_64
clock source: unix
detected number of CPU cores: 4
current working directory: /var/lib/awx
detected binary path: /var/lib/awx/venv/awx/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 inherited INET address 127.0.0.1:8050 fd 3
Python version: 3.9.13 (main, Jun  9 2022, 00:00:00)  [GCC 11.3.1 20220421 (Red Hat 11.3.1-2)]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x1d11d70
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 609552 bytes (595 KB) for 5 cores
*** Operational MODE: preforking ***
*** uWSGI is running in multiple interpreter mode ***
gracefully (RE)spawned uWSGI master process (pid: 28)
spawned uWSGI worker 1 (pid: 55, cores: 1)
spawned uWSGI worker 2 (pid: 56, cores: 1)
spawned uWSGI worker 3 (pid: 57, cores: 1)
spawned uWSGI worker 4 (pid: 58, cores: 1)
spawned uWSGI worker 5 (pid: 59, cores: 1)
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
2022-07-19 11:58:06,693 INFO waiting for superwatcher, uwsgi to die
2022-07-19 11:58:06,693 INFO waiting for superwatcher, uwsgi to die
WSGI app 0 (mountpoint='/') ready in 2 seconds on interpreter 0x1d11d70 pid: 55 (default app)
WSGI app 0 (mountpoint='/') ready in 2 seconds on interpreter 0x1d11d70 pid: 56 (default app)
WSGI app 0 (mountpoint='/') ready in 2 seconds on interpreter 0x1d11d70 pid: 57 (default app)
WSGI app 0 (mountpoint='/') ready in 2 seconds on interpreter 0x1d11d70 pid: 58 (default app)
WSGI app 0 (mountpoint='/') ready in 2 seconds on interpreter 0x1d11d70 pid: 59 (default app)
2022-07-19 11:58:10,646 INFO waiting for superwatcher, uwsgi to die
2022-07-19 11:58:10,646 INFO waiting for superwatcher, uwsgi to die
2022-07-19 11:58:13,649 WARN killing 'uwsgi' (28) with SIGKILL
2022-07-19 11:58:13,649 WARN killing 'uwsgi' (28) with SIGKILL
2022-07-19 11:58:13,649 INFO waiting for superwatcher, uwsgi to die
2022-07-19 11:58:13,649 INFO waiting for superwatcher, uwsgi to die
2022-07-19 11:58:13,665 INFO stopped: uwsgi (terminated by SIGKILL)
2022-07-19 11:58:13,665 INFO stopped: uwsgi (terminated by SIGKILL)
2022-07-19 11:58:13,665 INFO stopped: superwatcher (terminated by SIGTERM)
2022-07-19 11:58:13,665 INFO stopped: superwatcher (terminated by SIGTERM)

I don't undertand what awx-web container crash 

Can you help me ?

[kurokobo]

Hi,

Nginx has been exited because [::]:8052 can't be used.

> 2022/07/19 11:57:54 [emerg] 27#27: socket() [::]:8052 failed (97: Address family not supported by protocol)

> nginx: [emerg] socket() [::]:8052 failed (97: Address family not supported by protocol)

> 2022-07-19 11:57:54,795 INFO exited: nginx (exit status 1; not expected)

> 2022-07-19 11:57:54,795 INFO exited: nginx (exit status 1; not expected)

I think IPv6 on your K3s host seems to be completely disabled.

The awx-web container that deployed via Operator 0.24.0 or newer uses IPv6 in addition to IPv4 by default,

so could you please enable IPv6 on your K3s host, or try 0.23.0 or earlier instead, to determine if it is an IPv6 related issue?

------- Original Message -------

--
You received this message because you are subscribed to the Google Groups "AWX Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to awx-project...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/fc68cb18-acab-403b-85ed-28c164186f63n%40googlegroups.com.

[JS G.]

Hello, 

I have found the problem of this issue, IPV6 must be enable on k3s host, nginx in awx-web container has need ipv6 ....

Regards, 

[JS G.]

Thank's for your answer :)

Our answers crossed, I had come to the same conclusion.

Regards,