[docker-compose] awx-rsyslogd errored

[jean-christophe manciot]

Environment:

- Ubuntu 20.04 focal LTS

- docker-ce: 5:24.0.2-1~ubuntu.20.04~focal (deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable)

- docker-compose-plugin: 2.18.1-1~ubuntu.20.04~focal (deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable)

- python3.9: 3.9.5-3ubuntu0~20.04.1

- ansible 8.0.0 (from Pypi)

- setuptools-scm: 7.1.0  (from Pypi)

Workflow:

        su - awx
        git clone -b 22.3.0 https://github.com/ansible/awx.git
        cd awx
        git switch -c release_4.4

        . /opt/awx/venv/bin/activate
        make docker-compose-build

        make docker-compose

leads to:

        ...

        tools_awx_1       | 2023-06-19 16:28:05,695 ERROR    [-] awx.main.utils.reload supervisorctl restart awx-rsyslogd errored with exit code `7`, stdout:
        tools_awx_1       | b'tower-processes:awx-rsyslogd: stopped\ntower-processes:awx-rsyslogd: ERROR (spawn error)'stderr:

        ...

This issue is not new and has been addressed in the past with a quick fix by pinning an older rsyslog version.

Unfortunately, this old version 8.2102.0-106.el9 is not available anymore on Centos 9 stream:

        [root@localhost ~]# dnf --showduplicates list available rsyslog
        CentOS Stream 9 - BaseOS                                                                                                                      14 kB/s | 9.5 kB     00:00    
        CentOS Stream 9 - AppStream                                                                                                                   26 kB/s |  10 kB     00:00    
        CentOS Stream 9 - Extras packages                                                                                                             92 kB/s |  11 kB     00:00    
        Docker CE Stable - x86_64                                                                                                                     53 kB/s | 3.5 kB     00:00    
        Available Packages
        rsyslog.x86_64                                                                   8.2102.0-107.el9                                                                   appstream
        rsyslog.x86_64                                                                   8.2102.0-109.el9                                                                   appstream
        rsyslog.x86_64                                                                   8.2102.0-111.el9                                                                   appstream
        rsyslog.x86_64                                                                   8.2102.0-113.el9                                                                   appstream
        rsyslog.x86_64                                                                   8.2102.0-115.el9                                                                   appstream

I tried to pin other versions:

- 8.2102.0-107.el9

- 8.2102.0-109.el9

- 8.2102.0-115.el9

but they all lead to the same issue.

Is there a way to modify the docker file to include another official repo to pull the right rpm from?

N.B: attached logs

1. docker-compose-build.log

2. docker-compose.log

[jean-christophe manciot]

I'm baffled by the complete lack of comments on that matter: unless I'm missing something, it is now impossible to install any new AWX using docker-compose on any platform, unless a solution is found.

[AWX Project]

Hi,

Apologies for the delay here. This was an issue with the image builds which we have since fixed. Are you able to try your steps again with the latest? 

Thanks,

AWX Team

[jean-christophe manciot]

Thanks for your reply.

I'll try on Monday.

[jean-christophe manciot]

In the same environment, after deleting all docker containers and images, and checking out on the latest commit (6b381aa79e14613b3595b05af725c1a4cc4b6e0d):

        su - awx
        git clone https://github.com/ansible/awx.git

        cd awx
        git switch -c release_4.4

        . /opt/awx/venv/bin/activate
        make docker-compose-build

        make docker-compose &

leads to rsyslog-8.2102.0-106.el9 being pulled from copr.fedorainfracloud.org:ansible:Rsyslog and:

        $ docker ps

        IMAGE                                   COMMAND                  CREATED          STATUS          PORTS                 NAMES
        ghcr.io/ansible/awx_devel:release_4.4   "/entrypoint.sh laun…"   14 minutes ago   Up 14 minutes   0.0.0.0:2222...       tools_awx_1
        postgres:12                             "docker-entrypoint.s…"   14 minutes ago   Up 14 minutes   5432/tcp              tools_postgres_1
        redis:latest                            "redis-server /usr/l…"   14 minutes ago   Up 14 minutes   6379/tcp              tools_redis_1

and:

        $ cat supervisord.log
        2023-06-26 11:05:13,496 INFO RPC interface 'supervisor' initialized
        2023-06-26 11:05:13,496 CRIT Server 'unix_http_server' running without any HTTP authentication checking
        2023-06-26 11:05:13,497 INFO supervisord started with pid 250
        2023-06-26 11:05:14,499 INFO spawned: 'stdout' with pid 251
        2023-06-26 11:05:14,500 INFO spawned: 'superwatcher' with pid 252
        2023-06-26 11:05:14,501 INFO spawned: 'awx-dispatcher' with pid 253
        2023-06-26 11:05:14,502 INFO spawned: 'awx-receiver' with pid 254
        2023-06-26 11:05:14,502 INFO spawned: 'awx-uwsgi' with pid 255
        2023-06-26 11:05:14,503 INFO spawned: 'awx-daphne' with pid 256
        2023-06-26 11:05:14,504 INFO spawned: 'awx-nginx' with pid 257
        2023-06-26 11:05:14,505 INFO spawned: 'awx-wsrelay' with pid 258
        2023-06-26 11:05:14,506 INFO spawned: 'awx-rsyslogd' with pid 259
        2023-06-26 11:05:14,507 INFO spawned: 'awx-ws-heartbeat' with pid 260
        2023-06-26 11:05:14,508 INFO spawned: 'awx-rsyslog-configurer' with pid 261
        2023-06-26 11:05:14,509 INFO spawned: 'awx-cache-clear' with pid 262
        2023-06-26 11:05:14,515 INFO spawned: 'awx-autoreload' with pid 267
        2023-06-26 11:05:14,516 INFO spawned: 'awx-receptor' with pid 268
        2023-06-26 11:05:15,774 INFO success: stdout entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
        2023-06-26 11:05:15,775 INFO success: superwatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
        2023-06-26 11:05:15,775 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
        2023-06-26 11:05:15,775 INFO success: awx-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
        2023-06-26 11:05:15,775 INFO success: awx-uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
        2023-06-26 11:05:15,775 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
        2023-06-26 11:05:15,775 INFO success: awx-nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
        2023-06-26 11:05:15,775 INFO success: awx-wsrelay entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
        2023-06-26 11:05:15,775 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
        2023-06-26 11:05:15,775 INFO success: awx-ws-heartbeat entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
        2023-06-26 11:05:15,775 INFO success: awx-rsyslog-configurer entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
        2023-06-26 11:05:15,775 INFO success: awx-cache-clear entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
        2023-06-26 11:05:15,775 INFO success: awx-autoreload entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
        2023-06-26 11:05:15,775 INFO success: awx-receptor entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
        2023-06-26 11:05:18,373 INFO waiting for awx-rsyslogd to stop
        2023-06-26 11:05:18,374 INFO stopped: awx-rsyslogd (exit status 0)
        2023-06-26 11:05:18,376 INFO spawned: 'awx-rsyslogd' with pid 508
        2023-06-26 11:05:19,616 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)