awx deployment failed in k3s cluster

[Ahilan Swaminathan]

Hi Team

When I try to deploy the awx in my k3s cluster, awx pod not started and failed to start and got "CrashLoopBackOff". when I describe the pod I got the event as below.

Events:

  Type     Reason     Age                    From               Message

  ----     ------     ----                   ----               -------

  Normal   Scheduled  39m                    default-scheduler  Successfully assigned awx/awx-78ccbd4b9b-r7lsc to xl8phxc360awxtst

  Normal   Pulled     39m                    kubelet            Container image "quay.io/ansible/awx-ee:latest" already present on machine

  Normal   Created    39m                    kubelet            Created container init

  Normal   Started    39m                    kubelet            Started container init

  Normal   Pulled     39m                    kubelet            Container image "docker.io/redis:7" already present on machine

  Normal   Created    39m                    kubelet            Created container redis

  Normal   Started    39m                    kubelet            Started container redis

  Normal   Pulled     39m                    kubelet            Container image "quay.io/ansible/awx:21.8.0" already present on machine

  Normal   Created    39m                    kubelet            Created container awx-task

  Normal   Started    39m                    kubelet            Started container awx-task

  Normal   Pulled     39m                    kubelet            Container image "quay.io/ansible/awx-ee:latest" already present on machine

  Normal   Created    39m                    kubelet            Created container awx-ee

  Normal   Started    39m                    kubelet            Started container awx-ee

  Normal   Created    38m (x3 over 39m)      kubelet            Created container awx-web

  Normal   Started    38m (x3 over 39m)      kubelet            Started container awx-web

  Normal   Pulled     37m (x4 over 39m)      kubelet            Container image "quay.io/ansible/awx:21.8.0" already present on machine

  Warning  BackOff    4m40s (x139 over 38m)  kubelet            Back-off restarting failed container

My awx status as below!!!

[kubeadmin@xl8phxc360awxtst ~]$ kubectl -n awx get awx,all,ingress,secrets

NAME                      AGE

awx.awx.ansible.com/awx   81m

NAME                                                   READY   STATUS             RESTARTS         AGE

pod/awx-operator-controller-manager-585df5d96d-hk2xs   2/2     Running            0                94m

pod/awx-postgres-13-0                                  1/1     Running            0                47m

pod/awx-78ccbd4b9b-r7lsc                               3/4     CrashLoopBackOff   12 (4m55s ago)   47m

NAME                                                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE

service/awx-operator-controller-manager-metrics-service   ClusterIP   xx.xx.xx.xx   <none>        8443/TCP   94m

service/awx-postgres-13                                   ClusterIP   None            <none>        5432/TCP   80m

service/awx-service                                       ClusterIP   xx.xx.xx.xx   <none>        80/TCP     78m

NAME                                              READY   UP-TO-DATE   AVAILABLE   AGE

deployment.apps/awx-operator-controller-manager   1/1     1            1           94m

deployment.apps/awx                               0/1     1            0           78m

NAME                                                         DESIRED   CURRENT   READY   AGE

replicaset.apps/awx-operator-controller-manager-585df5d96d   1         1         1       94m

replicaset.apps/awx-78ccbd4b9b                               1         1         0       78m

NAME                               READY   AGE

statefulset.apps/awx-postgres-13   1/1     80m

NAME                                    CLASS     HOSTS                                ADDRESS        PORTS     AGE

ingress.networking.k8s.io/awx-ingress   traefik   xl8phxc360awxtst.ipengine360s1.net   xx.xx.xx.xx   80, 443   78m

NAME                                  TYPE                DATA   AGE

secret/awx-admin-password             Opaque              1      81m

secret/awx-postgres-configuration     Opaque              6      81m

secret/awx-secret-tls                 kubernetes.io/tls   2      81m

secret/redhat-operators-pull-secret   Opaque              1      81m

secret/awx-receptor-ca                kubernetes.io/tls   2      78m

secret/awx-receptor-work-signing      Opaque              2      78m

secret/awx-app-credentials            Opaque              3      78m

secret/awx-secret-key                 Opaque              1      80m

secret/awx-broadcast-websocket        Opaque              1      80m

[kubeadmin@xl8phxc360awxtst ~]$

Please suggest and hel;p me to resolve the issue

Thanks and Regards
S.Ahilan
Lead Analyst

CGI Software Solutions and consultant Pvt Ltd

DLF IT Park

Chennai 600089
+91-9486332230

[AWX Project]

sounds like it could be due to insufficient resources to start the container. Does bumping available resources help get the pod started?

take a look at this guide https://komodor.com/learn/how-to-fix-crashloopbackoff-kubernetes-error/

AWX Team

[Ahilan Swaminathan]

Hi Team

We have 12GB free RAM and our server having 8vcpu and we have enough free space in all filesystem including root. Not sure where/what we bump in resources. 

--
You received this message because you are subscribed to the Google Groups "AWX Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to awx-project...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/d4e0d361-f967-4e82-b2ec-c5a48445f668n%40googlegroups.com.

[Wei-Yen Tan]

Please describe the pod so we can see further errors

Sent from Outlook for iOS

---

From: awx-p...@googlegroups.com <awx-p...@googlegroups.com> on behalf of Ahilan Swaminathan <ahil...@gmail.com>
Sent: Tuesday, December 13, 2022 6:51:37 PM
To: awx-p...@googlegroups.com <awx-p...@googlegroups.com>
Cc: Ahilan Swaminathan <ahil...@gmail.com>; brookman.a...@cgi.com <brookman.a...@cgi.com>; brok...@gmail.com <brok...@gmail.com>
Subject: Re: [awx-project] Re: awx deployment failed in k3s cluster

 

To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/CAL6yCN3z7BhndGsKOg7eBzbfQViy9otFd%2BV2gpXgu1dQAK%3DV7w%40mail.gmail.com.

[Ahilan Swaminathan]

Hi Yen

Here attached output of pod describe and container status

[AWX Project]

Hi were you able to resolve this issue? If possible can you provide logging details emitted from the awx-web container (if it starts at all)? you can use tool like stern to aggregate all logs emitted from all pods in the awx namespace. With additional info we can probably help figure out the root cause to the crash backoff.

Thanks!

AWX Team