awx k8 - Postgres pod deploy failed getting CreateContainerConfigError

[Vish]

Hello all,

I'm trying to install a new and latest install of AWX in K8, but the postgres pod never gets created, it errored out saying : CreateContainerConfigError. Below are some command outputs.

Please let me know what is causing the problem.I do see pv and pvc bound also i have created /data/postgres and /data/projects. I have changed permission on those dire as well

root@k8-ctrlr:/awx-operator# chmod 755 /data/postgres

root@k8-ctrlr:/awx-operator# chown 1000:0 /data/projects/

root@k8-ctrlr:/awx-operator# ls -la /data/

drwxr-xr-x  2 root   root 4096 Feb  5 06:44 postgres

drwxr-xr-x  2 ubuntu root 4096 Feb  5 06:44 projects

#############################

kubectl get pods -o wide -n awx
NAME                                               READY   STATUS                       RESTARTS   AGE   IP           NODE        NOMINATED NODE   READINESS GATES
awx-operator-controller-manager-77c67cb7c6-jkzsx   2/2     Running                      0          29m   10.244.2.9   k8-node-2   <none>           <none>
awx-postgres-13-0                                  0/1     CreateContainerConfigError   0          15m   10.244.1.8   k8-node-1   <none>           <none>

kubectl describe pod awx-postgres-13-0 -n awx

Events:

  Type     Reason            Age                From               Message

  ----     ------            ----               ----               -------

  Warning  FailedScheduling  37s                default-scheduler  0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod..

  Normal   Scheduled         35s                default-scheduler  Successfully assigned awx/awx-postgres-13-0 to k8-node-1

  Normal   Pulled            10s (x4 over 35s)  kubelet            Container image "postgres:13" already present on machine

  Warning  Failed            10s (x4 over 35s)  kubelet            Error: stat /data/postgres: no such file or directory

I did create /project/postgres directories , more info in below installation steps

PV and PVC status: i see pv and pvc are bound

root@k8-ctrlr:/awx-deploy# kubectl get pv -n awx
NAME                  CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                               STORAGECLASS          REASON   AGE
awx-postgres-volume   15Gi       RWO            Retain           Bound    awx/postgres-13-awx-postgres-13-0   awx-postgres-volume            5m54s
awx-projects-volume   15Gi       RWO            Retain           Bound    awx/awx-projects-claim              awx-projects-volume            5m54s

root@k8-ctrlr:/awx-deploy# kubectl get pvc -n awx
NAME                            STATUS   VOLUME                CAPACITY   ACCESS MODES   STORAGECLASS          AGE
awx-projects-claim              Bound    awx-projects-volume   15Gi       RWO            awx-projects-volume   5m51s
postgres-13-awx-postgres-13-0   Bound    awx-postgres-volume   15Gi       RWO            awx-postgres-volume   5m30s
root@k8-ctrlr:/awx-deploy# 

root@k8-ctrlr:/awx-deploy# kubectl get pods -A
NAMESPACE      NAME                                               READY   STATUS                       RESTARTS   AGE
awx            awx-operator-controller-manager-77c67cb7c6-jkzsx   2/2     Running                      0          26m
awx            awx-postgres-13-0                                  0/1     CreateContainerConfigError   0          12m

Thanks

Vish

[kurokobo]

Hi,

Since you are on multi-noded cluster, you should use some sort of shared storage for your PV (like iSCSI, NFS, Ceph, Longhorn, GlusterFS...).

The hostPath based PV that you're using does not magically allow you to share your local directory between nodes, it only mounts that directory when the Pod is scheduled on exactly that node.

I mean, even if you create /data/postgres on k8-ctrlr, it cannot be referenced and mounted from k8-node-1/2/3.

https://kubernetes.io/docs/concepts/storage/volumes/#hostpath

https://kubernetes.io/docs/concepts/storage/persistent-volumes/#types-of-persistent-volumes

Creating /data/postgres on all nodes of k8-node-1/2/3 will work, but its contents (files inside the directory) will never be synchronized between nodes, so the data that PSQL will read will change depending on the node where the pod is launched. So hostPath based PV is impractical on multi-noded cluster unless the node where the pod runs are explicitly fixed.

Regards,

--

@kurokobo

------- Original Message -------

--
You received this message because you are subscribed to the Google Groups "AWX Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to awx-project...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/CAKuFsSQBeEaQsvEQAR4QAvTw3OscWCPu0DsDWc%3Daa4SheQ%3D5rA%40mail.gmail.com.

[Vish]

Thank you Kurokobo. creating /data/** in worker nodes fixed the container issue, but I will go ahead and change to NFS as my cluster is multinode.

Thanks

Vish

To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/BN-iCx-OPP-hgjkWKx7pA1yu7X9VNFMfwy3kOjMLJeBQxj9AYrQzv67INtEOmu9zL5sEWaFAxnPdI_ZPd4gLb5Czy1g4_fracp_ay6r-guE%3D%40protonmail.com.