AWX 19 kubernetes
479 views
Skip to first unread message
Tony
Sep 16, 2021, 12:54:01 PM9/16/21
to AWX Project
I've just checked the pod status on my kubenetes cluster and seeing two pods in pending status. Bit unsure why. Everything appears to be working correctly from an AWX point of view.
#kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system local-path-provisioner-5ff76fc89d-2wjpx 1/1 Running 0 13d
kube-system metrics-server-86cbb8457f-wfg85 1/1 Running 0 13d
kube-system coredns-7448499f4d-4p2fv 1/1 Running 0 13d
kube-system helm-install-traefik-crd-jfp9h 0/1 Completed 0 13d
kube-system helm-install-traefik-pkhxv 0/1 Completed 1 13d
kube-system svclb-traefik-xdzzr 2/2 Running 0 13d
kube-system traefik-97b44b794-g9mbs 1/1 Running 0 13d
default awx-operator-69c646c48f-lnsmq 1/1 Running 0 13d
awx awx-postgres-0 1/1 Running 0 13d
awx awx-59ff55b5b-4m8wv 4/4 Running 0 13d
default awx-postgres-0 0/1 Pending 0 11s
default awx-59ff55b5b-h86gm 0/4 Pending 0 4s
#kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
awx-operator 1/1 1 1 13d
awx 0/1 1 0 8d
Any ideas anyone?
Thank you
Tony
Sep 16, 2021, 1:00:11 PM9/16/21
to AWX Project
Just to add further info, I am seeing the below errors on the pods in pending state. I am unsure why this is happening now and not for the last 13 days or so?
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 9m41s default-scheduler 0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
Warning FailedScheduling 9m39s default-scheduler 0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
Cnu k
Sep 16, 2021, 2:14:25 PM9/16/21
to AWX Project
Check PV and PVCs status if you're using static storage provisioning such as NFS or similar. Either way the error is related to PVC that's not bound to a PV, so checking that should help.
Tony
Sep 16, 2021, 2:51:51 PM9/16/21
to AWX Project
Thanks
I'm still none the wiser.
Here are my yaml files
pv.yaml
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: awx-postgres-volume
spec:
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
capacity:
storage: 2Gi
storageClassName: awx-postgres-volume
hostPath:
path: /data/postgres
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: awx-projects-volume
spec:
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
capacity:
storage: 2Gi
storageClassName: awx-projects-volume
hostPath:
path: /data/projects
pvc.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: awx-projects-claim
spec:
accessModes:
- ReadWriteOnce
volumeMode: Filesystem
resources:
requests:
storage: 2Gi
storageClassName: awx-projects-volume
kubectl describe pvs
Name: awx-projects-claim
Namespace: default
StorageClass: awx-projects-volume
Status: Pending
Volume:
Labels: <none>
Annotations: <none>
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: awx-59ff55b5b-h86gm
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ProvisioningFailed 4m9s (x46567 over 8d) persistentvolume-controller storageclass.storage.k8s.io "awx-projects-volume" not found
Name: postgres-awx-postgres-0
Namespace: default
StorageClass: awx-postgres-volume
Status: Pending
Volume:
Annotations: <none>
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: awx-postgres-0
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ProvisioningFailed 3m54s (x46567 over 8d) persistentvolume-controller storageclass.storage.k8s.io "awx-postgres-volume" not found
kubectl describe pv
Name: awx-projects-volume
Labels: <none>
Annotations: pv.kubernetes.io/bound-by-controller: yes
Finalizers: [kubernetes.io/pv-protection]
StorageClass: awx-projects-volume
Status: Bound
Claim: awx/awx-projects-claim
Reclaim Policy: Retain
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 2Gi
Node Affinity: <none>
Message:
Source:
Type: HostPath (bare host directory volume)
Path: /data/projects
HostPathType:
Events: <none>
Name: awx-postgres-volume
Labels: <none>
Annotations: pv.kubernetes.io/bound-by-controller: yes
Finalizers: [kubernetes.io/pv-protection]
StorageClass: awx-postgres-volume
Status: Bound
Claim: awx/postgres-awx-postgres-0
Reclaim Policy: Retain
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 2Gi
Node Affinity: <none>
Message:
Source:
Type: HostPath (bare host directory volume)
Path: /data/postgres
HostPathType:
Events: <none>
any pointers would be helpful
Thanks!
Tony
Sep 16, 2021, 2:59:47 PM9/16/21
to AWX Project
when I check filesystems with df -h I dont see anything relating to /data/postgres or /data/projects.
Do I need to set these up before deploying?
hostPath:
path: /data/postgres
hostPath:
path: /data/projects
Wei-Yen Tan
Sep 16, 2021, 3:55:03 PM9/16/21
to Tony, AWX Project
This message is basically telling you that it does not have a persistent volume claim that it can fulfill. You need to provide that
From: 'Tony' via AWX Project <awx-p...@googlegroups.com>
Sent: Friday, September 17, 2021 6:59:47 AM
To: AWX Project <awx-p...@googlegroups.com>
Subject: [awx-project] Re: AWX 19 kubernetes
Sent: Friday, September 17, 2021 6:59:47 AM
To: AWX Project <awx-p...@googlegroups.com>
Subject: [awx-project] Re: AWX 19 kubernetes
--
You received this message because you are subscribed to the Google Groups "AWX Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to awx-project...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/871cf0dd-5759-48d2-86e6-b6b8331a970en%40googlegroups.com.
You received this message because you are subscribed to the Google Groups "AWX Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to awx-project...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/871cf0dd-5759-48d2-86e6-b6b8331a970en%40googlegroups.com.
Tony
Sep 16, 2021, 4:10:07 PM9/16/21
to AWX Project
Cheers
If you look at my first post, you will see that the original pods are in a running state, but then I have a postgres and awx pod in pending also. Additionally, everything worked fine. I rebooted the cluster and then lost my data - not fussed about this as I am testing at the moment.
From what I have read and If you check the yaml files, should that directory not work as a persistent volume? I can see the data writes to /data/postgres
Wei-Yen Tan
Sep 16, 2021, 4:13:25 PM9/16/21
to Tony, AWX Project
Yoy have a persistant volume. But you don't have a Persistant volume claim. So the deployment does not know it is there. There is no glue
Try doing "kubectl get pvc". :).
From: 'Tony' via AWX Project <awx-p...@googlegroups.com>
Sent: Friday, September 17, 2021 8:10:07 AM
To: AWX Project <awx-p...@googlegroups.com>
Subject: Re: [awx-project] Re: AWX 19 kubernetes
To: AWX Project <awx-p...@googlegroups.com>
Subject: Re: [awx-project] Re: AWX 19 kubernetes
To view this discussion on the web visit
https://groups.google.com/d/msgid/awx-project/142b87f6-8264-4e9a-97cd-251ba7e9cd50n%40googlegroups.com.
Message has been deleted
Tony
Sep 17, 2021, 5:11:14 AM9/17/21
to AWX Project
Hey, thanks for that. Something weird going on for sure...
kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
awx-projects-claim Pending awx-projects-volume 8d
postgres-awx-postgres-0 Pending awx-postgres-volume 8d
kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
awx-projects-volume 2Gi RWO Retain Bound awx/awx-projects-claim awx-projects-volume 13d
awx-postgres-volume 2Gi RWO Retain Bound awx/postgres-awx-postgres-0 awx-postgres-volume 13d
The pv's show as bound, but pvc as pending..
Here are my yaml files again:
awx.yaml
---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
name: awx
spec:
# These parameters are designed for use with:
# - AWX Operator: 0.13.0
# - AWX: 19.3.0
admin_user: admin
admin_password_secret: awx-admin-password
ingress_type: ingress
ingress_tls_secret: awx-secret-tls
hostname: hostname here
postgres_configuration_secret: awx-postgres-configuration
postgres_storage_class: awx-postgres-volume
postgres_storage_requirements:
requests:
storage: 2Gi
projects_persistence: true
projects_existing_claim: awx-projects-claim
Any further pointers would be appreciated.
Thank you
Wei-Yen Tan
Sep 17, 2021, 5:45:27 AM9/17/21
to Tony, AWX Project
Ok my bad. It was there! But gives us clues for sure. Can you please describe it?
Kubect describe pvc postgres-awx-postgres-0
From: 'Tony' via AWX Project <awx-p...@googlegroups.com>
Sent: Friday, September 17, 2021 9:11:14 PM
To view this discussion on the web visit
https://groups.google.com/d/msgid/awx-project/735f8f2b-f762-4a85-b904-de773483bc27n%40googlegroups.com.
Tony
Sep 17, 2021, 6:00:34 AM9/17/21
to AWX Project
Some more info
k describe pvc postgres-awx-postgres-0
Name: postgres-awx-postgres-0
Namespace: default
StorageClass: awx-postgres-volume
Status: Pending
Volume:
Annotations: <none>
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: awx-postgres-0
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ProvisioningFailed 26m (x3281 over 14h) persistentvolume-controller storageclass.storage.k8s.io "awx-postgres-volume" not found
Warning ProvisioningFailed 31s (x83 over 20m) persistentvolume-controller storageclass.storage.k8s.io "awx-postgres-volume" not found
k describe pvc postgres-awx-postgres-0 -n awx
Name: postgres-awx-postgres-0
Namespace: awx
StorageClass: awx-postgres-volume
Status: Bound
Volume: awx-postgres-volume
Annotations: pv.kubernetes.io/bind-completed: yes
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 2Gi
Access Modes: RWO
VolumeMode: Filesystem
Used By: awx-postgres-0
Events: <none>
Wei-Yen Tan
Sep 17, 2021, 6:02:09 AM9/17/21
to Tony, AWX Project
awx-postgres-volume" not found
Does this volume exist?
From: 'Tony' via AWX Project <awx-p...@googlegroups.com>
Sent: Friday, September 17, 2021 10:00:34 PM
To view this discussion on the web visit
https://groups.google.com/d/msgid/awx-project/b79cbca7-e1d6-46fd-aadc-4bf13b3ee5f0n%40googlegroups.com.
Tony
Sep 17, 2021, 6:19:17 AM9/17/21
to AWX Project
if you check the postgres in awx namespace, you can see it is
Wei-Yen Tan
Sep 17, 2021, 7:08:53 AM9/17/21
to Tony, AWX Project
I reviewed your email. If the path does not exist on the host /data/postgres and such it will bomb. The mount is a host path option
From: 'Tony' via AWX Project <awx-p...@googlegroups.com>
Sent: Friday, September 17, 2021 10:19:17 PM
To view this discussion on the web visit
https://groups.google.com/d/msgid/awx-project/00ef6928-d455-4467-8de2-d8d09eefbef4n%40googlegroups.com.
Tony
Sep 17, 2021, 7:45:47 AM9/17/21
to AWX Project
I just went through the install again..
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
awx-projects-volume 2Gi RWO Retain Bound awx/awx-projects-claim awx-projects-volume 11m
awx-postgres-volume 2Gi RWO Retain Bound default/postgres-awx-postgres-0 awx-postgres-volume 11m
why are the pvcs in different namespaces?
Should that be the case?
Andrei Mihai
Sep 6, 2022, 11:11:51 AM9/6/22
to AWX Project
I don't believe that should be the case, however just under the "name:
awx-postgres-volume " , in your
awx-postgres-volume .yaml file, you can specify the "namespace: awx", so that the pvc is created in teh correct namespace.
Reply all
Reply to author
Forward
0 new messages