AWX-Operator Installation error on awx-postgres-0
2,256 views
Skip to first unread message
Ariel Masciotta
Apr 26, 2021, 8:51:11 AM4/26/21
to AWX Project
Hello everyone.
I've used Tower for a long while now, on a daily basis, but I'm pretty new to installing an AWX instance in my kubernetes cluster. (For testing purposes).
So, I've setup 2 Centos 8, installed and configured a 2 node kubernetes cluster, and then tried to install AWX through the AWX_Operator.
In the past, I've successfully installed a trial version of ansible tower seamlessly, but AWX is giving me some headaches.
So, here's what I'm doing:
As per AWX_Operator installation guide (https://github.com/ansible/awx-operator/blob/devel/README.md), I've ran the following:
As per AWX_Operator installation guide (https://github.com/ansible/awx-operator/blob/devel/README.md), I've ran the following:
#> kubectl apply -f https://raw.githubusercontent.com/ansible/awx-operator/0.8.0/deploy/awx-operator.yaml
I had my awx-operator-xxxxx pod created.
then I create a my-awx.yml file with the following:
---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
name: awx
Then I run kubectl apply -f my-awx.yml
After a short while, I can see the services created, but awx-postgres-0 in a pending state.
When I describe the pod, I can see that it's not being able to create its storage:
When I describe the pod, I can see that it's not being able to create its storage:
default-scheduler 0/2 nodes are available: 2 pod has unbound immediate PersistentVolumeClaims.
Could someone let me know why I'm experiencing this error? I'm pretty new to k8s and I've googled around and tried dozens of things, but couldn't find how to solve it.
Thank you.
Ariel.
Sam
Jun 23, 2021, 8:14:18 AM6/23/21
to AWX Project
I got the same issue, have you been able to sort this out ?
Wei-Yen Tan
Jun 23, 2021, 8:38:32 AM6/23/21
to Sam, AWX Project
Yes. What's happening is thst it can't find storage for your pod.
In kubernetes a deployment will request for storage through a persistent volume claim otherwise called a pvc. You have to provide that storage with a persistrnt volume or a pv
--
You received this message because you are subscribed to the Google Groups "AWX Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to awx-project...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/ae9ff151-dc65-4a59-9a6c-13a34aa59e30n%40googlegroups.com.
Sam
Jun 23, 2021, 9:15:08 AM6/23/21
to AWX Project
Thanks for the hint, I am bit new to
kubernetes, would you be able please to elaborate more about where/how to change this ?
Wei-Yen Tan
Jun 23, 2021, 9:18:16 AM6/23/21
to Sam, AWX Project
Create a persistent volume
To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/675a9161-6c32-4be8-8a90-a65ccbb5a66an%40googlegroups.com.
Andrei Mihai
Sep 12, 2022, 5:25:12 AM9/12/22
to AWX Project
Hey,
I've followed all the guides and it seems like the postgress pod fails scheduling:
[sysadmin@dev-awx-01 k8awx]$ kubectl describe pod ****-postgres-13-0
Name: ****-postgres-13-0
Namespace: awx
Priority: 0
Node: <none>
Labels: app.kubernetes.io/component=database
app.kubernetes.io/instance=postgres-13-****
app.kubernetes.io/managed-by=awx-operator
app.kubernetes.io/name=postgres-13
app.kubernetes.io/part-of=****
controller-revision-hash=****-postgres-13-8677ccdd5d
statefulset.kubernetes.io/pod-name=****-postgres-13-0
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: StatefulSet/****-postgres-13
Containers:
postgres:
Image: postgres:13
Port: 5432/TCP
Host Port: 0/TCP
Requests:
cpu: 10m
memory: 64Mi
Environment:
POSTGRESQL_DATABASE: <set to the key 'database' in secret '****-postgres-configuration'> Optional: false
POSTGRESQL_USER: <set to the key 'username' in secret '****-postgres-configuration'> Optional: false
POSTGRESQL_PASSWORD: <set to the key 'password' in secret '****-postgres-configuration'> Optional: false
POSTGRES_DB: <set to the key 'database' in secret '****-postgres-configuration'> Optional: false
POSTGRES_USER: <set to the key 'username' in secret '****-postgres-configuration'> Optional: false
POSTGRES_PASSWORD: <set to the key 'password' in secret '****-postgres-configuration'> Optional: false
PGDATA: /var/lib/postgresql/data/pgdata
POSTGRES_INITDB_ARGS: --auth-host=scram-sha-256
POSTGRES_HOST_AUTH_METHOD: scram-sha-256
Mounts:
/var/lib/postgresql/data from postgres-13 (rw,path="data")
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-glhch (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
postgres-13:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: postgres-13-****-postgres-13-0
ReadOnly: false
kube-api-access-glhch:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 9s (x2 over 96s) default-scheduler 0/1 nodes are available: 1 node(s) didn't find available persistent volumes to bind.
I have two storage classes created. They are the same, except the bind mode. If I set the bind mode to Immediate, for the postgres SC, the postgres pod fails to scheduled with pod has unbound immediate PersistentVolumeClaims.
The projects-storage will bind regardless of binding mode.
[sysadmin@dev-awx-01 k8awx]$ kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
postgres-storage kubernetes.io/no-provisioner Delete WaitForFirstConsumer false 2m53s
projects-storage kubernetes.io/no-provisioner Delete Immediate false 2m53s
The PVs are the same. They are just named differently and point to their own storage class.
[sysadmin@dev-awx-01 k8awx]$ cat projects-storage-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: projects-storage-pv
namespace: awx
spec:
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
capacity:
storage: 20Gi
volumeMode: Filesystem
storageClassName: projects-storage
local:
path: /data/awx
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- dev-awx-0
[sysadmin@dev-awx-01 k8awx]$ cat postgres-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: postgres-storage-pv
spec:
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
capacity:
storage: 20Gi
volumeMode: Filesystem
storageClassName: postgres-storage
hostPath:
path: /data/postgress
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- dev-awx-0
The PVs get created properly.
We are using local storage for this:
[sysadmin@dev-awx-01 k8awx]$ df -h
/dev/mapper/ansible-awx--storage 25G 211M 25G 1% /data/awx
/dev/mapper/ansible-postgres--storage 25G 211M 25G 1% /data/postgress
[sysadmin@dev-awx-01 k8awx]$
[sysadmin@dev-awx-01 k8awx]$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
postgres-storage-pv 20Gi RWX Retain Available postgres-storage 3m
projects-storage-pv 20Gi RWX Retain Bound awx/projects-storage-pvc projects-storage 3m
However, the PVC for postgres-13-*****-postgres-13-0 will not bind.
[sysadmin@dev-awx-01 k8awx]$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
postgres-13-*****-postgres-13-0 Pending postgres-storage 2m40s
projects-storage-pvc Bound projects-storage-pv 20Gi RWX projects-storage 3m3s
Describing the PVC, shows it's waiting for the POD to be scheduled.
[sysadmin@dev-awx-01 k8awx]$ kubectl describe pvc postgres-13-****-postgres-13-0
Name: postgres-13-****-postgres-13-0
Namespace: awx
StorageClass: postgres-storage
Status: Pending
Volume:
Labels: app.kubernetes.io/component=database
app.kubernetes.io/instance=postgres-13-****
app.kubernetes.io/managed-by=awx-operator
app.kubernetes.io/name=postgres-13
Annotations: <none>
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: ****-postgres-13-0
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WaitForFirstConsumer 9m29s persistentvolume-controller waiting for first consumer to be created before binding
Normal WaitForPodScheduled 3m25s (x25 over 9m25s) persistentvolume-controller waiting for pod ****-postgres-13-0 to be scheduled
I've followed all the guides and it seems like the postgress pod fails scheduling:
[sysadmin@dev-awx-01 k8awx]$ kubectl describe pod ****-postgres-13-0
Name: ****-postgres-13-0
Namespace: awx
Priority: 0
Node: <none>
Labels: app.kubernetes.io/component=database
app.kubernetes.io/instance=postgres-13-****
app.kubernetes.io/managed-by=awx-operator
app.kubernetes.io/name=postgres-13
app.kubernetes.io/part-of=****
controller-revision-hash=****-postgres-13-8677ccdd5d
statefulset.kubernetes.io/pod-name=****-postgres-13-0
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: StatefulSet/****-postgres-13
Containers:
postgres:
Image: postgres:13
Port: 5432/TCP
Host Port: 0/TCP
Requests:
cpu: 10m
memory: 64Mi
Environment:
POSTGRESQL_DATABASE: <set to the key 'database' in secret '****-postgres-configuration'> Optional: false
POSTGRESQL_USER: <set to the key 'username' in secret '****-postgres-configuration'> Optional: false
POSTGRESQL_PASSWORD: <set to the key 'password' in secret '****-postgres-configuration'> Optional: false
POSTGRES_DB: <set to the key 'database' in secret '****-postgres-configuration'> Optional: false
POSTGRES_USER: <set to the key 'username' in secret '****-postgres-configuration'> Optional: false
POSTGRES_PASSWORD: <set to the key 'password' in secret '****-postgres-configuration'> Optional: false
PGDATA: /var/lib/postgresql/data/pgdata
POSTGRES_INITDB_ARGS: --auth-host=scram-sha-256
POSTGRES_HOST_AUTH_METHOD: scram-sha-256
Mounts:
/var/lib/postgresql/data from postgres-13 (rw,path="data")
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-glhch (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
postgres-13:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: postgres-13-****-postgres-13-0
ReadOnly: false
kube-api-access-glhch:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 9s (x2 over 96s) default-scheduler 0/1 nodes are available: 1 node(s) didn't find available persistent volumes to bind.
I have two storage classes created. They are the same, except the bind mode. If I set the bind mode to Immediate, for the postgres SC, the postgres pod fails to scheduled with pod has unbound immediate PersistentVolumeClaims.
The projects-storage will bind regardless of binding mode.
[sysadmin@dev-awx-01 k8awx]$ kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
postgres-storage kubernetes.io/no-provisioner Delete WaitForFirstConsumer false 2m53s
projects-storage kubernetes.io/no-provisioner Delete Immediate false 2m53s
The PVs are the same. They are just named differently and point to their own storage class.
[sysadmin@dev-awx-01 k8awx]$ cat projects-storage-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: projects-storage-pv
namespace: awx
spec:
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
capacity:
storage: 20Gi
volumeMode: Filesystem
storageClassName: projects-storage
local:
path: /data/awx
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- dev-awx-0
[sysadmin@dev-awx-01 k8awx]$ cat postgres-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: postgres-storage-pv
spec:
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
capacity:
storage: 20Gi
volumeMode: Filesystem
storageClassName: postgres-storage
hostPath:
path: /data/postgress
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- dev-awx-0
The PVs get created properly.
We are using local storage for this:
[sysadmin@dev-awx-01 k8awx]$ df -h
/dev/mapper/ansible-awx--storage 25G 211M 25G 1% /data/awx
/dev/mapper/ansible-postgres--storage 25G 211M 25G 1% /data/postgress
[sysadmin@dev-awx-01 k8awx]$
[sysadmin@dev-awx-01 k8awx]$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
postgres-storage-pv 20Gi RWX Retain Available postgres-storage 3m
projects-storage-pv 20Gi RWX Retain Bound awx/projects-storage-pvc projects-storage 3m
However, the PVC for postgres-13-*****-postgres-13-0 will not bind.
[sysadmin@dev-awx-01 k8awx]$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
postgres-13-*****-postgres-13-0 Pending postgres-storage 2m40s
projects-storage-pvc Bound projects-storage-pv 20Gi RWX projects-storage 3m3s
Describing the PVC, shows it's waiting for the POD to be scheduled.
[sysadmin@dev-awx-01 k8awx]$ kubectl describe pvc postgres-13-****-postgres-13-0
Name: postgres-13-****-postgres-13-0
Namespace: awx
StorageClass: postgres-storage
Status: Pending
Volume:
Labels: app.kubernetes.io/component=database
app.kubernetes.io/instance=postgres-13-****
app.kubernetes.io/managed-by=awx-operator
app.kubernetes.io/name=postgres-13
Annotations: <none>
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: ****-postgres-13-0
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WaitForFirstConsumer 9m29s persistentvolume-controller waiting for first consumer to be created before binding
Normal WaitForPodScheduled 3m25s (x25 over 9m25s) persistentvolume-controller waiting for pod ****-postgres-13-0 to be scheduled
Below is the awx yaml and the kustomization.yaml.
name: ****
namespace: awx
spec:
service_type: nodeport
ingress_type: ingress
hostname: ansible-lower.****.com
ingress_tls_secret: ****-tls
auto_upgrade: true
projects_persistence: true
projects_existing_claim: projects-storage-pvc
postgres_storage_class: postgres-storage
set_self_labels: false
no_log: 'false'
namespace: awx
spec:
service_type: nodeport
ingress_type: ingress
hostname: ansible-lower.****.com
ingress_tls_secret: ****-tls
auto_upgrade: true
projects_persistence: true
projects_existing_claim: projects-storage-pvc
postgres_storage_class: postgres-storage
set_self_labels: false
no_log: 'false'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
# Find the latest tag here: https://github.com/ansible/awx-operator/releases
- github.com/ansible/awx-operator/config/default?ref=0.28.0
- projects-sc.yaml
- postgres-sc.yaml
- projects-storage-pv.yaml
- postgres-pv.yaml
- projects-storage-pvc.yaml
- awx.yaml
# Set the image tags to match the git version from above
images:
- name: quay.io/ansible/awx-operator
newTag: 0.28.0
# Specify a custom namespace in which to install AWX
namespace: awx
kind: Kustomization
resources:
# Find the latest tag here: https://github.com/ansible/awx-operator/releases
- github.com/ansible/awx-operator/config/default?ref=0.28.0
- projects-sc.yaml
- postgres-sc.yaml
- projects-storage-pv.yaml
- postgres-pv.yaml
- projects-storage-pvc.yaml
- awx.yaml
# Set the image tags to match the git version from above
images:
- name: quay.io/ansible/awx-operator
newTag: 0.28.0
# Specify a custom namespace in which to install AWX
namespace: awx
I would appreciate any help/advice on this.
Thanks in advance.
Reply all
Reply to author
Forward
0 new messages