postgres-0 status is CrashLoopBackOff
822 views
Skip to first unread message
Yasu
Aug 30, 2021, 4:35:45 AM8/30/21
to AWX Project
Hi,
I've been using AWX 19.3.0 / AWX operator 0.13.0 on minikube.
After deploying AWX, I was able to access it without any problems. But, I have lost access to AWX today.
The status of postgres-0 pod was CrashLoopBackOff.
The log output was "initdb: error: directory "/var/lib/postgresql/data/pgdata" exists but is not empty".
Is there any way to recover?
I would be grateful if anyone could comment.
The following is the status of the pod and the output of describe and log.
[root@centos-tools ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
awx-demo-7bbb564887-m7t5n 4/4 Running 8 13d
awx-demo-postgres-0 0/1 CrashLoopBackOff 834 13d
awx-operator-69c646c48f-cbf5k 1/1 Running 1 13d
====================
[root@centos-tools ~]# kubectl describe pod/awx-demo-postgres-0
Name: awx-demo-postgres-0
Namespace: default
Priority: 0
Node: centos-tools/172.27.115.105
Start Time: Tue, 17 Aug 2021 16:17:30 +0900
controller-revision-hash=awx-demo-postgres-79f76985cb
Annotations: <none>
Status: Running
IP: 172.17.0.6
IPs:
IP: 172.17.0.6
Controlled By: StatefulSet/awx-demo-postgres
Containers:
postgres:
Container ID: docker://fb6205c01ffeb95c50b2c2c45c8a48af67da7b60fb996706fa312053696b9698
Image: postgres:12
Image ID: docker-pullable://postgres@sha256:e10cd754296c1b3e93a121bcc64550d183df56f20bab47e08cacb123fb969b5e
Port: 5432/TCP
Host Port: 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Mon, 30 Aug 2021 17:11:05 +0900
Finished: Mon, 30 Aug 2021 17:11:05 +0900
Ready: False
Restart Count: 839
Environment:
POSTGRESQL_DATABASE: <set to the key 'database' in secret 'awx-demo-postgres-configuration'> Optional: false
POSTGRESQL_USER: <set to the key 'username' in secret 'awx-demo-postgres-configuration'> Optional: false
POSTGRESQL_PASSWORD: <set to the key 'password' in secret 'awx-demo-postgres-configuration'> Optional: false
POSTGRES_DB: <set to the key 'database' in secret 'awx-demo-postgres-configuration'> Optional: false
POSTGRES_USER: <set to the key 'username' in secret 'awx-demo-postgres-configuration'> Optional: false
POSTGRES_PASSWORD: <set to the key 'password' in secret 'awx-demo-postgres-configuration'> Optional: false
PGDATA: /var/lib/postgresql/data/pgdata
POSTGRES_INITDB_ARGS: --auth-host=scram-sha-256
POSTGRES_HOST_AUTH_METHOD: scram-sha-256
Mounts:
/var/lib/postgresql/data from postgres (rw,path="data")
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bjd4p (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
postgres:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: postgres-awx-demo-postgres-0
ReadOnly: false
kube-api-access-bjd4p:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning BackOff 69m (x19429 over 2d23h) kubelet Back-off restarting failed container
Normal SandboxChanged 40m kubelet Pod sandbox changed, it will be killed and re-created.
Normal Pulled 39m (x4 over 40m) kubelet Container image "postgres:12" already present on machine
Normal Created 39m (x4 over 40m) kubelet Created container postgres
Normal Started 39m (x4 over 40m) kubelet Started container postgres
Warning BackOff 25s (x185 over 40m) kubelet Back-off restarting failed container
[root@centos-tools ~]#
====================
[root@centos-tools ~]# kubectl logs pod/awx-demo-postgres-0
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
initdb: error: directory "/var/lib/postgresql/data/pgdata" exists but is not empty
If you want to create a new database system, either remove or empty
the directory "/var/lib/postgresql/data/pgdata" or run initdb
with an argument other than "/var/lib/postgresql/data/pgdata".
[root@centos-tools ~]#
Best Regards,
Yasu
Abhinit Mallick
Aug 31, 2021, 12:20:45 PM8/31/21
to AWX Project
Hi Yasu
Few rhetorical queries
1. Are you using an external postgres DB for AWX ??
if external then where does "/var/lib/postgresql/data/pgdata" map to
Few rhetorical queries
1. Are you using an external postgres DB for AWX ??
if external then where does "/var/lib/postgresql/data/pgdata" map to
2. Is it a fresh installation or are you trying to migrate things ???
3. please share the yamls which you are using to start the AWX
Regards
Abhinit
Regards
Abhinit
Yasu
Sep 2, 2021, 3:23:16 AM9/2/21
to AWX Project
Hi Abhinit,
Thank you for your reply.
1. Are you using an external postgres DB for AWX ??
No, I using internal postgres DB.
2. Is it a fresh installation or are you trying to migrate things ???
It is the fresh installation.
3. please share the yamls which you are using to start the AWX
I used the following YAML.
kubectl apply -f https://raw.githubusercontent.com/ansible/awx-operator/0.13.0/deploy/awx-operator.yaml
kubectl apply -f awx-demo.yml
>>>>>
# cat awx-demo.yml
---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
name: awx-demo
spec:
service_type: nodeport
ingress_type: none
hostname: awx-demo.juniper.local
>>>>>
Best Regards,
Yasu
2021年9月1日水曜日 1:20:45 UTC+9 Abhinit Mallick:
Abhinit Mallick
Sep 5, 2021, 1:08:47 PM9/5/21
to AWX Project
Hi Yasu
As it's is fresh setup I would rather request you to use an external postgres database which will be hosted on your minikube where the postgres database will store the data in your local minikube host ( in that way if you backup the filesystem your postgres is getting backed up)
Your awx will basically access the external postgres and will keep running.
If you are fine with the solution
I can share the configuration with you .
Regards
Abhinit
Kosala Atapattu
Sep 5, 2021, 5:12:52 PM9/5/21
to Yasu, AWX Project
Hi,
This docker_setup_env() function gets called at the beginning of the _main() and the postgres goes to a initdb path since DATABASE_ALREADY_EXISTS is empty.
Looking at postgres:12 image, the docker-entry.sh has the following:
| docker_setup_env() { | |
| file_env 'POSTGRES_PASSWORD' | |
| file_env 'POSTGRES_USER' 'postgres' | |
| file_env 'POSTGRES_DB' "$POSTGRES_USER" | |
| file_env 'POSTGRES_INITDB_ARGS' | |
| # default authentication method is md5 | |
| : "${POSTGRES_HOST_AUTH_METHOD:=md5}" | |
| declare -g DATABASE_ALREADY_EXISTS | |
| # look specifically for PG_VERSION, as it is expected in the DB dir | |
| if [ -s "$PGDATA/PG_VERSION" ]; then | |
| DATABASE_ALREADY_EXISTS='true' | |
| fi | |
| } | |
This docker_setup_env() function gets called at the beginning of the _main() and the postgres goes to a initdb path since DATABASE_ALREADY_EXISTS is empty.
| # only run initialization on an empty data directory | |
| if [ -z "$DATABASE_ALREADY_EXISTS" ]; then | |
| docker_verify_minimum_env | |
| # check dir permissions to reduce likelihood of half-initialized database | |
| ls /docker-entrypoint-initdb.d/ > /dev/null | |
| docker_init_database_dir | |
| pg_setup_hba_conf |
Looks like in your case your PGDATA path exists, but the PG_VERSION file is not there or empty. I think your postgres db is corrupt. You need to rebuild the DB, and restore from a backup.
HTH,
HTH,
Kosala
--
You received this message because you are subscribed to the Google Groups "AWX Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to awx-project...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/3da1bd5c-a162-4cf4-bab5-3f0493c2f031n%40googlegroups.com.
Yasu
Sep 5, 2021, 8:24:13 PM9/5/21
to AWX Project
Hi Abhinit,
Thank you for your reply.
I understand it.
Could you please share the configuration?
Best Regard,
Yasu
2021年9月6日月曜日 2:08:47 UTC+9 Abhinit Mallick:
Reply all
Reply to author
Forward
0 new messages