Permission issues prevent pod starting
1,098 views
Skip to first unread message
Jean-Daniel
Sep 27, 2021, 2:17:08 PM9/27/21
to Postgres Operator
I'm trying to install and create a simple cluster on Kubernetes 1.22 following QuickStart guide, but my Postgres cluster fails to start because of some permissions issues:
kubectl logs -n postgres-operator pod/hippo-instance1-vnln-0 -c database-client-cert-init
install: cannot open '/pgconf/tls/replication/tls.crt' for reading: Permission denied
install: cannot open '/pgconf/tls/replication/tls.key' for reading: Permission denied
install: cannot open '/pgconf/tls/replication/ca.crt' for reading: Permission denied
I don't know if this is related, but when dumping the pod yaml, I see a dubious defaultMode for the cert-volume:
name: cert-volume
projected:
defaultMode: 384
I also have a error with backrest pod:
kubectl logs -n postgres-operator pod/hippo-repo-host-0 -c pgbackrest
/etc/ssh/sshd_config: Permission denied
And for this one too, there is a dubious default mode:
- name: ssh
projected:
defaultMode: 32
Did I miss something, or is there some issue with the controller ?
Jonathan S. Katz
Sep 27, 2021, 2:23:03 PM9/27/21
to Jean-Daniel, Postgres Operator
Can you please provide the version of PGO that you are using, as well as the manifests that you used?
What is the underlying storage system you are using?
Jean-Daniel
Sep 27, 2021, 3:17:17 PM9/27/21
to Postgres Operator, jonath...@crunchydata.com, Postgres Operator, Jean-Daniel
I'm using PGO 5.0.2
My manifest is simple as I did just a kubectl apply -k from the example repository.
My storage is a rook-ceph rdb pool.
Operator:
```
apiVersion: v1
kind: Namespace
metadata:
name: postgres-operator
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
postgres-operator.crunchydata.com/control-plane: postgres-operator
name: pgo
namespace: postgres-operator
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
postgres-operator.crunchydata.com/control-plane: postgres-operator
name: postgres-operator
rules:
- apiGroups:
- ""
resources:
- configmaps
- persistentvolumeclaims
- secrets
- services
verbs:
- create
- delete
- get
- list
- patch
- watch
- apiGroups:
- ""
resources:
- endpoints
verbs:
- create
- delete
- deletecollection
- get
- list
- patch
- watch
- apiGroups:
- ""
resources:
- endpoints/restricted
- pods/exec
verbs:
- create
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- ""
resources:
- pods
verbs:
- delete
- get
- list
- patch
- watch
- apiGroups:
- ""
resources:
- serviceaccounts
verbs:
- create
- get
- list
- patch
- watch
- apiGroups:
- apps
resources:
- deployments
- statefulsets
verbs:
- create
- delete
- get
- list
- patch
- watch
- apiGroups:
- batch
resources:
- cronjobs
- jobs
verbs:
- create
- delete
- get
- list
- patch
- watch
- apiGroups:
- postgres-operator.crunchydata.com
resources:
- postgresclusters
verbs:
- get
- list
- patch
- watch
- apiGroups:
- postgres-operator.crunchydata.com
resources:
- postgresclusters/finalizers
verbs:
- update
- apiGroups:
- postgres-operator.crunchydata.com
resources:
- postgresclusters/status
verbs:
- patch
- apiGroups:
resources:
- rolebindings
- roles
verbs:
- create
- get
- list
- patch
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
postgres-operator.crunchydata.com/control-plane: postgres-operator
name: postgres-operator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: postgres-operator
subjects:
- kind: ServiceAccount
name: pgo
namespace: postgres-operator
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
postgres-operator.crunchydata.com/control-plane: postgres-operator
name: pgo
namespace: postgres-operator
spec:
replicas: 1
selector:
matchLabels:
postgres-operator.crunchydata.com/control-plane: postgres-operator
template:
metadata:
labels:
postgres-operator.crunchydata.com/control-plane: postgres-operator
spec:
containers:
- env:
- name: CRUNCHY_DEBUG
value: "true"
- name: RELATED_IMAGE_POSTGRES_13
value: registry.developers.crunchydata.com/crunchydata/crunchy-postgres-ha:centos8-13.4-0
- name: RELATED_IMAGE_POSTGRES_13_GIS_3.1
value: registry.developers.crunchydata.com/crunchydata/crunchy-postgres-gis-ha:centos8-13.4-3.1-0
- name: RELATED_IMAGE_PGBACKREST
value: registry.developers.crunchydata.com/crunchydata/crunchy-pgbackrest:centos8-2.33-2
- name: RELATED_IMAGE_PGBOUNCER
value: registry.developers.crunchydata.com/crunchydata/crunchy-pgbouncer:centos8-1.15-2
- name: RELATED_IMAGE_PGEXPORTER
value: registry.developers.crunchydata.com/crunchydata/crunchy-postgres-exporter:ubi8-5.0.2-0
image: registry.developers.crunchydata.com/crunchydata/postgres-operator:ubi8-5.0.2-0
name: operator
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsNonRoot: true
serviceAccountName: pgo
```Cluster:
```
apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PostgresCluster
metadata:
name: hippo
namespace: postgres-operator
spec:
backups:
pgbackrest:
image: registry.developers.crunchydata.com/crunchydata/crunchy-pgbackrest:centos8-2.33-2
repoHost:
dedicated: {}
repos:
- name: repo1
volume:
volumeClaimSpec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
image: registry.developers.crunchydata.com/crunchydata/crunchy-postgres-ha:centos8-13.4-0
instances:
- dataVolumeClaimSpec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
name: instance1
postgresVersion: 13
```
Andrew L'Ecuyer
Sep 27, 2021, 3:33:33 PM9/27/21
to Postgres Operator, Jean-Daniel, jonath...@crunchydata.com, Postgres Operator
Would you also be able to provide the YAML output of both the PostgreSQL instance (hippo-instance1-vnln-0) and the pgBackRest repo host (hippo-repo-host-0)?
kubectl get pod hippo-instance1-vnln-0 -o yaml
kubectl get pod hippo-repo-host-0 -o yaml
I am specifically interested in the securityContext that is being set for each of these Pods.
Thanks,
Andrew
Jean-Daniel
Sep 27, 2021, 3:48:44 PM9/27/21
to Andrew L'Ecuyer, Postgres Operator, jonath...@crunchydata.com
kubectl get pod hippo-instance1-vnln-0 -o yaml
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: "2021-09-27T17:29:05Z"
generateName: hippo-instance1-vnln-
labels:
controller-revision-hash: hippo-instance1-vnln-79b77b55b6
postgres-operator.crunchydata.com/instance: hippo-instance1-vnln
statefulset.kubernetes.io/pod-name: hippo-instance1-vnln-0
name: hippo-instance1-vnln-0
namespace: postgres-operator
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: StatefulSet
name: hippo-instance1-vnln
uid: 74663980-292c-4dd9-b3fe-09bb0d4c0e1b
resourceVersion: "3473087"
uid: fab138ef-9b8f-4df8-89ca-9e4da976b03d
spec:
containers:
- command:
- patroni
- /etc/patroni
env:
- name: PGDATA
value: /pgdata/pg13
- name: PGHOST
value: /tmp/postgres
- name: PGPORT
value: "5432"
- name: PATRONI_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: PATRONI_KUBERNETES_POD_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: PATRONI_KUBERNETES_PORTS
value: |
- name: postgres
port: 5432
protocol: TCP
- name: PATRONI_POSTGRESQL_CONNECT_ADDRESS
value: $(PATRONI_NAME).hippo-pods:5432
- name: PATRONI_POSTGRESQL_LISTEN
value: '*:5432'
- name: PATRONI_POSTGRESQL_CONFIG_DIR
value: /pgdata/pg13
- name: PATRONI_POSTGRESQL_DATA_DIR
value: /pgdata/pg13
- name: PATRONI_RESTAPI_CONNECT_ADDRESS
value: $(PATRONI_NAME).hippo-pods:8008
- name: PATRONI_RESTAPI_LISTEN
value: '*:8008'
- name: PATRONICTL_CONFIG_FILE
value: /etc/patroni
- name: LD_PRELOAD
value: /usr/lib64/libnss_wrapper.so
- name: NSS_WRAPPER_PASSWD
value: /tmp/nss_wrapper/postgres/passwd
- name: NSS_WRAPPER_GROUP
value: /tmp/nss_wrapper/postgres/group
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /liveness
port: 8008
scheme: HTTPS
initialDelaySeconds: 3
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
name: database
ports:
- containerPort: 5432
name: postgres
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /readiness
port: 8008
scheme: HTTPS
initialDelaySeconds: 3
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
resources: {}
securityContext:
allowPrivilegeEscalation: false
privileged: false
readOnlyRootFilesystem: true
runAsNonRoot: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /pgdata
name: postgres-data
- mountPath: /etc/patroni
name: patroni-config
readOnly: true
- mountPath: /etc/ssh
name: ssh
readOnly: true
- mountPath: /etc/pgbackrest/conf.d
name: pgbackrest-config
- mountPath: /pgconf/tls
name: cert-volume
readOnly: true
- mountPath: /tmp
name: tmp
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-zc7vq
readOnly: true
- command:
- bash
- -c
- |2
declare -r mountDir=/pgconf/tls/replication
declare -r tmpDir=/tmp/replication
while sleep 5s; do
mkdir -p /tmp/replication
DIFF=$(diff ${mountDir} ${tmpDir})
if [ "$DIFF" != "" ]
then
date
echo Copying replication certificates and key and setting permissions
install -m 0600 ${mountDir}/{tls.crt,tls.key,ca.crt} ${tmpDir}
patronictl reload hippo-ha --force
fi
done
env:
- name: PGDATA
value: /pgdata/pg13
- name: PGHOST
value: /tmp/postgres
- name: PGPORT
value: "5432"
- name: PATRONI_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: PATRONI_KUBERNETES_POD_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: PATRONI_KUBERNETES_PORTS
value: |
- name: postgres
port: 5432
protocol: TCP
- name: PATRONI_POSTGRESQL_CONNECT_ADDRESS
value: $(PATRONI_NAME).hippo-pods:5432
- name: PATRONI_POSTGRESQL_LISTEN
value: '*:5432'
- name: PATRONI_POSTGRESQL_CONFIG_DIR
value: /pgdata/pg13
- name: PATRONI_POSTGRESQL_DATA_DIR
value: /pgdata/pg13
- name: PATRONI_RESTAPI_CONNECT_ADDRESS
value: $(PATRONI_NAME).hippo-pods:8008
- name: PATRONI_RESTAPI_LISTEN
value: '*:8008'
- name: PATRONICTL_CONFIG_FILE
value: /etc/patroni
imagePullPolicy: IfNotPresent
name: replication-cert-copy
resources: {}
securityContext:
allowPrivilegeEscalation: false
privileged: false
readOnlyRootFilesystem: true
runAsNonRoot: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/patroni
name: patroni-config
readOnly: true
- mountPath: /pgconf/tls
name: cert-volume
readOnly: true
- mountPath: /tmp
name: tmp
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-zc7vq
readOnly: true
- command:
- /usr/sbin/sshd
- -D
- -e
env:
- name: LD_PRELOAD
value: /usr/lib64/libnss_wrapper.so
- name: NSS_WRAPPER_PASSWD
value: /tmp/nss_wrapper/postgres/passwd
- name: NSS_WRAPPER_GROUP
value: /tmp/nss_wrapper/postgres/group
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
periodSeconds: 10
successThreshold: 1
tcpSocket:
port: 2022
timeoutSeconds: 1
name: pgbackrest
resources: {}
securityContext:
allowPrivilegeEscalation: false
privileged: false
readOnlyRootFilesystem: true
runAsNonRoot: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/ssh
name: ssh
readOnly: true
- mountPath: /pgdata
name: postgres-data
- mountPath: /etc/pgbackrest/conf.d
name: pgbackrest-config
- mountPath: /tmp
name: tmp
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-zc7vq
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
hostname: hippo-instance1-vnln-0
initContainers:
- command:
- bash
- -ceu
- --
- |-
declare -r expected_major_version="$1" pgwal_directory="$2"
results() { printf '::postgres-operator: %s::%s\n' "$@"; }
safelink() (
local desired="$1" name="$2" current
current=$(realpath "${name}")
if [ "${current}" = "${desired}" ]; then return; fi
set -x; mv --no-target-directory "${current}" "${desired}"
ln --no-dereference --force --symbolic "${desired}" "${name}"
)
echo Initializing ...
results 'uid' "$(id -u)" 'gid' "$(id -G)"
results 'postgres path' "$(command -v postgres)"
results 'postgres version' "${postgres_version:=$(postgres --version)}"
[[ "${postgres_version}" == *") ${expected_major_version}."* ]]
results 'config directory' "${PGDATA:?}"
postgres_data_directory=$([ -d "${PGDATA}" ] && postgres -C data_directory || echo "${PGDATA}")
results 'data directory' "${postgres_data_directory}"
[ "${postgres_data_directory}" = "${PGDATA}" ]
bootstrap_dir="${postgres_data_directory}_bootstrap"
[ -d "${bootstrap_dir}" ] && results 'bootstrap directory' "${bootstrap_dir}"
[ -d "${bootstrap_dir}" ] && postgres_data_directory="${bootstrap_dir}"
install --directory --mode=0700 "${postgres_data_directory}"
[ -f "${postgres_data_directory}/PG_VERSION" ] || exit 0
results 'data version' "${postgres_data_version:=$(< "${postgres_data_directory}/PG_VERSION")}"
[ "${postgres_data_version}" = "${expected_major_version}" ]
safelink "${pgwal_directory}" "${postgres_data_directory}/pg_wal"
results 'wal directory' "$(realpath "${postgres_data_directory}/pg_wal")"
- startup
- "13"
- /pgdata/pg13_wal
env:
- name: PGDATA
value: /pgdata/pg13
- name: PGHOST
value: /tmp/postgres
- name: PGPORT
value: "5432"
imagePullPolicy: IfNotPresent
name: postgres-startup
resources: {}
securityContext:
allowPrivilegeEscalation: false
privileged: false
readOnlyRootFilesystem: true
runAsNonRoot: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /pgdata
name: postgres-data
- mountPath: /tmp
name: tmp
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-zc7vq
readOnly: true
- command:
- bash
- -c
- mkdir -p /tmp/replication && install -m 0600 /pgconf/tls/replication/{tls.crt,tls.key,ca.crt}
/tmp/replication
imagePullPolicy: IfNotPresent
name: database-client-cert-init
resources: {}
securityContext:
allowPrivilegeEscalation: false
privileged: false
readOnlyRootFilesystem: true
runAsNonRoot: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /pgconf/tls
name: cert-volume
readOnly: true
- mountPath: /tmp
name: tmp
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-zc7vq
readOnly: true
- command:
- bash
- -c
- NSS_WRAPPER_SUBDIR=postgres CRUNCHY_NSS_USERNAME=postgres CRUNCHY_NSS_USER_DESC="postgres"
/opt/crunchy/bin/nss_wrapper.sh
imagePullPolicy: IfNotPresent
name: nss-wrapper-init
resources: {}
securityContext:
allowPrivilegeEscalation: false
privileged: false
readOnlyRootFilesystem: true
runAsNonRoot: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /tmp
name: tmp
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-zc7vq
readOnly: true
nodeName: worker-6.cluster
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
runAsNonRoot: true
supplementalGroups:
- 65534
serviceAccount: hippo-instance
serviceAccountName: hippo-instance
shareProcessNamespace: true
subdomain: hippo-pods
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
operator: Exists
tolerationSeconds: 300
volumes:
- name: postgres-data
persistentVolumeClaim:
claimName: hippo-instance1-vnln-pgdata
- name: patroni-config
projected:
defaultMode: 420
sources:
- configMap:
items:
- key: patroni.yaml
path: ~postgres-operator_cluster.yaml
name: hippo-config
- configMap:
items:
- key: patroni.yaml
path: ~postgres-operator_instance.yaml
name: hippo-instance1-vnln-config
- secret:
items:
- key: patroni.ca-roots
path: ~postgres-operator/patroni.ca-roots
- key: patroni.crt-combined
path: ~postgres-operator/patroni.crt+key
name: hippo-instance1-vnln-certs
- name: ssh
projected:
defaultMode: 32
sources:
- configMap:
name: hippo-ssh-config
- secret:
name: hippo-ssh
- name: pgbackrest-config
projected:
defaultMode: 420
sources:
- configMap:
items:
- key: hippo-instance1-vnln.conf
path: hippo-instance1-vnln.conf
- key: config-hash
path: config-hash
name: hippo-pgbackrest-config
- name: cert-volume
projected:
defaultMode: 384
sources:
- secret:
items:
- key: tls.crt
path: tls.crt
- key: tls.key
path: tls.key
- key: ca.crt
path: ca.crt
name: hippo-cluster-cert
- secret:
items:
- key: tls.crt
path: replication/tls.crt
- key: tls.key
path: replication/tls.key
- key: ca.crt
path: replication/ca.crt
name: hippo-replication-cert
- emptyDir:
sizeLimit: 16Mi
name: tmp
- name: kube-api-access-zc7vq
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2021-09-27T17:29:06Z"
message: 'containers with incomplete status: [database-client-cert-init nss-wrapper-init]'
reason: ContainersNotInitialized
status: "False"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2021-09-27T17:29:06Z"
message: 'containers with unready status: [database replication-cert-copy pgbackrest]'
reason: ContainersNotReady
status: "False"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2021-09-27T17:29:06Z"
message: 'containers with unready status: [database replication-cert-copy pgbackrest]'
reason: ContainersNotReady
status: "False"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2021-09-27T17:29:06Z"
status: "True"
type: PodScheduled
containerStatuses:
imageID: ""
lastState: {}
name: database
ready: false
restartCount: 0
started: false
state:
waiting:
reason: PodInitializing
imageID: ""
lastState: {}
name: pgbackrest
ready: false
restartCount: 0
started: false
state:
waiting:
reason: PodInitializing
imageID: ""
lastState: {}
name: replication-cert-copy
ready: false
restartCount: 0
started: false
state:
waiting:
reason: PodInitializing
hostIP: 172.22.0.21
initContainerStatuses:
lastState: {}
name: postgres-startup
ready: true
restartCount: 0
state:
terminated:
exitCode: 0
finishedAt: "2021-09-27T17:29:17Z"
reason: Completed
startedAt: "2021-09-27T17:29:17Z"
lastState:
terminated:
exitCode: 1
finishedAt: "2021-09-27T19:38:00Z"
reason: Error
startedAt: "2021-09-27T19:38:00Z"
name: database-client-cert-init
ready: false
restartCount: 30
state:
waiting:
message: back-off 5m0s restarting failed container=database-client-cert-init
pod=hippo-instance1-vnln-0_postgres-operator(fab138ef-9b8f-4df8-89ca-9e4da976b03d)
reason: CrashLoopBackOff
imageID: ""
lastState: {}
name: nss-wrapper-init
ready: false
restartCount: 0
state:
waiting:
reason: PodInitializing
phase: Pending
podIP: 10.128.8.52
podIPs:
- ip: 10.128.8.52
- ip: fd00:128::874
qosClass: BestEffort
startTime: "2021-09-27T17:29:06Z »
kubectl get pod hippo-repo-host-0 -o yaml
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: "2021-09-27T17:29:05Z"
generateName: hippo-repo-host-
labels:
controller-revision-hash: hippo-repo-host-77f68cc4d6
statefulset.kubernetes.io/pod-name: hippo-repo-host-0
name: hippo-repo-host-0
namespace: postgres-operator
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: StatefulSet
name: hippo-repo-host
uid: c62061db-79fe-493c-879d-aa677e8eb477
resourceVersion: "3474865"
uid: 81dba4e8-d24c-45c9-b4c0-685ccdd783e0
spec:
containers:
- command:
- /usr/sbin/sshd
- -D
- -e
env:
- name: LD_PRELOAD
value: /usr/lib64/libnss_wrapper.so
- name: NSS_WRAPPER_PASSWD
value: /tmp/nss_wrapper/postgres/passwd
- name: NSS_WRAPPER_GROUP
value: /tmp/nss_wrapper/postgres/group
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
periodSeconds: 10
successThreshold: 1
tcpSocket:
port: 2022
timeoutSeconds: 1
name: pgbackrest
resources: {}
securityContext:
allowPrivilegeEscalation: false
privileged: false
readOnlyRootFilesystem: true
runAsNonRoot: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/ssh
name: ssh
readOnly: true
- mountPath: /pgbackrest/repo1
name: repo1
- mountPath: /etc/pgbackrest/conf.d
name: pgbackrest-config
- mountPath: /tmp
name: tmp
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-k6f6j
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
hostname: hippo-repo-host-0
initContainers:
- command:
- bash
- -c
- NSS_WRAPPER_SUBDIR=postgres CRUNCHY_NSS_USERNAME=postgres CRUNCHY_NSS_USER_DESC="postgres"
/opt/crunchy/bin/nss_wrapper.sh
imagePullPolicy: IfNotPresent
name: nss-wrapper-init
resources: {}
securityContext:
allowPrivilegeEscalation: false
privileged: false
readOnlyRootFilesystem: true
runAsNonRoot: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /tmp
name: tmp
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-k6f6j
readOnly: true
nodeName: worker-3.cluster
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
runAsNonRoot: true
supplementalGroups:
- 65534
serviceAccount: default
serviceAccountName: default
subdomain: hippo-pods
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
operator: Exists
tolerationSeconds: 300
volumes:
- name: ssh
projected:
defaultMode: 32
sources:
- configMap:
name: hippo-ssh-config
- secret:
name: hippo-ssh
- name: repo1
persistentVolumeClaim:
claimName: hippo-repo1
- name: pgbackrest-config
projected:
defaultMode: 420
sources:
- configMap:
items:
- key: pgbackrest_repo.conf
path: pgbackrest_repo.conf
- key: config-hash
path: config-hash
name: hippo-pgbackrest-config
- emptyDir:
sizeLimit: 16Mi
name: tmp
- name: kube-api-access-k6f6j
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2021-09-27T17:29:38Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2021-09-27T17:29:06Z"
message: 'containers with unready status: [pgbackrest]'
reason: ContainersNotReady
status: "False"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2021-09-27T17:29:06Z"
message: 'containers with unready status: [pgbackrest]'
reason: ContainersNotReady
status: "False"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2021-09-27T17:29:06Z"
status: "True"
type: PodScheduled
containerStatuses:
lastState:
terminated:
exitCode: 1
finishedAt: "2021-09-27T19:43:16Z"
reason: Error
startedAt: "2021-09-27T19:43:16Z"
name: pgbackrest
ready: false
restartCount: 31
started: false
state:
waiting:
message: back-off 5m0s restarting failed container=pgbackrest pod=hippo-repo-host-0_postgres-operator(81dba4e8-d24c-45c9-b4c0-685ccdd783e0)
reason: CrashLoopBackOff
hostIP: 172.22.0.18
initContainerStatuses:
lastState: {}
name: nss-wrapper-init
ready: true
restartCount: 0
state:
terminated:
exitCode: 0
finishedAt: "2021-09-27T17:29:38Z"
reason: Completed
startedAt: "2021-09-27T17:29:38Z"
phase: Running
podIP: 10.128.5.19
podIPs:
- ip: 10.128.5.19
- ip: fd00:128::54c
qosClass: BestEffort
startTime: "2021-09-27T17:29:06Z"
--
You received this message because you are subscribed to a topic in the Google Groups "Postgres Operator" group.
To unsubscribe from this topic, visit https://groups.google.com/a/crunchydata.com/d/topic/postgres-operator/F4OdXSBP_Mg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to postgres-opera...@crunchydata.com.
Jean-Daniel
Sep 27, 2021, 3:48:48 PM9/27/21
to Andrew L'Ecuyer, Postgres Operator, jonath...@crunchydata.com
And by the way, this is where the unusual defaultMode (32 and 384) are defined:
Le 27 sept. 2021 à 21:43, Jean-Daniel <jdd...@xooloo.com> a écrit :kubectl get pod hippo-instance1-vnln-0 -o yaml
…
…volumes:- name: sshprojected:defaultMode: 32sources:- configMap:name: hippo-ssh-config- secret:name: hippo-ssh
- name: cert-volume
Andrew L'Ecuyer
Sep 27, 2021, 4:22:15 PM9/27/21
to Postgres Operator, Jean-Daniel, Postgres Operator, jonath...@crunchydata.com, Andrew L'Ecuyer
Thanks for the additional info. Looking at the Pod specs you provided, it appears as though the proper fsGroup setting is missing from the securityContext.
This could specifically be due to a false-positive with our OpenShift auto-detection logic, which controls the fsGroup setting.
If you look at the logs of the "pgo" Deployment (e.g. kubectl logs pgo-7d4d585cc7-h4b9p), do you see the following information log towards the top?
"detected OpenShift environment"
Also, can you try re-creating your PostgresCluster with the "openshift" setting explicitly set to "false", e.g.:
spec:
openshift: false
Thanks,
Andrew
Jonathan S. Katz
Sep 27, 2021, 4:30:58 PM9/27/21
to Andrew L'Ecuyer, Postgres Operator, Jean-Daniel
Separately, both of those default modes are on the projected volumes. 384 translates to "0600" in octal, so said items are only RW by the UID user and are writeable in events such as certificate rotation. 32 is 040, so it's only readable by the group. In general PGO typically gives out minimal privileges to accomplish tasks.
Jean-Daniel
Sep 27, 2021, 5:20:28 PM9/27/21
to Andrew L'Ecuyer, Postgres Operator, jonath...@crunchydata.com
Le 27 sept. 2021 à 22:22, Andrew L'Ecuyer <andrew....@crunchydata.com> a écrit :Thanks for the additional info. Looking at the Pod specs you provided, it appears as though the proper fsGroup setting is missing from the securityContext.This could specifically be due to a false-positive with our OpenShift auto-detection logic, which controls the fsGroup setting.If you look at the logs of the "pgo" Deployment (e.g. kubectl logs pgo-7d4d585cc7-h4b9p), do you see the following information log towards the top?"detected OpenShift environment"
Yes, It detects it as OpenShift:
level=info msg="detected OpenShift environment" file="cmd/postgres-operator/main.go:116" func=main.isOpenshift version=5.0.2-0
This is due to rook-ceph CRDs ( VolumeReplication and VolumeReplicationClasses with group: replication.storage.openshift.io )
They are causing similar issue with other softwares: https://github.com/rook/rook/issues/8573
Also, can you try re-creating your PostgresCluster with the "openshift" setting explicitly set to "false", e.g.:spec:openshift: false
Thank you very much, that fixed my issue :-)
Andrew L'Ecuyer
Sep 27, 2021, 5:23:03 PM9/27/21
to Postgres Operator, Jean-Daniel, Postgres Operator, jonath...@crunchydata.com, Andrew L'Ecuyer
Great to hear your issue has been resolved!
And thank you for providing the source of the false positive with the OpenShift auto-detection.
We will review further in order to determine a way to prevent similar false-positives in the future.
Thanks,
Andrew
On Monday, September 27, 2021 at 4:20:28 PM UTC-5 Jean-Daniel wrote:
Reply all
Reply to author
Forward
0 new messages