[cluster]-stanza-create pods unable to create path /backrestrepo/[cluster]-backrest-shared-repo

[Jeff Maaks]

Every time I create a new cluster, everything appears to work fine except for the [cluster]-stanza-create-* pods, which all show and error of being able to create path /backrestrepo/[cluster]-backrest-shared-repo.

I can see from inside the test-backrest-shared-repo pod that /backrestrepo is mounted, and it's owned by UID 5000 and group postgres, with drwxrwsr-x permissions.

id shows:

uid=2000(pgbackrest) gid=2000(pgbackrest) groups=2000(pgbackrest),26(postgres)

However, when I try "touch /backrestrepo/test" I get a permission denied error.

Any ideas why that one PV appears to have permission problems?  /pgdata seems to be mounting fine.

[Jonathan S. Katz]

Hello Jeff,

To understand what is happening I'll need to see a bunch more information to get a better understanding of your environment, including:

- Platform: (`Kubernetes`, `OpenShift`, `Rancher`, `GKE`, `EKS`, `AKS` etc.)

- Platform Version: (e.g. `1.20.3`, `4.7.0`)

- PGO Image Tag: (e.g. `centos8-4.7.0`)

- Postgres Version (e.g. `13`)

- Storage: (e.g. `hostpath`, `nfs`, or the name of your storage class)

Additionally, I would need to see what command you used to create your Postgres cluster.

Thanks,

Jonathan

[Jeff Maaks]

BTW, the instructions I followed were: https://www.crunchydata.com/developers/download-postgres/containers/postgres-operator

[Jeff Maaks]

Hi Jonathan,

Absolutely:

- Platform & Version: Ezmeral Container Platform - Kubernetes 1.18.6

- PGO Image Tag: centos8-4.7.0

- Postgres Version: crunchy-postgres-ha:centos8-13.3-4.7.0

- Storage: default storageclass

I used "pgo create cluster test" to create the cluster.  The database itself seems to come up fine and /pgdata looks to be ok, it's just /backrestrepo PV that seems to have permission problems.

Let me know if you need anything else...thanks!

Jeff

On Wednesday, May 26, 2021 at 10:29:22 AM UTC-5 jonath...@crunchydata.com wrote:

[Jonathan S. Katz]

Hi Jeff,

Thanks for the info.

My initial reaction is that the error is either at the storage level or something in your Kubernetes environment that is disallowing the proper UID from being set. The pgBackRest repo directory should be owned by UID 26 (postgres).

Are you able to exec into the Postgres Pod and run "pgbackrest info"?

Jonathan

Jonathan S. Katz

VP Platform Engineering

Crunchy Data
Enterprise PostgreSQL 

www.crunchydata.com

[Jeff Maaks]

Hi Jonathan,

Yes, I'm coming to that same conclusion.  Some other points:

• The in the test-backrest-share-repo pod the backrestrepo directory is owned by uid=5000 (which is the user that corresponds to the service providing the default storageclass I'm using) and gid=26(postgres).
• However, I see here https://github.com/CrunchyData/postgres-operator/issues/2172 that you said "Note that the pgBackRest repository requires the filesystem to be owned by a UID 2000. I would ensure that the pgBackRest repository Pod is able to do so."
• Regardless, when I exec into the test-backrest-share-repo- pod and run id, the pbbackrest user shows to be in groups pgbackrest,postgres, but it still is unable to create content under /backrestrepo

If that dir must be owned by uid=2000, is there a way to create an init container for the test-backrest-shared-repo pod that fixes the permissions?

Or is there something more obvious I'm missing?

Thanks!

[Jeff Maaks]

Sorry, I missed replying to your question.  Yes, I can run that command and it returns:

stanza: db

       status: error (missing stanza path)

On Thursday, May 27, 2021 at 9:39:51 PM UTC-5 jonath...@crunchydata.com wrote:

[Jeff Maaks]

Update: I was able to get the stanza jobs to complete by switching to an NFS PVC for the BackrestStorage.