S3 Backup not working

Graham Boyle

unread,

Oct 13, 2023, 2:12:42 PM10/13/23

to Postgres Operator

Hello

I'm having an issue getting backups to work to S3 (I can get them to work to a local volume fine)

The Environment is: Openshift 4.12.36
The Operator version is: 5.4.2

My YAML looks like this, I create the Cluster from scratch using this YAML (IE I have not modified it):
spec:
port: 5432
monitoring:
pgmonitor:
exporter:
image: >-
registry.connect.redhat.com/crunchydata/crunchy-postgres-exporter@sha256:e78286ec8a35eabc0e9a5478139f07157656c0d77a55fd6fb8efe880796eeecc
users:
- name: postgres
password:
type: ASCII
- databases:
- hippo
name: hippo
password:
type: ASCII
proxy:
pgBouncer:
port: 5432
replicas: 2
service:
type: LoadBalancer
backups:
pgbackrest:
configuration:
- secret:
name: pgo-s3-creds
global:
repo1-retention-full: '14'
repo1-retention-full-type: time
image: >-
registry.connect.redhat.com/crunchydata/crunchy-pgbackrest@sha256:890e2c5d5501ef74a750b761c3fb37b6a6d09849525ff46a32aa9ded6d2b16fe
manual:
options:
- '--type=full'
repoName: repo1
repos:
- name: repo1
s3:
bucket: lab-postgres-backups
endpoint: mys3.mysub.mydomain.com
region: none
service:
type: LoadBalancer
openshift: true
image: >-
registry.connect.redhat.com/crunchydata/crunchy-postgres@sha256:57c3a8aaa67dea40c38880f7aba833dec0f3b26c5c8102bf24b1999af5345aca
instances:
- affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
postgres-operator.crunchydata.com/cluster: hippo15s3prim
postgres-operator.crunchydata.com/instance-set: pgha1
topologyKey: kubernetes.io/hostname
dataVolumeClaimSpec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
name: pgha1
replicas: 2
postgresVersion: 15

I created the secret hash by doing:
echo "[global]
repo1-s3-key=PSFBSAZUTHHDDCJLOEOFBALLOBEHNOPKJMCLABHIBA
repo1-s3-key-secret=DAB410B2C9uied741+16a3/B8DCA57B75cf7e47NGTS" | base64

And then I stored the output in secret I created called pgo-s3-creds, when I reveal the values, the data looks correct.

There are no backups happening though. and when I try to trigger a manual backup nothing happens. I've verified the S£ creds are good but placing some data in there myself

LAB [gboyle@server01 ~]$s3cmd -c /home/gboyle/s3_keys/postgres_backup_key ls s3://lab-postgres-backups
2023-10-11 11:24 15 s3://lab-postgres-backups/gbs3text.txt

I cannot find logs anywhere. Any idea what I'm doing wrong or where to look for the logs?

Regards
Graham

Graham Boyle

unread,

Oct 16, 2023, 10:25:01 AM10/16/23

to Postgres Operator, Graham Boyle

I got this going in the end:

The issue was that pgbackrest did not like the cert being presented by the S3 server

the fix is:

global:
repo1-retention-full: '14'
repo1-retention-full-type: time

repo1-path: "/pgbackrest/hippo15s3prim/repo1"
repo1-s3-verify-ssl: 'n'

Either that or import the CA certs and reference them in Global.
TLS slows things down significantly so I'm happy enough to have this as our setup is all internal.

Backups working now
G.

Andrew L'Ecuyer

unread,

Oct 16, 2023, 1:55:50 PM10/16/23

to Postgres Operator, Graham Boyle

Glad to hear your backups are now working!

I will note that pgBackRest does include a "repo-storage-ca-file" setting that can be utilized to provide a custom CA for your S3 server. I recommend checking out the following docs for additional information about this setting, as well as how to configure backups for a PostgresCluster:

And if you would like to continue this conversation, and/or if have any additional questions about backups, I recommend reaching out via the #backups channel in our new Discord server:

https://discord.com/invite/a7vWKG8Ec9

Thanks!

Andrew

Reply all

Reply to author

Forward