Barman cloud restore is throwing error in kuberentes environment.

[Gaurav Oberai]

Hi Team,

I am using the Enterprise DB Postgres cluster in the Kubernetes environment and the EDB Postgres cluster is using Barman cloud backup to store the data backups to OpenStack swift s3 storage. I am also able to store the backups successfully without any errors. 

But when Barman Cloud restore is restoring the data backups to bring up another Postgres cluster, it throws the below error and says it is not a gzip file. error{"level":"info","ts":"2023-08-01T07:31:05Z","logger":"barman-cloud-restore","msg":"2023-08-01 07:31:05,944 [16] ERROR: Barman cloud restore exception: not a gzip file","pipe":"stderr","logging_pod":"edb-primary-cluster-1-full-recovery"}

I can see backups are available in OpenStack swift s3 storage in compressed format(i.e. gzip). Could you please let me know what is the issue?

Thanks 

Gaurav

[Mike Wallace]

Hi Gaurav,

Thanks for the report - I have been trying to reproduce this issue using barman cloud with a containerised OpenStack Swift instance but have not had any success so far.

Have you tried manually downloading the gzipped files in the Swift store for the backup you are trying to restore and decompressing them locally with gzip? This would help determine whether there is an issue with the gzipped objects themselves.

To help further efforts to reproduce, can you please confirm:

1. Whether you are using EDB Postgres for Kubernetes [1], CloudNativePG [2] or something else.

2. The exact version of the above tool you are using.

3. The system architecture in use (e.g. linux/amd64).

Thanks,

Mike

--
--
You received this message because you are subscribed to the "Barman for PostgreSQL" group.
To post to this group, send email to pgba...@googlegroups.com
To unsubscribe from this group, send email to
pgbarman+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/pgbarman?hl=en?hl=en-GB

---
You received this message because you are subscribed to the Google Groups "Barman, Backup and Recovery Manager for PostgreSQL" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pgbarman+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/pgbarman/9a4b4bdf-efd0-4346-b59b-d17459addf84n%40googlegroups.com.

[Gaurav Oberai]

Thanks, 

Mike for the reply. Yes, I downloaded the below backup file from the OpenStack swift object server manually and tried to extract the file but it 

gave me an error. It seems the file is corrupted and even size is not correct.  My only concern is why I don't see any error in the backup logs about 

data corruption/failure. If it is corrupted, then the Braman cloud Backup tool running inside the Postgres cluster kuberentes Pod should through an error but it completed successfully.

kma001:~$ gzip -t data.tar.gz

gzip: data.tar.gz: not in gzip format

1. Whether you are using EDB Postgres for Kubernetes [1], CloudNativePG [2] or something else. ---  Yes, I using cloudnative-pg which

is the community version (i.e. open-source) of EDB Postgres for kubernetes. Here is the link: https://cloudnative-pg.io/documentation/1.20/

2. The exact version of the above tool you are using. ---- working in the kubernetes environment so, I am using the below versions.

Docker images : 

Postgis: ghcr.io/cloudnative-pg/postgis:11-3.3-64
Operator: ghcr.io/cloudnative-pg/cloudnative-pg:1.20.0

Barman tools version:

Barman-cloud-backup:    3.6.0
Barman-cloud-restore:    3.6.0

3. The system architecture in use (e.g. linux/amd64).: Linux

Please let me know if you need for details.

Thanks & Regards,

Gaurav

[Mike Wallace]

Thanks for the additional information.

Can you confirm whether or not an object named `backup.info` exists
for that backup? It is a plain text file which should be under the
same prefix as the data.tar.gz object. If it exists could you please
share its content here?

Thanks,

Mike

> To view this discussion on the web, visit https://groups.google.com/d/msgid/pgbarman/db31931d-4ac0-4249-8d21-b37f20a29134n%40googlegroups.com.

[Gaurav Oberai]

Sure, Here it is:

kma001:~$ cat backup.info backup_label='START WAL LOCATION: 0/7000028 (file 000000010000000000000007)\nCHECKPOINT LOCATION: 0/7000060\nBACKUP METHOD: streamed\nBACKUP FROM: master\nSTART TIME: 2023-08-04 06:08:38 UTC\nLABEL: Barman backup cloud 20230804T060837\nSTART TIMELINE: 1\n' backup_name=backup-1691129316 begin_offset=40 begin_time=2023-08-04 06:08:37.828972+00:00 begin_wal=000000010000000000000007 begin_xlog=0/7000028 compression=None config_file=/var/lib/postgresql/data/pgdata/postgresql.conf copy_stats={'total_time': 17.392676, 'number_of_workers': 2, 'analysis_time': 0, 'analysis_time_per_item': {'data': 0}, 'copy_time_per_item': {'data': 0.943551}, 'serialized_copy_time_per_item': {'data': 0.685219}, 'copy_time': 0.943551, 'serialized_copy_time': 0.685219} deduplicated_size=None end_offset=248 end_time=2023-08-04 06:08:49.978710+00:00 end_wal=000000010000000000000007 end_xlog=0/70000F8 error=None hba_file=/var/lib/postgresql/data/pgdata/pg_hba.conf ident_file=/var/lib/postgresql/data/pgdata/pg_ident.conf included_files=['/var/lib/postgresql/data/pgdata/custom.conf'] mode=None pgdata=/var/lib/postgresql/data/pgdata server_name=cloud size=None status=DONE systemid=7263343371348295700 tablespaces=None timeline=1 version=110020 xlog_segment_size=16777216

[Mike Wallace]

The backup.info contents suggests the backup completed successfully
(at least as far as barman-cloud-backup was concerned). If there had
been an error during the upload process then the backup.info status
field would be set to FAILED or the file would not have been created
at all.

Is this scenario consistently reproducible? If this is happening when
you attempt to restore every backup then it would be worth trying a
manual backup/restore directly on one of the pods with verbose logging
- I don't know how possible this is in your environment since you
would need to get a shell on a running container to do this.

> To view this discussion on the web, visit https://groups.google.com/d/msgid/pgbarman/ace07643-8bc0-4fe8-a110-7d839719221an%40googlegroups.com.

[Gaurav Oberai]

Thanks for looking into it.
Is this scenario consistently reproducible? ---- Yes, for each backup it is happening. 

I can try to restore it manually inside one of the Postgres cluster pods. Could you please share the commands to restore it manually?

I mean the Barman cloud backup/restore commands to create and restore backup manually inside the pod.

[Gaurav Oberai]

Hi Mike, 

It would be more helpful if you could guide me by providing the commands to perform the backup/restore manually.

Thanks & Regards,

Gaurav

[Mike Wallace]

Hi Gaurav,

I put some notes together on how to run backup/restore within a container provisioned by CNPG here (tested using the example cluster in the CNPG documentation with the PostGIS 11 image): https://gist.github.com/mikewallace1979/da9722e56d22db06658017ee72f32aeb

If you run both backup and restore with the `-vv` option then the verbose logging should hopefully provide some clues as to what might be going wrong.

Best,

Mike

To view this discussion on the web, visit https://groups.google.com/d/msgid/pgbarman/a2e7ff64-63c7-4a90-941b-ad1b7dbd149cn%40googlegroups.com.

[Gaurav Oberai]

Thanks, Mike. I will try it.