Managing cloud-stored backup copies

54 views
Skip to first unread message

SteveB

unread,
May 13, 2022, 4:18:08 PM5/13/22
to Barman, Backup and Recovery Manager for PostgreSQL
Hello,

We are new to Barman and new to this group, researching different ways to store and manage redundant off-site copies of Barman backups.  One option we are considering is to use hook scripts to forward copies of base backups and WAL files to an aws-s3 compliant store (Cloudian).

This is probably a newbie question, but can someone confirm -- does the scope of the server configuration (retention policy and other parameters) include management of files stored in the cloud?  Or do we need to manage file retention of the cloud files separate from Barman?  I suspect it is the second case but I want to be sure.

Best regards,
Stephen Barrett

Michael Wallace

unread,
May 16, 2022, 4:56:52 AM5/16/22
to pgba...@googlegroups.com
Hello Stephen,

Barman will not directly apply its retention policy to files uploaded to cloud storage via post backup hook scripts. However, if you were to add a pre or post backup delete script which deleted the files in cloud storage for a given backup then this script would be invoked whenever a Barman backup is deleted, whether the deletion is manual or as a result of a retention policy being applied. The same is true for the pre/post WAL delete scripts - the scripts will run whenever WALs are deleted by Barman server due to the configured retention policy.

Hope this helps,

Mike



--
--
You received this message because you are subscribed to the "Barman for PostgreSQL" group.
To post to this group, send email to pgba...@googlegroups.com
To unsubscribe from this group, send email to
pgbarman+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/pgbarman?hl=en?hl=en-GB

---
You received this message because you are subscribed to the Google Groups "Barman, Backup and Recovery Manager for PostgreSQL" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pgbarman+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/pgbarman/5677fc9b-5b4a-4f2c-8584-e77319d3822fn%40googlegroups.com.

SteveB

unread,
May 16, 2022, 4:33:26 PM5/16/22
to Barman, Backup and Recovery Manager for PostgreSQL
Hi Mike,

Yes, this helps a lot.  Thanks for confirming the behavior, and it makes sense to use delete hook scripts as you suggest.

We were considering having a shorter retention period in Barman (<7 days) and a longer retention period for the replicated files in the cloud (30 days), using separate retention periods (one in Barman and one in the Cloudian bucket).  If the retention period were the same for both, delete hook scripts would be the way to go I would think.

Best regards,
Stephen

SteveB

unread,
May 24, 2022, 11:45:05 AM5/24/22
to Barman, Backup and Recovery Manager for PostgreSQL
Hello,

Following up on forwarding a backup copy to cloud storage via hook scripts -- for example, using post_backup_retry_script to execute barman-cloud-backup -- will the copies of the backup files be packed into a tarball before storage in the cloud?  Or does a tarball only get generated by barman-cloud-backup when backing up directly into cloud storage?

Thanks,
Stephen

Michael Wallace

unread,
May 25, 2022, 1:15:56 AM5/25/22
to pgba...@googlegroups.com
Hi Stephen,

If you run barman-cloud-backup as a post_backup_retry_script it will generate the same tarballs as it would when running directly on the PostgreSQL server - internally it's a very similar operation, the main difference being the source is the base backup directory on a Barman server rather than the PGDATA and tablespace locations on a PostgreSQL server.

Best,

Mike

Ravi Chauhan

unread,
Jun 19, 2022, 4:24:19 AM6/19/22
to Barman, Backup and Recovery Manager for PostgreSQL
Hi All,

I have a similar question so I am asking it on this thread. I hope no one minds.

I am planning to use barman-cloud only which will upload the backups and walls directly to Azure blob storage. 
Question - How to apply retention policy on wals and backups taken through barman-cloud-backup and barman-cloud-wal-archive

Ravi

Ravi Chauhan

unread,
Jun 19, 2022, 5:01:05 PM6/19/22
to Barman, Backup and Recovery Manager for PostgreSQL
Another Question -

What if I use BLOBFuse to mount Azure Blob Storage on barman server and take backup to the new mount point. Will barman be able to perform and apply retention policy as it is suppose to.
I will be performing this whole activity to take backup on Azure blob storage using Barman and Not by Barman Cloud for testing -
  1. The efficiency and performance using RAMDISK.
  2. Check if retention policies are working as it is suppose to work on the local mount points.
  3. How much time will it take in case of disaster recovery scenarios.
I am looking for any expert comments on this.

Thanks,
Ravi Chauhan
Reply all
Reply to author
Forward
0 new messages