WAL-E pushing to separate daily and weekly S3 buckets

[Max Wendel]

Hi all...

Just wondering of it's OK to push backups of the same db to different S3 buckets while having pg wal archive mode on in postgres.conf and archive_command pointing to just one of those buckets...

i.e. My setup is as follows (shortened for brevity):

cron jobs:

00 20 * * 0   wal-e --s3-prefix=s3://weekly-bucket delete --confirm retain 13
00 20 * * 1-6 wal-e --s3-prefix=s3://daily-bucket delete --confirm retain 11

00 21 * * 0   wal-e --s3-prefix=s3://weekly-bucket backup-push /pgdir
00 21 * * 1-6 wal-e --s3-prefix=s3://daily-bucket backup-push /pgdir

postgresql.conf:
archive_command = wal-e --s3-prefix=s3://daily-bucket wal-push %p

So I am currently running a Sunday weekly backup (kept for 3 months) in between the daily backups (kept for 11 days)...

The main problem I see is that I would have some WAL files belonging to the weekly backup being pushed by postgres to the daily bucket via wal-push in postgres.conf.

If I don't really need WAL files in the weekly backup, am I likely to run into other problems? Are these "foreign" WAL files inside the daily bucket going to be deleted eventually? Do they interfere with the "proper" daily backups?

Also, am I likely to run into catalogue issues? Or restoring problems etc?

In a nutshell, is it OK or advisable to use WAL-E like this?
Or am I using it outside spec, "wandering off into uncharted territory", etc?

Would there be a better way to achieve the same / or similar result?

Any advice would be greatly appreciated...
Many thanks in advance...

Regards

Max

[Daniel Farina]

On Nov 23, 2014 9:50 PM, "Max Wendel" <m...@profilercorp.com> wrote:
> If I don't really need WAL files in the weekly backup, am I likely to run into other problems? Are these "foreign" WAL files inside the daily bucket going to be deleted eventually? Do they interfere with the "proper" daily backups?

This is not correct. You need WAL to be able to restore a backup, hence, your backups that use a different prefix than the archive command are incomplete unless you happen to have the retention of WAL somewhere else.  As you are deleting "daily" more aggressively than "weekly", that doesn't seem like it's going to happen.

I would like some form of generational backup retention, but am not sure how to write the UI for it, so for now I think you are left writing a bespoke program to do your own pruning.