Feature requst: Flush pending bundles from tmp to bundles
52 views
Skip to first unread message
jo...@telpit.de
Mar 3, 2015, 5:15:22 AM3/3/15
to zba...@googlegroups.com
I'm trying to perform a backup (`tar c | zbackup`) without having enough local disk space to store all of the newly created bundle files. What I'd like to do is:
- write out a "part" of the bundle files of the backup (let's say half of it)
- (externally) upload the files in `bundles/` to remote storage
- delete the local copies in `bundles/` to free disk space
- write the next bundle part
I'm not sure if ZBackup even needs to be aware of this. I'm currently thinking of two ways to solve this:
1. Add some sort of signal handler to ZBackup (i.e. SIGUSR2) to tell it to flush all the (finished) bundle files. I'm not sure if this is even possible (do bundle files still get appended to after some point?) and it could leave the repo with inconsistent state.
2. Solve this externally by doing "partial backups". I.e. write a program like `tar c | splitter-zbackup` that reads the incoming tar stream, pipes it into zbackup, but aborts the zbackup child at some point. zbackups then just sees a smaller file and is happy. Then, start a new zbackup continuing the still-open tar stream. (Assume the tar stream cannot be rewound)
I quite like solution (2), however, it comes with the disadvantage that at no point I ever store the entire tar archive with zbackup. I could just re-open and pipe the tar stream into zbackup again (should write no more data because redundancy). Maybe zbackup could help here by allowing to "append to" an already-existing archive file, acting as though the entire base file's content was piped in just before the new part.
Thoughts?
- write out a "part" of the bundle files of the backup (let's say half of it)
- (externally) upload the files in `bundles/` to remote storage
- delete the local copies in `bundles/` to free disk space
- write the next bundle part
I'm not sure if ZBackup even needs to be aware of this. I'm currently thinking of two ways to solve this:
1. Add some sort of signal handler to ZBackup (i.e. SIGUSR2) to tell it to flush all the (finished) bundle files. I'm not sure if this is even possible (do bundle files still get appended to after some point?) and it could leave the repo with inconsistent state.
2. Solve this externally by doing "partial backups". I.e. write a program like `tar c | splitter-zbackup` that reads the incoming tar stream, pipes it into zbackup, but aborts the zbackup child at some point. zbackups then just sees a smaller file and is happy. Then, start a new zbackup continuing the still-open tar stream. (Assume the tar stream cannot be rewound)
I quite like solution (2), however, it comes with the disadvantage that at no point I ever store the entire tar archive with zbackup. I could just re-open and pipe the tar stream into zbackup again (should write no more data because redundancy). Maybe zbackup could help here by allowing to "append to" an already-existing archive file, acting as though the entire base file's content was piped in just before the new part.
Thoughts?
Alex Sayers
Mar 3, 2015, 10:57:49 AM3/3/15
to jo...@telpit.de, zba...@googlegroups.com
Thoughts?
A third suggestion: add the ability to create a new backup by merging a number of existing backups. I think it's better to keep zbackup non-destructive if possible.
All the best,
Alex
Jonas Schneider
Mar 3, 2015, 11:06:25 AM3/3/15
to Alex Sayers, zba...@googlegroups.com
Cool, yeah, what I was referring to by appending was a new header
field/backup instruction à la "when restoring, output that backup's
contents before mine" -- your idea takes this one step further.
When I get a chance, I'll see if I can't hack something together.
- Jonas
field/backup instruction à la "when restoring, output that backup's
contents before mine" -- your idea takes this one step further.
When I get a chance, I'll see if I can't hack something together.
- Jonas
Reply all
Reply to author
Forward
0 new messages