deleting older backups

[chandan kumar]

Hello,

I am using bup to backup my Virtual machine images (qemu images on CentOS). Now I do daily backup and every backup is stored incrementally I believe and hence saving space. Is there any way to have some kind of periodic cleanup or deleting older backup to keep the overall size of ~/.bup directory under check?

As of now the all backup looks like (output from bup web )

Name Size

..

2013-09-25-225556@  

2013-09-27-001040@  

2013-09-27-002802@  

2013-09-27-005405@  

2013-10-01-020002@  

2013-10-02-020002@  

2013-10-03-020003@  

2013-10-04-020002@  

2013-10-05-020002@  

2013-10-06-020002@  

2013-10-07-020003@  

2013-10-08-020002@  

2013-10-09-020003@  

2013-10-10-020002@  

2013-10-11-020002@  

2013-10-12-020002@  

2013-10-13-020002@  

2013-10-14-020002@  

2013-10-15-020002@  

2013-10-16-020003@  

2013-10-17-020002@  

2013-10-18-020003@  

2013-10-19-020002@  

2013-10-20-020002@  

2013-10-21-020003@  

2013-10-22-020003@  

2013-10-23-020004@  

2013-10-24-020004@  

latest@

Say if I want to keep only 3 days worth of backup and delete rest.

Thanks

Chandan

[Rob Browning]

chandan kumar <chandan...@gmail.com> writes:

> Is there any way to have some kind of periodic cleanup or deleting
> older backup to keep the overall size of ~/.bup directory under check?

Not yet, but that will be a high priority (which Zoran has been working
on) after we release 0.25.

For now, one option is to just create a new repository periodically and
eventually delete the older one(s).

i.e. have a bup-2011, bup-2012, ...

--
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

[Chandan Kumar]

Thanks Rob. I will see how I could automate it to delete say every week and create new repo.

--

http://about.me/chandank

[Zoran Zaric]

On Thu, Oct 24, 2013 at 03:08:51PM -0500, Rob Browning wrote:
> chandan kumar <chandan...@gmail.com> writes:
>
> > Is there any way to have some kind of periodic cleanup or deleting
> > older backup to keep the overall size of ~/.bup directory under check?
>
> Not yet, but that will be a high priority (which Zoran has been working
> on) after we release 0.25.

Chandan, if you want to explore unmerged stuff you can use my git delete script
[1] to delete single commits and remove all unneeded objects from your
repository with bup gc [2] afterwards.

Please try this on a copy of your repository only. This code hasn't been
reviewed and is potentially buggy with a high probability of data loss!

> For now, one option is to just create a new repository periodically and
> eventually delete the older one(s).

Rob, once deletion and purging are merged, we might think about a workflow of
merging bup repos.


Thanks,
Zoran


[1]: http://rantsideasstuff.com/git-delete-a-git-filter-branch-commit-filter-wrapper/
[2]: https://github.com/zoranzaric/bup/tree/tmp/gc

[Mark J Hewitt]

On 25/10/13 09:55, Zoran Zaric wrote:
> Rob, once deletion and purging are merged, we might think about a
> workflow of merging bup repos.

Isn't merging pretty straightforward in that essentially it is just
copying over the pack, midx and index files and recreating the bloom file.
Maybe repacking/gc is appropriate at that point too.

It might also be interesting to go the other way - splitting to multiple
repositories based on, perhaps, path prefix or even branch age.
That would be rather harder I expect.

Mark.

[Zoran Zaric]

On Fri, Oct 25, 2013 at 10:43:00AM +0100, Mark J Hewitt wrote:
> On 25/10/13 09:55, Zoran Zaric wrote:
> >Rob, once deletion and purging are merged, we might think about a
> >workflow of merging bup repos.
>
> Isn't merging pretty straightforward in that essentially it is just
> copying over the pack, midx and index files and recreating the bloom
> file.
> Maybe repacking/gc is appropriate at that point too.

Grafting the commits onto the older history would be necessary, too.


Thanks,
Zoran

[Rob Browning]

Zoran Zaric <z...@zoranzaric.de> writes:

> Rob, once deletion and purging are merged, we might think about a workflow of
> merging bup repos.

Right, and I'd imagined that pull/push commands might share some
relevant infrastructure as well.

[Tim Riemenschneider]

Am Freitag, 25. Oktober 2013 10:55:37 UTC+2 schrieb Zoran Zaric:

Chandan, if you want to explore unmerged stuff you can use my git delete script
[1] to delete single commits and remove all unneeded objects from your
repository with bup gc [2] afterwards.

I'm not 100% sure, but isn't it needed to use some "git pack-objects", or "git repack" (without -a/-A) to move the new commit-objects to a packfile?

Since IIRC at least with your older repack-branch it did not work otherwise (since it directly read the pack-files instead of using git catpipe)

However I didn't look into your branch, maybe it's not needed anymore.

This is the reason I use a new branch for each commit, named something like "home-2013-10-25" or even "home/2013/10/25" with the ref-with-slashes patch.

Then deletion is easy: "cd $BUP_DIR ; git branch -D old-branch". I then used normal git-gc / git-repack to reclaim space, but that needs a HUGE amount of memory.

I don't see much need for git-like history in those backups, and from the user-perspective, it doesn't make a difference if the backups are named "home/<date-time>" or "home/<splitted date>/latest"

But maybe there is some aspect in it that I currently see?

And merging would be easy then, too, simply copy over the refs as well (since their names are disjount. Or use directories/rename them). (Didn't try that....)

Of course for merging to make any sense, a repack is needed afterwards for deduplication between the branches.

Tim

[Zoran Zaric]

On Fri, Oct 25, 2013 at 01:23:41PM -0700, Tim Riemenschneider wrote:
>
>
> Am Freitag, 25. Oktober 2013 10:55:37 UTC+2 schrieb Zoran Zaric:
> >
> > Chandan, if you want to explore unmerged stuff you can use my git delete
> > script
> > [1] to delete single commits and remove all unneeded objects from your
> > repository with bup gc [2] afterwards.
> >
> >
> I'm not 100% sure, but isn't it needed to use some "git pack-objects", or
> "git repack" (without -a/-A) to move the new commit-objects to a packfile?
> Since IIRC at least with your older repack-branch it did not work otherwise
> (since it directly read the pack-files instead of using git catpipe)

Good point. I added a tmp/delete-commit branch [1] with my git-delete script as
`bup delete commit` [2] which now runs a `git repack` after deleting.

> However I didn't look into your branch, maybe it's not needed anymore.
> This is the reason I use a new branch for each commit, named something like
> "home-2013-10-25" or even "home/2013/10/25" with the ref-with-slashes patch.
> Then deletion is easy: "cd $BUP_DIR ; git branch -D old-branch". I then
> used normal git-gc / git-repack to reclaim space, but that needs a HUGE
> amount of memory.
> I don't see much need for git-like history in those backups, and from the
> user-perspective, it doesn't make a difference if the backups are named
> "home/<date-time>" or "home/<splitted date>/latest"
> But maybe there is some aspect in it that I currently see?

Currently bup doesn't really provide a benefit from having a commit history.
I think having a diff command in the future would be a nice addition.


Thanks,
Zoran

[1]: https://github.com/zoranzaric/bup/tree/tmp/delete-commit
[2]: https://github.com/zoranzaric/bup/blob/tmp/delete-commit/cmd/delete-commit-cmd.sh

[Olaf TNSB]

Hi Chandan,

I'm going to shamelessly plug my script here:
   https://github.com/resuscv/buply

It handles all the creating/using/rolling to a new repo that it sounds like you are looking for...

If you search this list for "buply", you'll find a discussion and a couple of other similar scripts too.

Cheers,

Olaf

--
You received this message because you are subscribed to the Google Groups "bup-list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bup-list+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Rob Browning]

Rob Browning <r...@defaultvalue.org> writes:

> chandan kumar <chandan...@gmail.com> writes:
>
>> Is there any way to have some kind of periodic cleanup or deleting
>> older backup to keep the overall size of ~/.bup directory under check?
>
> Not yet, but that will be a high priority (which Zoran has been working
> on) after we release 0.25.
>
> For now, one option is to just create a new repository periodically and
> eventually delete the older one(s).

And now there's another possibility -- the *very* experimental "bup get"
patches I posted (also in tmp/bup-get-v3). From the new manpage:

Assuming you have sufficient disk space (and until/unless bup supports
something like rm/gc), this command can be used to drop old, unwanted
backups by creating a new repository, fetching the saves that you want
to keep into it, and then deleting the old repository.

But don't trust it with any important data yet, and we may or may not
keep it.