Get tar archive of a bup save

[Nathaniel Hourt]

What is the best way to get a bup save as a simple tarball, notably
without copying the whole thing to disk first (bup restore) and
tarballing the result?

—
Nathaniel

[Rob Browning]

There's not really a good option right now other than restoring the
data and then archiving it, and if you do that, you might need to
either be root, or use something like fakeroot in order to preserve all
the permissions if that's important (say for a whole system).

Of course, passing through the local filesystem with the local users and
groups might not preserve them exactly (e.g. if the local system has
different user/group names/ids). Depending on the requirements, the
--numeric-ids (bup restore) and --numeric-owner (tar) options might be
relevant.

I do have a branch where I started exploring a tar export command, but I
haven't gotten back to it in a while.

It's also possible to use "bup fuse" to mount the repository and then
archive that (with similar considerations), but the current fuse stack
has some limitations. For example, it can't (or couldn't) represent
pre-epoch dates, and so we clamp them to UTC 0.

After the recent discussion about python-fuse not migrating to libfuse3,
I decided to see what might be involved in just using libfuse3 directly,
and at least for a prototype, it hasn't been too difficult, so we may
at least have improved fuse support, probably after 0.34.

--
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

[Nathaniel Hourt]

> It's also possible to use "bup fuse" to mount the repository and then
> archive that (with similar considerations), but the current fuse stack
> has some limitations. For example, it can't (or couldn't) represent
> pre-epoch dates, and so we clamp them to UTC 0.

This is the option I've gone with so far, and it's workable, albeit
cumbersome. If I want to parallelize the backup retrieval, I can't just
run multiple tar commands because the single bup instance is handling
both behind fuse, so I'm gonna have to make multiple fuse mounts, one
for each parallel retrieval. It isn't difficult, but it is unwieldy.

> After the recent discussion about python-fuse not migrating to
> libfuse3,
> I decided to see what might be involved in just using libfuse3
> directly,
> and at least for a prototype, it hasn't been too difficult, so we may
> at least have improved fuse support, probably after 0.34.

Looking forward to this! It's going to simplify my logistics immensely.
=)

—
Nathaniel

[Rob Browning]

"'Nathaniel Hourt' via bup-list" <bup-...@googlegroups.com> writes:

> Looking forward to this! It's going to simplify my logistics immensely.

It does appear to fix the timestamp issue, and now it'll also clean up
and unmount on C-c (SIGINT) which is more convenient when testing...

I suspect it might also be possible to improve the metadata support
further, e.g. linux attrs, xattrs, etc., but I haven't investigated, and
the first version will probably be about the same feature-wise.

Given current bup fuse, if more than just the basic stat data is
important, you'll want to go the bup restore -> tar route, with or
without root/fakeroot depending on your needs.

[Rob Browning]

Rob Browning <r...@defaultvalue.org> writes:

> I do have a branch where I started exploring a tar export command, but I
> haven't gotten back to it in a while.

I glanced back at this and had done more than I thought, but also
recalled that if we were to take the easy path, python's tar support
might not allow "full fidelity",
e.g. https://docs.python.org/3/library/tarfile.html#tarfile.TarInfo.mtime

Note that the type is either int or float, but what we actually have is
nanoseconds. I can't recall offhand whether I'd noticed other
limitations.

Ideally, at least for the timestamps, python would extend TarInfo the
same way it did stat_result, i.e. with separate ns fields:
https://docs.python.org/3/library/os.html#os.stat_result.st_mtime_ns

Though we might still fall afoul of the python stat value
"homogenization" issues we recently encountered:
https://github.com/bup/bup/commit/e5cb526940a583d8bd212f13d8bd4d989c66b3ac