Metadata support?

[Michael Witten]

I have used `bup' in the past to split and join an image of my hard
disk for the purposes of a crude backup solution. Obviously, this is
much less efficient and much more restrictive than backing up via
`bup index', `bup save', and `bup restore'. However, I've never felt
comfortable with bup's level of support for metadata (and I suppose
special files), which are data (and filesystem objects) that I
consider essential for a backup system to support.

Unfortunately, I can't tell if [detailed] metadata support is progressing.

What exactly is the state of that work?
What is left to be done?
What are the challenges?
What's going on with branch head `origin/pending/meta'?
Is `bup meta' the intended solution? Could we have more examples?

I'd like to see such information clearly spelled out in one location,
such as in a document somewhere in the repo.

Sincerely,
Michael Witten

[Gabriel Filion]

On 12-04-13 01:48 AM, Michael Witten wrote:
> I have used `bup' in the past to split and join an image of my hard
> disk for the purposes of a crude backup solution. Obviously, this is
> much less efficient and much more restrictive than backing up via
> `bup index', `bup save', and `bup restore'. However, I've never felt
> comfortable with bup's level of support for metadata (and I suppose
> special files), which are data (and filesystem objects) that I
> consider essential for a backup system to support.

metadata support is scheduled to mark the next release.

> Unfortunately, I can't tell if [detailed] metadata support is progressing.
>
> What exactly is the state of that work?

Rob Browning has been working on a patchset to add support for metadata,
for quite some time now, and the patchset should be nearing maturity for
inclusion.

Check it out here:

http://anonscm.debian.org/gitweb/?p=users/rlb/bup.git;a=summary

(branch name for metadata support is named "tmp/pending/meta")

Please be advised: this work is still in progress and could be affected
by bugs.. so make sure you have a working archive of your data.
Although, I've been using the branch for my personal backups for about 2
or 3 weeks now and it's been working nicely since then.

> What is left to be done?

Support for hard links.
Rob currently has this on his boiler plate and is trying to get this in
as soon as possible so that we can review and test the whole thing as
much as possible, and then push Avery into reviewing / merging the set.

> What are the challenges?

We need testers! the more the merrier. we need to catch ugly and/or
sneaky bugs and to make them go away.

> What's going on with branch head `origin/pending/meta'?

This is an (quite) old state of Rob's patch set. It's not up to date and
is not reflecting the latest state of how it's working. You should check
out the branch I pointed out above.

> Is `bup meta' the intended solution? Could we have more examples?

'bup meta' is a "plumbing" command. It's intended to be called by
"porcelain" commands such as bup-save and bup-restore.

> I'd like to see such information clearly spelled out in one location,
> such as in a document somewhere in the repo.

Rob's branch adds a man page for bup-meta, and some information into
bup-restore's man page about metadata. There are options for metadata
that are documented in the 'restore' man page.

Unfortunately, before we get the patch set merged into Avery's
repository, there won't be much about it in the project's "main" repository.

If you want very precise examples of how to save/restore data with
metadata, we can show you series of commands that'll do what you'd like.
Also, if you need help with getting Rob's branch working on your setup,
don't hesitate to ask.

--
Gabriel Filion

[Michael Witten]

On Fri, Apr 13, 2012 at 06:07, Gabriel Filion <lel...@gmail.com> wrote:

> Check it out here:
>
> http://anonscm.debian.org/gitweb/?p=users/rlb/bup.git;a=summary
>
> (branch name for metadata support is named "tmp/pending/meta")

Commit b8dc42c6101a8fe25885c0e60fd3c37a08d949bc fails a test:

t/test-meta.sh:105 [...] FAILED
[...]
WvTest: 2677 tests, 1 failure [...]

To my surprise, that commit is from 24 March; perhaps, Rob, you've
got some more recent changes to push?

[Gabriel Filion]

oh... that commit was actually made to fix a wobbly test.

I've just run the whole test suite against ubuntu maverick / python
2.6.6 and everything went ok.

what os and python version are you using?

does it error out on every run?

can you paste the output of the failing test?

--
Gabriel Filion

[Rob Browning]

Michael Witten <mfwi...@gmail.com> writes:

> To my surprise, that commit is from 24 March; perhaps, Rob, you've
> got some more recent changes to push?

Nothing significant yet. Can you run t/test-meta.sh directly and see
what actually fails (presuming you can't see the error already)?

Thanks
--
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

[Treeve Jelbert]

On Saturday 14 April 2012 12:56:03 Rob Browning wrote:
> Michael Witten <mfwi...@gmail.com> writes:
> > To my surprise, that commit is from 24 March; perhaps, Rob, you've
> > got some more recent changes to push?
>
> Nothing significant yet. Can you run t/test-meta.sh directly and see
> what actually fails (presuming you can't see the error already)?
>

Itried tunning the latest git tmp/pending/meta to backup a winxp drive from
linux,
I got the following error:

Indexing: 41227, done.
Reading index: 41227, done.
Traceback (most recent call last):
File "/usr/lib/bup/cmd/bup-save", line 284, in <module>
metalists[-1].append((sort_key, metadata.from_path(ent.name)))
File "/usr/lib/bup/bup/metadata.py", line 673, in from_path
result._add_linux_attr(path, st)
File "/usr/lib/bup/bup/metadata.py", line 486, in _add_linux_attr
attr = get_linux_file_attr(path)pyxattr
OSError: [Errno 38] Function not implemented:
'/mnt/wxp/spoolerlogs/spooler.xml'

I have pyxattr and pylibacl 0.5.0 installed

Regards
> Thanks

[Rob Browning]

Treeve Jelbert <tre...@scarlet.be> writes:

> Itried tunning the latest git tmp/pending/meta to backup a winxp drive
> from linux, I got the following error:

> File "/usr/lib/bup/bup/metadata.py", line 486, in _add_linux_attr

> attr = get_linux_file_attr(path)pyxattr
> OSError: [Errno 38] Function not implemented:

So that comes via _helpers.c:bup_get_linux_file_attr(). I assume it's
returning ENOSYS when the underlying filesystem doesn't support attrs.

In the caller, metadata.py:_add_linux_attr() we already check for ENOTTY
"Inappropriate I/O control operation (POSIX.1)", which is what I thought
was returned in this situation. (I don't recall for sure, but I think I
either looked at the relevant source or ran a test.)

So either I was wrong, or both values are possible. For now, I'll add a
check for both. If you want to test immediately, just change
metadata.py to look like this:

elif e.errno == errno.ENOTTY or e.errno == errno.ENOSYS:
# Assume filesystem doesn't support attrs.
return

[Treeve Jelbert]

On Sunday 15 April 2012 17:38:27 Rob Browning wrote:
> Treeve Jelbert <tre...@scarlet.be> writes:
> > Itried tunning the latest git tmp/pending/meta to backup a winxp drive
> >
> > from linux, I got the following error:
> > File "/usr/lib/bup/bup/metadata.py", line 486, in _add_linux_attr
> >
> > attr = get_linux_file_attr(path)pyxattr
> >
> > OSError: [Errno 38] Function not implemented:
> So that comes via _helpers.c:bup_get_linux_file_attr(). I assume it's
> returning ENOSYS when the underlying filesystem doesn't support attrs.
>
> In the caller, metadata.py:_add_linux_attr() we already check for ENOTTY
> "Inappropriate I/O control operation (POSIX.1)", which is what I thought
> was returned in this situation. (I don't recall for sure, but I think I
> either looked at the relevant source or ran a test.)
>
> So either I was wrong, or both values are possible. For now, I'll add a
> check for both. If you want to test immediately, just change
> metadata.py to look like this:
>
> elif e.errno == errno.ENOTTY or e.errno == errno.ENOSYS:
> # Assume filesystem doesn't support attrs.
> return
>

saves ok from both ntfs-3g and ext3 partitions

> Thanks

[Gabriel Filion]

On 12-04-14 03:09 PM, Treeve Jelbert wrote:
> Itried tunning the latest git tmp/pending/meta to backup a winxp drive from
> linux

Is the partition formatted with NTFS or FAT32 ?

--
Gabriel Filion

[Treeve Jelbert]

NTFS

[Niklas Hambüchen]

On 04/13/2012 08:07 AM, Gabriel Filion wrote:
> metadata support is scheduled to mark the next release.

Today I came across
http://www.n8gray.org/blog/2007/04/27/introducing-backup-bouncer/, which
apparently has been discussed shortly last year.

What's the state of it? Using something like this seems to be great to
make sure metadata backups really work.

[Gabriel Filion]

it sure looks pretty nice. but judging from the text on the page, it's
only for OS X. We'd need to have that same functionality for linux,
*BSD, and Solaris, too.

There was some talk about porting backup bouncer to other platforms.
does anyone know about the status of that?

--
Gabriel Filion

[Rob Browning]

Thanks for testing -- I've added this fix to my tmp/pending/meta branch.