release status?

295 views
Skip to first unread message

Thomas Klausner

unread,
Apr 30, 2012, 4:12:46 AM4/30/12
to bup list
Hi!

I've lost track a bit about what the current release plans for bup
are. The last tag in the official git (apenwarr) I see is for 0.25rc1
from last June. From the mailing list traffic lots of stuff happened
since then.

So my questions are:

Is there a semi-official git where these are tracked? I gather Rob has
one like that but I don't see it immediately on
https://github.com/apenwarr/bup/network

What's the plan for a new release?

Thanks,
Thomas

Avery Pennarun

unread,
Apr 30, 2012, 4:34:47 AM4/30/12
to Thomas Klausner, bup list
On Mon, Apr 30, 2012 at 4:12 AM, Thomas Klausner <t...@giga.or.at> wrote:
> I've lost track a bit about what the current release plans for bup
> are. The last tag in the official git (apenwarr) I see is for 0.25rc1
> from last June. From the mailing list traffic lots of stuff happened
> since then.

Realistically I should just re-tag that one as 0.25 :) I don't think
there were too many serious problems with it.

The main problem is that I've been overwhelmed with doing too many
projects at once and haven't had time to keep up.

> So my questions are:
>
> Is there a semi-official git where these are tracked? I gather Rob has
> one like that but I don't see it immediately on
> https://github.com/apenwarr/bup/network
>
> What's the plan for a new release?

There's no plan, but maybe it's time to make one. Given my
overworked-ness, it might be smart to just elect someone as the new
"official" maintainer and patch collector. I still have lots of
things I want to do with bup, but obviously I haven't been doing most
of them lately, and I feel like people are worse off for it.

Would someone like to volunteer for the super-exciting job of
officially being the bup maintainer? I think a couple of people are
maintaining patch queues already, so it's probably not much of a
stretch. And you can't do much worse than me :)

Have fun,

Avery

Gabriel Filion

unread,
May 2, 2012, 1:30:45 AM5/2/12
to Avery Pennarun, Thomas Klausner, bup list
On 12-04-30 04:34 AM, Avery Pennarun wrote:
> On Mon, Apr 30, 2012 at 4:12 AM, Thomas Klausner <t...@giga.or.at> wrote:
>> Is there a semi-official git where these are tracked? I gather Rob has
>> one like that but I don't see it immediately on
>> https://github.com/apenwarr/bup/network

the reason you don't see Rob's repository is because it's not on github :P

You can follow my repository on github (lelutin/bup); I've been pushing
the branches that Rob maintains and I'll continue to do so.

>> What's the plan for a new release?
>
> There's no plan, but maybe it's time to make one.

Well, actually the original plan was to bring the meta-data branch up to
par and then publish it as the main feature for 0.25.

I've been using tmp/pending/meta for some time now and haven't
encountered problems so far. I guess a couple more people are also using
it, and we haven't yet heard of any complaints.

Rob is currently trying to get hardlink support to work, and I guess as
soon as this is reached, and tested, we can go ahead and make that
over-due 0.25 release and then move on ;)

> Given my
> overworked-ness, it might be smart to just elect someone as the new
> "official" maintainer and patch collector.

Rob, are you up to this?

--
Gabriel Filion

Gabriele Santilli

unread,
May 2, 2012, 3:04:10 AM5/2/12
to Gabriel Filion, Avery Pennarun, Thomas Klausner, bup list
On Wed, May 2, 2012 at 7:30 AM, Gabriel Filion <lel...@gmail.com> wrote:

> I've been using tmp/pending/meta for some time now and haven't
> encountered problems so far. I guess a couple more people are also using
> it, and we haven't yet heard of any complaints.

Yep, no significant problems so far either.

Thomas Klausner

unread,
May 2, 2012, 5:09:39 AM5/2/12
to Gabriel Filion, Avery Pennarun, bup list
On Wed, May 02, 2012 at 01:30:45AM -0400, Gabriel Filion wrote:
> I've been using tmp/pending/meta for some time now and haven't
> encountered problems so far. I guess a couple more people are also using
> it, and we haven't yet heard of any complaints.

Ok, so here's a patch for that branch to make configure work if 'make'
is BSD make but the Makefile is run using GNU make.
Thomas
gnumake

Rob Browning

unread,
May 5, 2012, 4:47:30 PM5/5/12
to Avery Pennarun, Thomas Klausner, bup list
Avery Pennarun <apen...@gmail.com> writes:

> Would someone like to volunteer for the super-exciting job of
> officially being the bup maintainer? I think a couple of people are
> maintaining patch queues already, so it's probably not much of a
> stretch. And you can't do much worse than me :)

I'm willing to give it a try.

I don't know how we'll want to change things, but here's where my work
stands at the moment. I've been maintaining two branches at

http://git.debian.org/?p=users/rlb/bup.git

which I think Gabriel may have been publishing as well.

tmp/pu/master is a set of hopefully uncontroversial patches for master,
and nearly all of them have a "Reviewed-By:" in addition to the author.

tmp/pending/meta contains the metadata patches. I've tried to follow my
inner Avery while writing this code, but I'm sure there are things that
you'd want changed.

Note that I've just rebased both branches so that tmp/pending/meta is an
extension of tmp/pu/master (which is itself an extension of master).

I still consider tmp/pending/meta experimental, and don't want to
include it in an official release without additional review. There are
also a few things that I still need to fix. I imagine the next step
would be to post the patches to the list.

Conversely, I suspect that most if not all of tmp/pu/master is
appropriate for a stable release.

Thanks
--
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Rob Browning

unread,
May 5, 2012, 4:50:45 PM5/5/12
to Gabriel Filion, Avery Pennarun, Thomas Klausner, bup list
Gabriel Filion <lel...@gmail.com> writes:

> Rob is currently trying to get hardlink support to work

I just pushed a preliminary patch to tmp/pending/meta to add hardlink
support. The code tracks multiple targets at restore time (so it can
restore partial sets if the mount points have changed), but it would be
easy to switch back to tracking single targets if we decide that's
preferable.

The patch still needs work, but I wanted to go ahead and let people who
are interested have a look.

Gabriel Filion

unread,
May 6, 2012, 1:28:33 AM5/6/12
to Thomas Klausner, Avery Pennarun, bup list, Jimmy Tang
[added Jimmy on the CC list to ask for your opinion on the change in
config/configure]
could you describe the issue a little bit more:

on which system(s) are you having trouble with 'make'?
if I understand you correctly, you mean that when using GNU make, the
default make binary (which is not GNU make on some systems) is used for
some files, am I misinterpreting your patch?


I tried your patch on Ubuntu and I get this:

$ make
rm -f lib/bup/_version.py lib/bup/_version.py.new
./format-subst.pl lib/bup/_version.py.pre >lib/bup/_version.py.new
mv lib/bup/_version.py.new lib/bup/_version.py
cd config && make config.h
make[1]: Entering directory `/home/gabster/dev/bup/config'
./configure MAKE=make
Bad option MAKE=make. Use --help to show options
make[1]: *** [config.h] Error 1
make[1]: Leaving directory `/home/gabster/dev/bup/config'
make: *** [config/config.h] Error 2

you probably want to push that variable to the environment instead of as
one of its arguments.. I tried just flipping the order to the following
and it worked:

MAKE=${MAKE} ./configure


as for:

> diff --git a/config/configure b/config/configure
> index 902bca6..2fac775 100755
> --- a/config/configure
> +++ b/config/configure
> @@ -12,11 +12,6 @@ if ! AC_PROG_CC; then
> fi
>
> TLOGN "checking the GNU make"
> -MAKE=`acLookFor make`
> -if [ -z "$MAKE" ]; then
> - AC_FAIL " Cannot find make";
> -fi
> -
> MAKE_GNU=`$MAKE --version | grep "GNU Make"`
> if [ -z "$MAKE_GNU" ]; then
> AC_FAIL " $MAKE is not GNU Make"

those lines were added by Jimmy Tang in 543b24ad. Jimmy: do you think
that removing those lines would be reasonable if we suppose the variable
$MAKE is set via the environment in the Makefile ?


p.s.: Thomas, I'd suggest you try using "git send-email" or "git
format-patch" to send your patch to the list (inlined, not as an
attachment). It makes it easier to review and apply your changes.

--
Gabriel Filion

Thomas Klausner

unread,
May 6, 2012, 3:49:32 AM5/6/12
to Gabriel Filion, Avery Pennarun, bup list, Jimmy Tang
On Sun, May 06, 2012 at 01:28:33AM -0400, Gabriel Filion wrote:
> on which system(s) are you having trouble with 'make'?

On NetBSD, but it probably affects the other BSDs as well.

> if I understand you correctly, you mean that when using GNU make, the
> default make binary (which is not GNU make on some systems) is used for
> some files, am I misinterpreting your patch?

Yes. And since the Makefiles currently are not parsable by BSD make, this fails.

> I tried your patch on Ubuntu and I get this:
>
> $ make
> rm -f lib/bup/_version.py lib/bup/_version.py.new
> ./format-subst.pl lib/bup/_version.py.pre >lib/bup/_version.py.new
> mv lib/bup/_version.py.new lib/bup/_version.py
> cd config && make config.h
> make[1]: Entering directory `/home/gabster/dev/bup/config'
> ./configure MAKE=make
> Bad option MAKE=make. Use --help to show options
> make[1]: *** [config.h] Error 1
> make[1]: Leaving directory `/home/gabster/dev/bup/config'
> make: *** [config/config.h] Error 2
>
> you probably want to push that variable to the environment instead of as
> one of its arguments.. I tried just flipping the order to the following
> and it worked:
>
> MAKE=${MAKE} ./configure

Weird, this usually works. But switching is of course fine, so please do that.

> p.s.: Thomas, I'd suggest you try using "git send-email" or "git
> format-patch" to send your patch to the list (inlined, not as an
> attachment). It makes it easier to review and apply your changes.

I'll take a look at that, thanks.
Thomas

Rob Browning

unread,
May 7, 2012, 7:41:15 PM5/7/12
to Gabriel Filion, Avery Pennarun, Thomas Klausner, bup list
Rob Browning <r...@defaultvalue.org> writes:

> Gabriel Filion <lel...@gmail.com> writes:
>
>> Rob is currently trying to get hardlink support to work
>
> I just pushed a preliminary patch to tmp/pending/meta to add hardlink
> support.

It was just pointed out on IRC that the hardlink code doesn't handle
index updates reasonably if the previous index was created by a version
of bup without the hardlink support (i.e. it may choke); I'll see about
fixing that.

Rob Browning

unread,
May 12, 2012, 3:58:30 PM5/12/12
to Gabriel Filion, Avery Pennarun, Thomas Klausner, bup list
Rob Browning <r...@defaultvalue.org> writes:

> I just pushed a preliminary patch to tmp/pending/meta to add hardlink
> support. The code tracks multiple targets at restore time (so it can
> restore partial sets if the mount points have changed), but it would be
> easy to switch back to tracking single targets if we decide that's
> preferable.
>
> The patch still needs work, but I wanted to go ahead and let people who
> are interested have a look.

As at least one more substantial test, I just ran a full backup of a
large root filesystem and verified the result via rsync.

If anyone else would like to test (which would be much appreciated),
you'll need a quiet source (either via lvm snapshot, or a filesystem
that's not in use), and then after the backup, you can run something
(roughly) like this:

bup restore -C tmp-restore /foo/latest/
rsync -niaHAX --exclude WHATEVER ... /src/snapshot/ tmp-restore/

You'll need to use the rsync --exclude options to omit anything that you
excluded from the index, and if all goes well, rsync -i
(--itemize-changes) won't report any unexpected differences.

If anyone's willing, it would be really nice (when feasible) if you
could add a cross-check like this to your normal backup routine and let
us know if rsync reports any unexpected differences.

Thanks

Oei, YC

unread,
May 19, 2012, 6:02:43 PM5/19/12
to Rob Browning, Gabriel Filion, Avery Pennarun, Thomas Klausner, bup list
Oops - I just sent the following to Rob only, here's the same thing
for the list. Apologies for my incompetence with webmail (and its
horrible line wrapping) - it will have to do this time, I'm afraid.

Also, as a PS (where P=pre): I'm currently running another restore
from the same backup below, to see if the same files are affected.


On 12 May 2012 20:58, Rob Browning <r...@defaultvalue.org> wrote:
> If anyone's willing, it would be really nice (when feasible) if you
> could add a cross-check like this to your normal backup routine and let
> us know if rsync reports any unexpected differences.

Hi Rob,

Thanks so much for all the hard work on bup-meta. I've switched to
using tmp/pending/meta (to be precise: bup-version says
"0.25-rc1-35-gce29408"), and I'm now working on putting that
cross-check in place.

For my first "data point", I'm afraid I didn't get to check against
rsync due to time constraints, but instead I checked against dar (see
http://dar.linux.free.fr/ -- for some background: before now, I had
mostly used bup as a "storage backend" for dar...). Among backup
tools, I believe dar stands out as being especially careful preserving
metadata, so it should be a useful check.

First of all the good news: all the hard links were properly restored.
However, some unexpected differences turned up on 16 files (out of
more than 200000) which for some reason did not have their mtimes
restored.

Backup commands were these, where the source was an ext3, non-system
partition on a Ubuntu system and the destination was OS X (and so
restore testing was onto hfs+):

$ bup index -vx .
$ bup save -r mac1: -n d630_lab .
$ dar --create - --noconf --alter=atime --no-mount-points | bup split
-r mac1: -n d630_lab.1.dar

where dar's summary for this backup set read:
--------------------------------------------
229331 inode(s) saved
with 24 hard link(s) recorded
0 inode(s) changed at the moment of the backup
0 inode(s) not saved (no inode/file change)
0 inode(s) failed to save (filesystem error)
0 inode(s) ignored (excluded by filters)
0 inode(s) recorded as deleted from reference backup
--------------------------------------------
Total number of inodes considered: 229331
--------------------------------------------
EA saved for 0 inode(s)
--------------------------------------------


Ok, so on to restoring (this is on the remote machine) and testing:

$ mkdir /tmp/bup_tmp_restore
$ cd /tmp/bup_tmp_restore
$ sudo sh -c "BUP_DIR=/Users/yungchin/.bup bup restore --numeric-ids
/d630_lab/latest/home/yungchin/lab"
$ bup join d630_lab.1.dar > /tmp/dar_tmp_restore.1.dar
$ sudo dar --diff /tmp/dar_tmp_restore > /tmp/dar_diff.out

This results in a bunch of lines like these:

DIFF /private/tmp/bup_tmp_restore/lab/tracserver/trac/db/trac.db:
difference of last modification date: Wed Dec 14 18:52:23 2011 <-->
Sat May 19 21:07:33 2012
DIFF /private/tmp/bup_tmp_restore/lab/tracserver/repos_091109.tar.gz:
difference of last modification date: Mon Nov 9 20:30:00 2009 <-->
Sat May 19 21:07:33 2012
DIFF /private/tmp/bup_tmp_restore/lab/tracserver/repos/.bzr/repository/packs/2bdb93e9209a86e01677b6bb1dc897be.pack:
difference of last modification date: Mon Jul 12 12:06:36 2010 <-->
Sat May 19 21:07:33 2012

...etc etc. The latter happens to be a file in a long unused
repository. And indeed:

$ sudo ls -lhat tracserver/repos/.bzr/repository/packs
total 4504
-rw-r--r-- 1 1000 1000 1.4M May 19 21:07
2bdb93e9209a86e01677b6bb1dc897be.pack
drwxr-xr-x 11 1000 1000 374B Aug 17 2011 ..
drwxr-xr-x 15 1000 1000 510B Aug 17 2011 .
-rw-r--r-- 1 1000 1000 26K Aug 17 2011
f415a51660d01c7b3e80bd05a6c07d13.pack
-rw-r--r-- 1 1000 1000 19K Jul 11 2011
ce262d4c43b084a126627fffc371dcc2.pack
-rw-r--r-- 1 1000 1000 52K Jun 20 2011
473d735fe62cdcfa77ad051549376fb5.pack
-rw-r--r-- 1 1000 1000 54K Oct 8 2010
02b66f31ce7c5c1b0d477996b34fac07.pack
-rw-r--r-- 1 1000 1000 34K Sep 23 2010
f33f0e0f89fd90ecfa537828d4c155d6.pack
-rw-r--r-- 1 1000 1000 50K Sep 18 2010
5624f8a2b328904d0c989b2199bba112.pack
-rw-r--r-- 1 1000 1000 45K Aug 26 2010
ad02fc459823092a0eef04f5d5f09a9e.pack
-rw-r--r-- 1 1000 1000 40K Aug 24 2010
cb6c50970edd57b13b8e32e22ee26e92.pack
-rw-r--r-- 1 1000 1000 45K Jul 28 2010
b3f9af034c856c20030a2ed27ed4cd46.pack
-rw-r--r-- 1 1000 1000 19K Jul 13 2010
349b768635f0589d81f77df771f56b1f.pack
-rw-r--r-- 1 1000 1000 80K Nov 19 2009
67a664df19e01c3aa72497ce92d42070.pack
-rw-r--r-- 1 1000 1000 362K Nov 9 2009
6d227213472f5c4b852a0d3acbacee86.pack

... I've no idea why that one file in the directory was different.
Similarly for all the other files: they had mtimes between 21:04 and
21:09 today, that is, restore time. Any thoughts?

I'll do some more tests, also involving system partitions (thus more
special files) very soon.

Greetings, YC

Oei, YC

unread,
May 19, 2012, 7:25:17 PM5/19/12
to Rob Browning, Gabriel Filion, Avery Pennarun, Thomas Klausner, bup list
On 19 May 2012 23:02, Oei, YC <oei.yu...@gmail.com> wrote:
> Also, as a PS (where P=pre): I'm currently running another restore
> from the same backup below, to see if the same files are affected.
[...]
> However, some unexpected differences turned up on 16 files (out of
> more than 200000) which for some reason did not have their mtimes
> restored.

Ok, as it turns out, after a second bup-restore of the same backup
set, "dar --diff" throws up more than 300 files which got restore
times for their mtimes. Several, but not all of the files that came up
in the first round, came up again in this second round.

This may well be an OS X issue, I suppose? But another issue is that
bup-restore/bup-meta doesn't print warnings for failed mtime restores?

Thanks. YC

Rob Browning

unread,
May 21, 2012, 9:02:50 PM5/21/12
to Oei, YC, Gabriel Filion, Avery Pennarun, Thomas Klausner, bup list
"Oei, YC" <oei.yu...@gmail.com> writes:

> Ok, as it turns out, after a second bup-restore of the same backup
> set, "dar --diff" throws up more than 300 files which got restore
> times for their mtimes. Several, but not all of the files that came up
> in the first round, came up again in this second round.

If you have time, could you try directly restoring just some (or one) of
those files via "restore PATH" to see if the problem is repeatable
outside a full restore?

Though were you saying that the set of broken files varies from restore
to restore?

In case you want to look at the relevant code, it's probably
Metadata._restore_common_rec() in metadata.py, which calls
xstat.*utime(), and ends up in _helpers.c:bup_*utime_ns().

Oei, YC

unread,
May 23, 2012, 6:29:32 AM5/23/12
to Rob Browning, Gabriel Filion, Avery Pennarun, Thomas Klausner, bup list
On 22 May 2012 02:02, Rob Browning <r...@defaultvalue.org> wrote:
> If you have time, could you try directly restoring just some (or one) of
> those files via "restore PATH" to see if the problem is repeatable
> outside a full restore?

Well, that turns out a puzzle. I picked a subdirectory that had a
couple of files that failed the dar diff, and restored it 10 times:
all the mtimes were correct, every time.

Doing another full restore, just to see if I hadn't been dreaming, I
got 700+ files with mtimes not restored.

> Though were you saying that the set of broken files varies from restore
> to restore?

Yes. They're all normal files, too, no symlinks or other special things.

I also tested the full restore on the Linux host I backed up from now,
and that's absolutely flawless.


> In case you want to look at the relevant code, it's probably
> Metadata._restore_common_rec() in metadata.py, which calls
> xstat.*utime(), and ends up in _helpers.c:bup_*utime_ns().

Thanks, well I can't see any obvious problems there (for what it's
worth... given that I'm only very newly familiar with Python C API
stuff).

I guess a problem for further debugging is that I can only seem to
reproduce the problem when restoring lots of files (plus that I only
have access to OS X on a shared machine, so I can't properly abuse the
file system without annoying colleagues...).

YC

Rob Browning

unread,
May 23, 2012, 9:37:33 PM5/23/12
to Oei, YC, Gabriel Filion, Avery Pennarun, Thomas Klausner, bup list

"Oei, YC" <oei.yu...@gmail.com> writes:

> On 22 May 2012 02:02, Rob Browning <r...@defaultvalue.org> wrote:

> Well, that turns out a puzzle. I picked a subdirectory that had a
> couple of files that failed the dar diff, and restored it 10 times:
> all the mtimes were correct, every time.
>
> Doing another full restore, just to see if I hadn't been dreaming, I
> got 700+ files with mtimes not restored.

If you have time, it might be interesting to add some code just after
the (l)utime calls to (x)stat the file on disk and verify that the
values that should have been recorded were recorded. Working from
there, you might be able to narrow down the cause. It might also be
worth checking just before returning from _apply_common_rec() (and
wherever else looks interesting).

Thanks for the help

Aaron M. Ucko

unread,
May 24, 2012, 10:41:42 PM5/24/12
to Rob Browning, bup list
Rob Browning <r...@defaultvalue.org> writes:

> It was just pointed out on IRC that the hardlink code doesn't handle
> index updates reasonably if the previous index was created by a version
> of bup without the hardlink support (i.e. it may choke); I'll see about
> fixing that.

Thanks! Meanwhile, I am pleased to confirm that your preliminary patch
works well in a simple real-world test with no old index to worry about.

--
Aaron M. Ucko, KB1CJC (amu at alum.mit.edu, ucko at debian.org)
http://www.mit.edu/~amu/ | http://stuff.mit.edu/cgi/finger/?a...@monk.mit.edu

Oei, Yung-Chin

unread,
May 25, 2012, 6:39:00 AM5/25/12
to Rob Browning, Gabriel Filion, Avery Pennarun, Thomas Klausner, bup list
On Wed, May 23, 2012 at 08:37:33PM -0500, Rob Browning wrote:
> If you have time, it might be interesting to add some code just after
> the (l)utime calls to (x)stat the file on disk and verify that the
> values that should have been recorded were recorded. Working from
> there, you might be able to narrow down the cause. It might also be
> worth checking just before returning from _apply_common_rec() (and
> wherever else looks interesting).

I plugged in a few asserts - one in Metadata._apply_common_rec(), one at the
end of Metadata.apply_to_path(), and two in xstat.utime() (see the diff below)
- but none of these were triggered on a subsequent restore (I also
double-checked that I am indeed running the altered code...). Nevertheless, I
end up with 590 new mtimes.

Statting some of the files that were reported by hand, I find that their atime
has been restored from the backup correctly. Could it be that the mtimes get
clobbered at some other, later point in the restore process?

> Thanks for the help

No, thank you for guiding me through a bit, I'm learning a lot.

For completeness, here's a diff with the extra asserts. I'll poke around some
more, but it will probably be after the weekend. Thanks!



diff --git a/lib/bup/metadata.py b/lib/bup/metadata.py
index 15fb30f..674e854 100644
--- a/lib/bup/metadata.py
+++ b/lib/bup/metadata.py
@@ -346,6 +346,7 @@ class Metadata:
else:
try:
utime(path, (self.atime, self.mtime))
+ assert(self.mtime == xstat.lstat(path).st_mtime)
except OSError, e:
if e.errno == errno.EACCES:
raise ApplyError('utime: %s' % e)
@@ -709,6 +710,7 @@ class Metadata:
self._apply_linux_xattr_rec(path, restore_numeric_ids=num_ids)
except ApplyError, e:
add_error(e)
+ assert(self.mtime == xstat.lstat(path).st_mtime)

def same_file(self, other):
"""Compare this to other for equivalency. Return true if
diff --git a/lib/bup/xstat.py b/lib/bup/xstat.py
index be6f062..a32ecd6 100644
--- a/lib/bup/xstat.py
+++ b/lib/bup/xstat.py
@@ -53,7 +53,9 @@ if _have_bup_utime_ns:
"""Times must be provided as (atime_ns, mtime_ns)."""
atime = nsecs_to_timespec(times[0])
mtime = nsecs_to_timespec(times[1])
- _helpers.bup_utime_ns(path, (atime, mtime))
+ retval = _helpers.bup_utime_ns(path, (atime, mtime))
+ assert(not retval)
+ assert(times[1] == lstat(path).st_mtime)
else:
def utime(path, times):
"""Times must be provided as (atime_ns, mtime_ns)."""

Rob Browning

unread,
May 26, 2012, 7:34:12 PM5/26/12
to Oei, Yung-Chin, Gabriel Filion, Avery Pennarun, Thomas Klausner, bup list
"Oei, Yung-Chin" <oei.yu...@gmail.com> writes:

> Statting some of the files that were reported by hand, I find that their atime
> has been restored from the backup correctly. Could it be that the mtimes get
> clobbered at some other, later point in the restore process?

Offhand, not that I know of. If I recall correctly, do_node() in
restore-cmd.py should be the only place where the metadata is adjusted.

The fact that it's a different set of files from run to run makes me
suspicious. I can't think of any reason why the restore code would be
non-deterministic (of course that doesn't mean there isn't one).

Is there any chance something else on your system might also be touching
the files?

Thanks

SaintGermain

unread,
May 30, 2012, 4:16:40 AM5/30/12
to bup-list
On May 5, 10:47 pm, Rob Browning <r...@defaultvalue.org> wrote:
> Avery Pennarun <apenw...@gmail.com> writes:
> > Would someone like to volunteer for the super-exciting job of
> > officially being the bup maintainer?  I think a couple of people are
> > maintaining patch queues already, so it's probably not much of a
> > stretch.  And you can't do much worse than me :)
>
> I'm willing to give it a try.
>
> I don't know how we'll want to change things, but here's where my work
> stands at the moment.  I've been maintaining two branches at
>
>  http://git.debian.org/?p=users/rlb/bup.git
>
> which I think Gabriel may have been publishing as well.
>
> tmp/pu/master is a set of hopefully uncontroversial patches for master,
> and nearly all of them have a "Reviewed-By:" in addition to the author.
>
> tmp/pending/meta contains the metadata patches.  I've tried to follow my
> inner Avery while writing this code, but I'm sure there are things that
> you'd want changed.
>
> Note that I've just rebased both branches so that tmp/pending/meta is an
> extension of tmp/pu/master (which is itself an extension of master).
>
> I still consider tmp/pending/meta experimental, and don't want to
> include it in an official release without additional review.  There are
> also a few things that I still need to fix.  I imagine the next step
> would be to post the patches to the list.
>

Hello,

I've just tried your tmp/pending/meta branch (bup --version gives 0.25-
rc1-35-gce29408) on a Debian Testing with a new index/backup
(approximately 17 GB) and it seems that bup is choking on one big file
(2 GB):

test@debian:~$ bup save -n local-test /home/test
Reading index: 84815, done.
bloom: creating from 1 file (143242 objects).
bloom: adding 1 file (145084 objects).
bloom: adding 1 file (134095 objects).
bloom: creating from 4 files (622421 objects).
bloom: adding 1 file (157798 objects).
bloom: adding 1 file (146155 objects).
bloom: adding 1 file (132418 objects).
bloom: adding 1 file (131744 objects).
bloom: adding 1 file (132839 objects).
bloom: adding 1 file (133668 objects).
bloom: adding 1 file (146320 objects).
/home/test/Data/Local.pst: [Errno 5] Input/output error
Traceback (most recent call last):
File "/home/test/local/usr/lib/bup/cmd/bup-save", line 272, in
<module>
newtree = _pop(force_tree = oldtree)
File "/home/test/local/usr/lib/bup/cmd/bup-save", line 120, in _pop
shalist.append((0100644, '.bupm', w.new_blob(metadata)))
File "/home/test/local/usr/lib/bup/bup/git.py", line 584, in new_blob
return self.maybe_write('blob', blob)
File "/home/test/local/usr/lib/bup/bup/git.py", line 577, in
maybe_write
self._write(sha, type, content)
File "/home/test/local/usr/lib/bup/bup/git.py", line 552, in _write
self.breakpoint()
File "/home/test/local/usr/lib/bup/bup/git.py", line 557, in
breakpoint
id = self._end()
File "/home/test/local/usr/lib/bup/bup/git.py", line 641, in _end
obj_list_sha = self._write_pack_idx_v2(self.filename + '.idx', idx,
packbin)
File "/home/test/local/usr/lib/bup/bup/git.py", line 667, in
_write_pack_idx_v2
assert(count == self.count)
AssertionError

However if I try to just backup this file, it works.
Let me know if you need additional information or testing.

Best regards,

Oei, Yung-Chin

unread,
May 30, 2012, 8:45:52 AM5/30/12
to Rob Browning, Gabriel Filion, Avery Pennarun, Thomas Klausner, bup list
On Sat, May 26, 2012 at 06:34:12PM -0500, Rob Browning wrote:
> The fact that it's a different set of files from run to run makes me
> suspicious. I can't think of any reason why the restore code would be
> non-deterministic (of course that doesn't mean there isn't one).
>
> Is there any chance something else on your system might also be touching
> the files?

I can't be entirely sure, because it's quite a heavily used system, but I
did restore within a directory listable only by me, and I think it's unlikely.

I even added an "and False" just now, to make sure that my asserts didn't get
optimised out, but that's not the case. I find it hard to come to any other
conclusion than that the problem is somewhere outside of bup.

If anybody else could try to reproduce this - that is, restoring a largish tree
on OS X, then checking its mtimes - that might be informative.

Thanks!

Oei, Yung-Chin

unread,
May 30, 2012, 7:41:04 PM5/30/12
to Rob Browning, Gabriel Filion, Avery Pennarun, Thomas Klausner, bup list
On Sat, May 12, 2012 at 02:58:30PM -0500, Rob Browning wrote:
> If anyone else would like to test (which would be much appreciated),
> you'll need a quiet source (either via lvm snapshot, or a filesystem
> that's not in use), and then after the backup, you can run something
> (roughly) like this:

Hi Rob,

Me again. I've laid the OS X restore issue aside for a bit, and was now testing
one of my private backups. This is still a bit exotic perhaps: the source is a
btrfs-snapshot of the root subvolume of a Debian system, and the restore had to
be into an ext3 partition on a different machine (space/time constraints...).

You can see where this is going... it breaks while trying to restore xattrs
(trace pasted below this email). I plugged in some print statements to figure
out what path and k were passed to xattr.set(), and it turns out:

path=.snapshots/1337985850/etc/.git/COMMIT_EDITMSG
(this is just a normal git-repo, generated by etckeeper)
k=system.posix_acl_access

I'm not really sure what the expected behaviour is. I made sure I was running
the restore as root. Any advice?

Thanks! YC


Traceback (most recent call last):
File "/usr/local/lib/bup/cmd/bup-restore", line 187, in <module>
do_node(n, sub)
File "/usr/local/lib/bup/cmd/bup-restore", line 149, in do_node
do_node(top, sub, m)
File "/usr/local/lib/bup/cmd/bup-restore", line 149, in do_node
do_node(top, sub, m)
File "/usr/local/lib/bup/cmd/bup-restore", line 149, in do_node
do_node(top, sub, m)
File "/usr/local/lib/bup/cmd/bup-restore", line 149, in do_node
do_node(top, sub, m)
File "/usr/local/lib/bup/cmd/bup-restore", line 152, in do_node
restore_numeric_ids=opt.numeric_ids)
File "/usr/local/lib/bup/bup/metadata.py", line 709, in apply_to_path
self._apply_linux_xattr_rec(path, restore_numeric_ids=num_ids)
File "/usr/local/lib/bup/bup/metadata.py", line 602, in _apply_linux_xattr_rec
xattr.set(path, k, v, nofollow=True)
IOError: [Errno 95] Operation not supported


Oei, Yung-Chin

unread,
Jun 1, 2012, 8:35:14 PM6/1/12
to Rob Browning, Gabriel Filion, Avery Pennarun, Thomas Klausner, bup list
On Thu, May 31, 2012 at 12:41:04AM +0100, Oei, Yung-Chin wrote:
> one of my private backups. This is still a bit exotic perhaps: the source is a
> btrfs-snapshot of the root subvolume of a Debian system, and the restore had to
> be into an ext3 partition on a different machine (space/time constraints...).

Ok, it turns out that that ext3 partition I have on there is so old, it was
created without ext_attr flag. So obviously it wouldn't play nice when we try
to set xattrs on it. (Isn't this something the pyxattr lib should deal with?)

On a happier note for the weekend, I found some space for a temporary btrfs
volume on the restore machine, and can report that the restore completes just
fine on that, and that the restored tree is deemed a perfect copy by both "dar
--diff" and "rsync -nciaHAX --delete --numeric-ids".

(According to dar, this had 521 hard links, and extended attributes on 196
inodes, so nothing spectacular perhaps, but it's the best test I can offer for
now).

Thanks! YC

Rob Browning

unread,
Jun 2, 2012, 4:25:26 PM6/2/12
to Oei, Yung-Chin, Gabriel Filion, Avery Pennarun, Thomas Klausner, bup list
"Oei, Yung-Chin" <oei.yu...@gmail.com> writes:

> Me again. I've laid the OS X restore issue aside for a bit, and was
> now testing one of my private backups. This is still a bit exotic
> perhaps: the source is a btrfs-snapshot of the root subvolume of a
> Debian system, and the restore had to be into an ext3 partition on a
> different machine (space/time constraints...).

That's good -- testing more unusual cases.

> I'm not really sure what the expected behaviour is. I made sure I was
> running the restore as root. Any advice?

See next message.

Rob Browning

unread,
Jun 2, 2012, 4:27:37 PM6/2/12
to Oei, Yung-Chin, Gabriel Filion, Avery Pennarun, Thomas Klausner, bup list
"Oei, Yung-Chin" <oei.yu...@gmail.com> writes:

> Ok, it turns out that that ext3 partition I have on there is so old,
> it was created without ext_attr flag. So obviously it wouldn't play
> nice when we try to set xattrs on it. (Isn't this something the
> pyxattr lib should deal with?)

Actually, I suspect this is something we'd intended to handle, but I'll
have to examine the situation more carefully. i.e. bup shouldn't just
die.

> On a happier note for the weekend, I found some space for a temporary
> btrfs volume on the restore machine, and can report that the restore
> completes just fine on that, and that the restored tree is deemed a
> perfect copy by both "dar --diff" and "rsync -nciaHAX --delete
> --numeric-ids".

That's good to hear, and thanks again for the help.

Rob Browning

unread,
Jun 2, 2012, 5:45:15 PM6/2/12
to Oei, Yung-Chin, Gabriel Filion, Avery Pennarun, Thomas Klausner, bup list
"Oei, Yung-Chin" <oei.yu...@gmail.com> writes:

> Ok, it turns out that that ext3 partition I have on there is so old,
> it was created without ext_attr flag. So obviously it wouldn't play
> nice when we try to set xattrs on it. (Isn't this something the
> pyxattr lib should deal with?)

I think that bup probably just needs to check for EOPNOTSUPP in
_apply_linux_xattr_rec(). If we want the failures to result in a
delayed error (via add_error()), then the code could look like this:

try:
xattr.set(path, k, v, nofollow=True)
except IOError, e:
if e.errno in (errno.EPERM, errno.EOPNOTSUPP):
raise ApplyError('xattr.set: %s' % e)
else:
raise

We should probably also check the underlying system calls for
xattr.remove(), ACLs, and Linux attributes for similar issues -- I'll
handle that if no one beats me to it.

> On a happier note for the weekend, I found some space for a temporary btrfs
> volume on the restore machine, and can report that the restore completes just
> fine on that, and that the restored tree is deemed a perfect copy by both "dar
> --diff" and "rsync -nciaHAX --delete --numeric-ids".

Good, and thanks.

Rob Browning

unread,
Jun 3, 2012, 6:27:20 PM6/3/12
to Oei, Yung-Chin, Gabriel Filion, Avery Pennarun, Thomas Klausner, bup list
Rob Browning <r...@defaultvalue.org> writes:

> We should probably also check the underlying system calls for
> xattr.remove(), ACLs, and Linux attributes for similar issues -- I'll
> handle that if no one beats me to it.

OK, I've pushed a rebased tmp/pending/meta branch which attempts to
address these issues:

Defer errors when the restore target doesn't support relevant metadata.

When the restore target doesn't support POSIX1e ACLs, extended
attributes, or Linux attributes, handle the failures via add_error()
(i.e. defer them).

Test these cases by creating a loopback VFAT filesystem
(testfs-limited) and using it as a restore target.

I've also re-enabled metadata testing during "make check" since this
*is* the metadata dev branch. Avery disabled it last year in master
(unless TEST_META was set -- i.e. unless you ran "make TEST_META=1
check").

However, I've disabled some of the root-only tests whenever the uname
output doesn't match Linux. Those tests create ext3 and vfat loopback
filesystems, etc., and aren't likely to work in other environments yet.

Rob Browning

unread,
Jun 3, 2012, 6:55:38 PM6/3/12
to Oei, Yung-Chin, Gabriel Filion, Avery Pennarun, Thomas Klausner, bup list
Rob Browning <r...@defaultvalue.org> writes:

> OK, I've pushed a rebased tmp/pending/meta branch which attempts to
> address these issues:
>
> Defer errors when the restore target doesn't support relevant metadata.

Oh, and I imagine that we may eventually want to provide some way to
suppress errors like this in cases where you know the target FS doesn't
support all of the available metadata.

Saint Germain

unread,
Jun 7, 2012, 7:35:34 PM6/7/12
to bup-...@googlegroups.com
On Sun, 03 Jun 2012 17:27:20 -0500, Rob Browning <r...@defaultvalue.org>
wrote :

> Rob Browning <r...@defaultvalue.org> writes:
>
> > We should probably also check the underlying system calls for
> > xattr.remove(), ACLs, and Linux attributes for similar issues --
> > I'll handle that if no one beats me to it.
>
> OK, I've pushed a rebased tmp/pending/meta branch which attempts to
> address these issues:
>
> Defer errors when the restore target doesn't support relevant
> metadata.
>

Hello,

I've tested the last revision (commit
291a3a3a4975c429fc16c561509aeabd7152e465 from Rob on 2012-06-03) and
I've noticed a huge drop in performance in incremental backup.

I compared it to the current available version from Debian
(0.25~git2011.11.04-3) on a 24 GB / 10000 files directory.

With both versions, initial backup takes 35 mn.

With Debian version, following backup (after a few modifications) takes
1mn48.
With current Git version, following backup takes 12mn44.

I can try to bisect to find the culprit if someone is interested ?
But if you are in the middle of huge changes, I will wait.

By the way I didn't received any feedback from my last email which
found a bug:
http://groups.google.com/group/bup-list/msg/69cbd41a63edc60a

Regards,

Jimmy Tang

unread,
Jun 13, 2012, 6:03:32 PM6/13/12
to bup-...@googlegroups.com, Thomas Klausner, Avery Pennarun, Jimmy Tang


On Sunday, May 6, 2012 6:28:33 AM UTC+1, Lelutin wrote:


as for:

> diff --git a/config/configure b/config/configure
> index 902bca6..2fac775 100755
> --- a/config/configure
> +++ b/config/configure
> @@ -12,11 +12,6 @@ if ! AC_PROG_CC; then
>  fi
>
>  TLOGN "checking the GNU make"
> -MAKE=`acLookFor make`
> -if [ -z "$MAKE" ]; then
> -    AC_FAIL " Cannot find make";
> -fi
> -
>  MAKE_GNU=`$MAKE --version | grep "GNU Make"`
>  if [ -z "$MAKE_GNU" ]; then
>      AC_FAIL " $MAKE is not GNU Make"

those lines were added by Jimmy Tang in 543b24ad. Jimmy: do you think
that removing those lines would be reasonable if we suppose the variable
$MAKE is set via the environment in the Makefile ?



Apologies for not keeping up with the list, removing those lines would be reasonable all the makefiles in the project dont have any gnu dependent stuff anymore, I had that in there as at the time I think there was some gnuism's in the makefiles and it was sensible to check for gnumake.

Thanks,
Jimmy 

Thomas Klausner

unread,
Jun 14, 2012, 2:20:22 AM6/14/12
to Jimmy Tang, bup-...@googlegroups.com, Avery Pennarun, Jimmy Tang
On Wed, Jun 13, 2012 at 03:03:32PM -0700, Jimmy Tang wrote:
> Apologies for not keeping up with the list, removing those lines would be
> reasonable all the makefiles in the project dont have any gnu dependent
> stuff anymore, I had that in there as at the time I think there was some
> gnuism's in the makefiles and it was sensible to check for gnumake.

There still are gnuisms in the Makefile:
make: "bup/Makefile" line 5: Missing dependency operator
make: "bup/Makefile" line 7: Need an operator
make: "bup/Makefile" line 111: warning: duplicate script for target "cmd/bup-%" ignored
make: "bup/Makefile" line 100: warning: using previous script for "cmd/bup-%" defined here
make: "bup/Makefile" line 112: warning: duplicate script for target "cmd/bup-%" ignored
make: "bup/Makefile" line 100: warning: using previous script for "cmd/bup-%" defined here
make: Fatal errors encountered -- cannot continue

at least in:
02bd2b566ea5eec2fd656e0ae572b4c7b6b9550a branch 'master' of git://github.com/apenwarr/bup
Thomas

Thomas Klausner

unread,
Jun 14, 2012, 3:41:24 AM6/14/12
to Jimmy Tang, bup-...@googlegroups.com, Avery Pennarun, Jimmy Tang
And to add some more details: The patch I sent is not for removing
gmake detection, but to use the 'MAKE' variable that is usually set
correctly by make(1) (both BSD and GNU) anyway.
Thomas

Gabriel Filion

unread,
Jun 15, 2012, 8:12:52 PM6/15/12
to SaintGermain, bup-list
On 12-05-30 04:16 AM, SaintGermain wrote:
> I've just tried your tmp/pending/meta branch (bup --version gives 0.25-
> rc1-35-gce29408) on a Debian Testing with a new index/backup
> (approximately 17 GB) and it seems that bup is choking on one big file
> (2 GB):
>
> test@debian:~$ bup save -n local-test /home/test
> Reading index: 84815, done.
> bloom: creating from 1 file (143242 objects).
> bloom: adding 1 file (145084 objects).
> bloom: adding 1 file (134095 objects).
> bloom: creating from 4 files (622421 objects).
> bloom: adding 1 file (157798 objects).
> bloom: adding 1 file (146155 objects).
> bloom: adding 1 file (132418 objects).
> bloom: adding 1 file (131744 objects).
> bloom: adding 1 file (132839 objects).
> bloom: adding 1 file (133668 objects).
> bloom: adding 1 file (146320 objects).
> /home/test/Data/Local.pst: [Errno 5] Input/output error

That error's quite fishy. Have you checked your system logs to see if
you see disk errors?

> However if I try to just backup this file, it works.
> Let me know if you need additional information or testing.

that does sound weird indeed..

do you have that same issue with an older revision of the bup code?

--
Gabriel Filion

Saint Germain

unread,
Jun 16, 2012, 8:30:29 AM6/16/12
to bup-list, Gabriel Filion


On Fri, 15 Jun 2012 20:12:52 -0400, Gabriel Filion <lel...@gmail.com>
wrote :

> On 12-05-30 04:16 AM, SaintGermain wrote:
> > I've just tried your tmp/pending/meta branch (bup --version gives
> > 0.25- rc1-35-gce29408) on a Debian Testing with a new index/backup
> > (approximately 17 GB) and it seems that bup is choking on one big
> > file (2 GB):
> >
> > test@debian:~$ bup save -n local-test /home/test
> > Reading index: 84815, done.
> > bloom: creating from 1 file (143242 objects).
> > bloom: adding 1 file (145084 objects).
> > bloom: adding 1 file (134095 objects).
> > bloom: creating from 4 files (622421 objects).
> > bloom: adding 1 file (157798 objects).
> > bloom: adding 1 file (146155 objects).
> > bloom: adding 1 file (132418 objects).
> > bloom: adding 1 file (131744 objects).
> > bloom: adding 1 file (132839 objects).
> > bloom: adding 1 file (133668 objects).
> > bloom: adding 1 file (146320 objects).
> > /home/test/Data/Local.pst: [Errno 5] Input/output error
>
> That error's quite fishy. Have you checked your system logs to see if
> you see disk errors?
>

Yes I checked and no disk errors.

> > However if I try to just backup this file, it works.
> > Let me know if you need additional information or testing.
>
> that does sound weird indeed..
>
> do you have that same issue with an older revision of the bup code?
>

No everything works fine with the bup version available on Debian
Unstable.

Regards,

Gabriel Filion

unread,
Jun 16, 2012, 1:31:50 PM6/16/12
to Saint Germain, bup-list
On 12-06-16 08:30 AM, Saint Germain wrote:
>>> However if I try to just backup this file, it works.
>>> > > Let me know if you need additional information or testing.
>> >
>> > that does sound weird indeed..
>> >
>> > do you have that same issue with an older revision of the bup code?
>> >
> No everything works fine with the bup version available on Debian
> Unstable.

ah, so we do have a bug in the code.

could you try and git bisect the code between current tmp/pending/meta
and the debian unstable version to try and find the first commit that
shows the error?

debian unstable shows version 0.25~git2011.11.04-5 .. which I don't
quite understand what it corresponds to.. maybe you could test the lates
master and base your bisection there if it doesn't shoot an error, so
you'll be bisecting over the tmp/pending/meta branch

--
Gabriel Filion

Gabriel Filion

unread,
Jun 17, 2012, 10:27:59 PM6/17/12
to Saint Germain, bup-list
[re-adding the mailing list in the CC. remember to reply-all when
replying to the list ;) ]

On 12-06-17 07:13 PM, Saint Germain wrote:
> On Sat, 16 Jun 2012 13:31:50 -0400, Gabriel Filion <lel...@gmail.com>
> wrote :
> Ok I'll try. But that may take a loooong time given that I have to
> backup my whole home directory (1h each time).

right, it could take a bunch of time :\ but removing bugs in the
application is worth the trouble, isn't it? ^-^

> Is it not possible to improve the error reporting ?

you're right, it would make your time a little bit more worthwhile to
add more output .. but for this we'd need to understand where things are
getting cought up.

we can see an I/O error, and then some values are off for an assertion
in git.py, but I have the feeling they're off as a result of the
previous error (from which we unfortunately don't know much).

Since the backup works when running only on that file, it's possibly not
related to the file's type or contents.. So it must have to do with
previously considered files. Do you have any special files somewhere in
your home directory (and below it)?

Try reducing the subset of the files being backed up and see if the
error is reproducible in a smaller set. (e.g. start by backing up just
the files directly in /home/yourself/ , then add one subdirectory at a
time, and when you hit the error, try backing up just that directory and
the big file)

Maybe someone else could help out a little bit by pointing where adding
debug output would give some clues?

--
Gabriel Filion

yungchin

unread,
Sep 14, 2012, 6:34:53 PM9/14/12
to bup-...@googlegroups.com, Rob Browning, Gabriel Filion, Avery Pennarun, Thomas Klausner
On Wednesday, May 30, 2012 1:45:52 PM UTC+1, Yung-Chin Oei wrote:
On Sat, May 26, 2012 at 06:34:12PM -0500, Rob Browning wrote:
> The fact that it's a different set of files from run to run makes me
> suspicious.  I can't think of any reason why the restore code would be
> non-deterministic (of course that doesn't mean there isn't one).
>
> Is there any chance something else on your system might also be touching
> the files?

I can't be entirely sure, because it's quite a heavily used system, but I
did restore within a directory listable only by me, and I think it's unlikely.

I even added an "and False" just now, to make sure that my asserts didn't get
optimised out, but that's not the case. I find it hard to come to any other
conclusion than that the problem is somewhere outside of bup.

Just to add an update to this: I still have this problem on OS X, with the latest tmp/pending/meta code, and I still don't know how to debug it. 
 
If anybody else could try to reproduce this - that is, restoring a largish tree
on OS X, then checking its mtimes - that might be informative.

Thanks!

PS: apologies for sending this from the web interface. I can't figure out how to download a list archive off Google Groups...

Gabriel Filion

unread,
Sep 17, 2012, 8:26:25 PM9/17/12
to yungchin, bup-...@googlegroups.com, Rob Browning, Avery Pennarun, Thomas Klausner
On 12-09-14 06:34 PM, yungchin wrote:
> On Wednesday, May 30, 2012 1:45:52 PM UTC+1, Yung-Chin Oei wrote:
>
> On Sat, May 26, 2012 at 06:34:12PM -0500, Rob Browning wrote:
> > The fact that it's a different set of files from run to run makes me
> > suspicious. I can't think of any reason why the restore code
> would be
> > non-deterministic (of course that doesn't mean there isn't one).
> >
> > Is there any chance something else on your system might also be
> touching
> > the files?
>
> I can't be entirely sure, because it's quite a heavily used system,
> but I
> did restore within a directory listable only by me, and I think it's
> unlikely.
>
> I even added an "and False" just now, to make sure that my asserts
> didn't get
> optimised out, but that's not the case. I find it hard to come to
> any other
> conclusion than that the problem is somewhere outside of bup.
>
>
> Just to add an update to this: I still have this problem on OS X, with
> the latest tmp/pending/meta code, and I still don't know how to debug it.

dunno if it could help you out, but I have an unfinished branch based on
top of tmp/pending/meta that adds metadata exposition to frontends. for
now I completed the feature only for bup-fuse.

maybe that way you'd be able to confirm that the recorded mtime is
correct, and that the bug would be happening during the restore?

if it can help I can send this patch alone to the list.

--
Gabriel Filion

Oei, Yung-Chin

unread,
Sep 18, 2012, 7:22:40 AM9/18/12
to Gabriel Filion, bup-...@googlegroups.com, Rob Browning, Avery Pennarun, Thomas Klausner
On Mon, Sep 17, 2012 at 08:26:25PM -0400, Gabriel Filion wrote:
> dunno if it could help you out, but I have an unfinished branch based on
> top of tmp/pending/meta that adds metadata exposition to frontends. for
> now I completed the feature only for bup-fuse.
>
> maybe that way you'd be able to confirm that the recorded mtime is
> correct, and that the bug would be happening during the restore?

Thanks. The strange thing is, when I restore multiple times, a different
subset of files get the wrong mtime. I also restored the same on Linux
with zero anomalies.

But nevertheless I'd be very very keen to try your bup-fuse with
metadata patches! :)

YC

Zoran Zaric

unread,
Sep 24, 2012, 10:01:37 AM9/24/12
to Saint Germain, Rob Browning, bup-...@googlegroups.com
On Fri, Jun 08, 2012 at 01:35:34AM +0200, Saint Germain wrote:
> On Sun, 03 Jun 2012 17:27:20 -0500, Rob Browning <r...@defaultvalue.org>
> wrote :
>
> > Rob Browning <r...@defaultvalue.org> writes:
> >
> > > We should probably also check the underlying system calls for
> > > xattr.remove(), ACLs, and Linux attributes for similar issues --
> > > I'll handle that if no one beats me to it.
> >
> > OK, I've pushed a rebased tmp/pending/meta branch which attempts to
> > address these issues:
> >
> > Defer errors when the restore target doesn't support relevant
> > metadata.
> >
>
> Hello,
>
> I've tested the last revision (commit
> 291a3a3a4975c429fc16c561509aeabd7152e465 from Rob on 2012-06-03) and
> I've noticed a huge drop in performance in incremental backup.
>
> I compared it to the current available version from Debian
> (0.25~git2011.11.04-3) on a 24 GB / 10000 files directory.
>
> With both versions, initial backup takes 35 mn.
>
> With Debian version, following backup (after a few modifications) takes
> 1mn48.
> With current Git version, following backup takes 12mn44.
>
> I can try to bisect to find the culprit if someone is interested ?
> But if you are in the middle of huge changes, I will wait.

Is this still the case?

Rob, any idea? Would a bisect help?

Rob Browning

unread,
Sep 24, 2012, 3:26:59 PM9/24/12
to Zoran Zaric, Saint Germain, bup-...@googlegroups.com
Zoran Zaric <z...@zoranzaric.de> writes:

> Is this still the case?
>
> Rob, any idea? Would a bisect help?

If possible, I'd like to see if the current code is still slow. I'm
sure that handling metadata will have a cost, but that seems high.

Thanks
Reply all
Reply to author
Forward
0 new messages