bup-list Google Group

Re: [PATCH 1/1] bup-index: don't warn on lstat failures for excluded paths

r...@defaultvalue.org (Rob Browning) — Wed, 19 Jun 2013 04:48:44 UT

Thanks, I'll try to take a look this weekend, if no one else beats me to
it.

--
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: How to start backing up a virtual machine?

r...@defaultvalue.org (Rob Browning) — Wed, 19 Jun 2013 00:58:33 UT

As long as the VM is stopped, you should be able to do something
like this:

$ bup -d ./backup init
$ bup -d ./backup index /some/where
$ bup -d ./backup save -n name-for-your-backup-set /some/where

As long as the VM images are under /some/where, you should be set, and
to perform another (incremental) backup after that:

How to start backing up a virtual machine?

m...@ojkastl.de (Johannes Kastl) — Tue, 18 Jun 2013 18:36:39 UT

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi everyone,

I am looking for a tutorial or dscription on how to get started with
bup. Especially using it to backup a virtual machine.

Of course I found the readme on [link], but that
one is a bit to fast for me... ;-)

[PATCH 1/1] bup-index: don't warn on lstat failures for excluded paths

thom...@gmail.com (Thomas Haller) — Tue, 18 Jun 2013 15:45:10 UT

Before, when walking the directories, whenever the call to
lstat failed, a warning got raised and bup-index returned with
non-zero exit code. A common case where this happens are fuse
mount points of other users.

This warning could not be suppressed with the exclude* options
because the check for excludes was only performed after the warning

Re: Restore from a Remote Server

mad.vi...@gmail.com (Madireddy Samuel Vijaykumar) — Tue, 18 Jun 2013 06:44:47 UT

Excellent. Thanks James. This was definitely helpful.

Sam

-----

samuel vijaykumar
[link] | samo...@gmail.com | +91-9381034583
a3 valathi flats. 24 dr thomas. road t nagar. chennai 600 017

*all hard work brings a profit, but mere talk leads only to poverty. --
from the Bible*

May tag 0.25-rc2 soon, speak now...

r...@defaultvalue.org (Rob Browning) — Tue, 18 Jun 2013 04:09:30 UT

It's been too long, and too quiet; I may go ahead and tag 0.25-rc2 soon.

I think there's probably still trouble on OSX, and I've been waiting to
see if anyone has time to help there, but I don't want to let that hold
up the release any further.

Instead, I may just add a suitable section to the README, and disable

Re: Restore from a Remote Server

ja...@lottspot.com (James Lott) — Mon, 17 Jun 2013 23:17:41 UT

Hey Samuel,

I had the same question not too long ago and Gabriel shared with me his
solution, which I have yet to test on my own setups, but will most
likely be my preferred method once I get around to doing so. In any
case, I figured it may interest you as well, so I included his reply for

Re: Restore from a Remote Server

mad.vi...@gmail.com (Madireddy Samuel Vijaykumar) — Mon, 17 Jun 2013 04:10:02 UT

@robert. Thanks for your reply.

-----

samuel vijaykumar
[link] | samo...@gmail.com | +91-9381034583
a3 valathi flats. 24 dr thomas. road t nagar. chennai 600 017

*all hard work brings a profit, but mere talk leads only to poverty. --
from the Bible*

-----

Re: Restore from a Remote Server

r...@defaultvalue.org (Rob Browning) — Sat, 15 Jun 2013 01:21:54 UT

I don't believe bup restore supports a remote source yet, so you'll
proably either have to copy the remote bup repository to the local
machine (via say rsync), and then use "bup -d local-bup-dir-copy restore
...", or you'll need to restore the desired files on the remote machine,
and then copy them (via tar, rsync, scp -r) to the local host.

Restore from a Remote Server

mad.vi...@gmail.com (Samuel Vijaykumar) — Thu, 13 Jun 2013 12:50:13 UT

Hi.
i just started using the bup. please forgive me if this is wrong question.
I have done a backup of my /etc folder onto a remote server, using the
standard procedure as give below.
But I am unable to figure out how it restore to my local server again.
Thanks in advance for all the answers.
Sam

Re: Tried to back up a live filesystem, got exception

na...@iki.fi (Nuutti Kotivuori) — Mon, 10 Jun 2013 10:30:57 UT

I have not tried reproducing. I just back up my files daily and the issue
has not happened since, so not terribly common.
In general, bup seems to work nicely, with metadata. My homedir is 1.1M
files and 178G, on SSD. Incremental backup takes 7 minutes.
-- Naked

Re: Tried to back up a live filesystem, got exception

lelu...@gmail.com (Gabriel Filion) — Mon, 10 Jun 2013 06:26:49 UT

hmm getting a race condition there sounds quite unlocky (although not
impossible). so I guess you can't reproduce the issue consistently?

--
Gabriel Filion

Re: [PATCH] Change pack index writing - remove unsafe mmap - fixes ARM

r...@defaultvalue.org (Rob Browning) — Sat, 08 Jun 2013 15:40:59 UT

And a Reported-by for good measure. Pushed to master -- thanks for the
help.

--
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: [PATCH] Change pack index writing - remove unsafe mmap - fixes ARM

alan.christopher.jenk...@gmail.com (Alan Jenkins) — Sat, 08 Jun 2013 10:49:24 UT

That'll do it, thanks!

Just to be sure, I applied your version of the patch. I've been using
it while I set up BUP on my ARM box. It seems to work now, just as
well as on my x86 pc.

So you can have my Reviewed-by if you like :). And the same for Tested-by.

Alan

Tried to back up a live filesystem, got exception

na...@iki.fi (Nuutti Kotivuori) — Fri, 07 Jun 2013 10:37:37 UT

Hello,
I got the following traceback while doing bup index:
Traceback (most recent call last):
File "/usr/lib/bup/cmd/bup-index", line 237, in <module>
update_index(rp, excluded_paths)
File "/usr/lib/bup/cmd/bup-index", line 132, in update_index
meta = metadata.from_path(path, statinfo=pst)

Re: [PATCH 3/7] Remove lib.sh actually-root in favor of t/root-status.

r...@defaultvalue.org (Rob Browning) — Fri, 07 Jun 2013 05:36:17 UT

Umm, ok right, so there is that.

Yes, I had that originally, which is probably why I ended up with the
completely broken code, but decided to change it because we need to be
able to distinguish between "root" and "fake".

Thanks for the help -- should be fixed in master now.
--
Rob Browning

Re: [PATCH 0/7] Fix some portability problems

r...@defaultvalue.org (Rob Browning) — Thu, 06 Jun 2013 00:15:50 UT

Pushed to master. Please let me know if you notice any problems.

Thanks
--
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: [PATCH] Change pack index writing - remove unsafe mmap - fixes ARM

r...@defaultvalue.org (Rob Browning) — Mon, 03 Jun 2013 23:59:35 UT

So maybe something like the following, which we can always rewrite later
if we detect a significant difference in performance?

diff --git a/lib/bup/git.py b/lib/bup/git.py
index 403b969..00975a7 100644
--- a/lib/bup/git.py
+++ b/lib/bup/git.py
@@ -664,6 +664,10 @@ class PackWriter:
idx_f.seek(0, os.SEEK_END)

Re: [PATCH] Change pack index writing - remove unsafe mmap - fixes ARM

alan.christopher.jenk...@gmail.com (Alan Jenkins) — Mon, 03 Jun 2013 06:58:52 UT

Yes.

Makes sense. The longer patch is designed to avoid hurting performance
at all, but correctness can come first :). You might say I'm
over-pedantic in trying to avoid one fsync() per gigabyte packfile...

Thank you for working on the project :).

Alan

Re: [PATCH] Change pack index writing - remove unsafe mmap - fixes ARM

r...@defaultvalue.org (Rob Browning) — Sun, 02 Jun 2013 18:34:18 UT

I may not understand the situation correctly yet, but does that mean
that if we did just add that flush() the problem would be fixed?

If so, even if that's not the solution we want longer term, I'd be
tempted to start there, and come back to this after 0.25, hoping that we
have more resources for review.

Re: New xattr failure?

lelu...@gmail.com (Gabriel Filion) — Fri, 31 May 2013 15:43:32 UT

Ryan Brown wrote a fix for that some time ago, but it was flawed (was
changing the negative test into a positive one -- see my comment in the
discussion)

[link]

maybe you could try to come up with a new version of the patch?

Re: [PATCH] Change pack index writing - remove unsafe mmap - fixes ARM

g...@work.lexort.com (Greg Troxel) — Fri, 31 May 2013 14:46:34 UT

thanks. This does seems tricky. It strikes me as buggy that munmap
does not imply at least consistency with read. (I do not expect munmap
to necessarily cause the bits to hit permanent storage.)

Re: [PATCH] Change pack index writing - remove unsafe mmap - fixes ARM

alan.christopher.jenk...@gmail.com (Alan Jenkins) — Fri, 31 May 2013 14:21:50 UT

TBH I don't know.

"The application must ensure correct synchronization when using /mmap/()
in conjunction with any other file access method, such as /read/()
<[link]> and
/write/()
<[link]>,

Re: [PATCH] Change pack index writing - remove unsafe mmap - fixes ARM

g...@work.lexort.com (Greg Troxel) — Fri, 31 May 2013 14:16:14 UT

The reason is that on my ARM box, writes to mmap() do not immediately
change the result of read(). (Virtually addressed cpu cache?)
If idx_map.flush() (msync) is called before idx_map.close() (munmap),
the problem goes away. Unfortunately that one-line fix is equivalent
to fsync(), before the file is complete, and I'm a pedant.

[PATCH] Change pack index writing - remove unsafe mmap - fixes ARM

alan.christopher.jenk...@gmail.com (Alan Jenkins) — Fri, 31 May 2013 13:52:01 UT

make test was failing on my ARM box. It reported missing blobs.
Let's make it pass.

After `bup save`, `git fsck` would show messages like
"Packfile index for %s SHA1 mismatch"
This indicated a bad trailing checksum on the pack index file.

The reason is that on my ARM box, writes to mmap() do not immediately

Re: New xattr failure?

m...@idnet.com (Mark J Hewitt) — Wed, 29 May 2013 20:00:45 UT

Hmm ... But not when running "make test" as root it seems:

! lib/bup/t/tmetadata.py test_handling_of_incorrect_exi sting_linux_xattrs:
umount: testfs: not found
dd if=/dev/zero of=testfs.img bs=1M count=32
32+0 records in
32+0 records out
33554432 bytes (34 MB) copied, 0.12848 s, 261 MB/s
mke2fs -F -j -m 0 testfs.img

Re: New xattr failure?

m...@idnet.com (Mark J Hewitt) — Wed, 29 May 2013 19:21:24 UT

So I'm replying to myself again ...

This issue is because the system in question did not have selinux
enabled. The attribute that is being set is "security.selinux" to
"system_u:object_r:home_root_t :s0".
I have not yet figured out how this comes about, but turning on selinux
in the lowest security level possible (and tediously relabelling all

Re: Bup-on support for restorations?

lelu...@gmail.com (Gabriel Filion) — Wed, 29 May 2013 14:19:21 UT

Hi there,

iirc -- Rob, correct me if I'm wrong -- bup doesn't support remote
restoration yet (but I'd really like to have that feature ;) )

The way I currently achieve that is via sshfs and a "local restore" on
the server using the mounted repository.

or you can copy the repos over to the other side, but this means you'll

New xattr failure?

m...@idnet.com (Mark J Hewitt) — Wed, 29 May 2013 11:04:57 UT

I'm sure I'm missing something here ..

I've just tried the same bup version I use successfully on three other
machines on another, and see this failure in "make test":

! t/test.sh:481 bup save --strip-path graft-points.tmp/foo -n foo
graft-points.tmp/src/x ok
xattr.set: [Errno 1] Operation not permitted

[PATCH 4/7] t/compare-trees: don't check ACLs on Cygwin (no support).

r...@defaultvalue.org (Rob Browning) — Mon, 27 May 2013 17:53:09 UT

Signed-off-by: Rob Browning <r...@defaultvalue.org>
---
README.md | 7 +++++++
t/compare-trees | 7 ++++++-
2 files changed, 13 insertions(+), 1 deletion(-)

diff --git a/README.md b/README.md
index 19998d2..2351c77 100644
--- a/README.md
+++ b/README.md
@@ -256,6 +256,13 @@ Notes on NetBSD/pkgsrc

[PATCH 1/7] Use sys.platform instead of platform.system().

r...@defaultvalue.org (Rob Browning) — Mon, 27 May 2013 17:53:06 UT

If nothing else, the former is computed at compile time, and so should
be more efficient. Use the recommended "startswith()" idiom.

Signed-off-by: Rob Browning <r...@defaultvalue.org>
---
cmd/ftp-cmd.py | 2 +-
lib/bup/helpers.py | 4 ++--
lib/bup/metadata.py | 6 +++---

[PATCH 0/7] Fix some portability problems

r...@defaultvalue.org (Rob Browning) — Mon, 27 May 2013 17:53:05 UT

This set should fix some portability problems, primarily on Cygwin,
but the tbloom.py changes should help other places too (other 32-bit
platforms, s390).

Note that some of these fixes are just workarounds for now (tbloom.py,
disabling the "same tree" tests, etc.). My goal was just to get
things into a reasonable enough state for 0.25-rc2.

[PATCH 5/7] README.md: add Cygwin dependencies.

r...@defaultvalue.org (Rob Browning) — Mon, 27 May 2013 17:53:10 UT

Signed-off-by: Rob Browning <r...@defaultvalue.org>
---
README.md | 2 ++
1 file changed, 2 insertions(+)

diff --git a/README.md b/README.md
index 2351c77..46b4e7a 100644
--- a/README.md
+++ b/README.md
@@ -115,6 +115,8 @@ From source
In addition to the default CentOS repositories, you may need to add

[PATCH 6/7] t/test.sh: skip two "tree SHA1 didn't change" tests on Cygwin for now.

r...@defaultvalue.org (Rob Browning) — Mon, 27 May 2013 17:53:11 UT

Apparently Cygwin has some unusual behaviors with respect to access
times, so disable these two tests for now.

Possibly related: [link]

Signed-off-by: Rob Browning <r...@defaultvalue.org>
---
README.md | 7 +++++++
t/test.sh | 10 ++++++++--
2 files changed, 15 insertions(+), 2 deletions(-)

[PATCH 3/7] Remove lib.sh actually-root in favor of t/root-status.

r...@defaultvalue.org (Rob Browning) — Mon, 27 May 2013 17:53:08 UT

Create a t/root-status command that outputs root, fake, or none and
that uses the same cross-platform logic as helpers.py is_superuser().
Use it everywhere.

Signed-off-by: Rob Browning <r...@defaultvalue.org>
---
t/lib.sh | 5 -----
t/root-status | 19 +++++++++++++++++++

[PATCH 7/7] Skip the large bloom filter test if mmap fails on 32-bit archtectures.

r...@defaultvalue.org (Rob Browning) — Mon, 27 May 2013 17:53:12 UT

Signed-off-by: Rob Browning <r...@defaultvalue.org>
---
lib/bup/t/tbloom.py | 21 ++++++++++++++++++---
1 file changed, 18 insertions(+), 3 deletions(-)

diff --git a/lib/bup/t/tbloom.py b/lib/bup/t/tbloom.py
index de89e66..2f1cce5 100644
--- a/lib/bup/t/tbloom.py
+++ b/lib/bup/t/tbloom.py

[PATCH 2/7] metadata.py: use socket() instead of mknod(...IF_SOCK) on Cygwin.

r...@defaultvalue.org (Rob Browning) — Mon, 27 May 2013 17:53:07 UT

Apparently os.mknod(...IF_SOCK) fails on Cygwin, but our t/mksock tool
(which uses socket()) works just fine, so use that in
_create_via_common_rec() when on Cygwin.

Signed-off-by: Rob Browning <r...@defaultvalue.org>
---
lib/bup/metadata.py | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)

Re: Bup segfaults when trying to write to remote on busybox

ecthell...@gmail.com — Mon, 27 May 2013 17:38:41 UT

1GB physical, 2GB swap; I don't think this is really a problem though I
haven't tried watching while doing a backup.

I did find this; it's been working fine since I enabled it.
Thanks!

Re: Added mmap to "Things that are stupid" (potential area to help)

r...@defaultvalue.org (Rob Browning) — Mon, 27 May 2013 16:34:05 UT

I'm also wondering what's the right thing to do with respect to the
final test in tbloom.py (in the short term).

Right now a failure to mmap a bit over 1GB is fatal, which can cause
make check to fail on s390, Cygwin, etc.

We could change the test to just print a message and continue if

Bup-on support for restorations?

ja...@lottspot.com (James Lott) — Sun, 26 May 2013 20:02:04 UT

Does the 'bup on' command support any kind of restoration of a backup
to a server in addition to indexing/pulling backups?

No biggie if it doesn't, just means I'll have to put in some work with
my own tools :)

Great project btw! Thanks for all your hard work on it, and some day
when I tighten up my C, I'll come back to contribute ;)

Re: Added mmap to "Things that are stupid" (potential area to help)

apenw...@gmail.com (Avery Pennarun) — Sat, 25 May 2013 21:57:33 UT

Of course there is plenty of room to further optimize that FakeMmap
implementation. In particular, the first few indexes in the MRU list
should probably be mmaped, and it's maybe better for other ones not to
be. But it's more important to fix the total non-functioning than to
make it super fast right away.

Added mmap to "Things that are stupid" (potential area to help)

r...@defaultvalue.org (Rob Browning) — Sat, 25 May 2013 21:34:09 UT

After some investigation, I've added a section to "Things that are
stupid" in the README.md that describes the current problems with mmap,
and one potential solution (originally suggested by Avery).

This might be a good, relatively standalone bit of work for anyone who'd
like to help. It'll mostly benefit people running bup on <= 32-bit

Re: [PATCH 5/5] List unreferenced commits by date

t.riemenschnei...@detco.de (Tim Riemenschneider) — Wed, 22 May 2013 12:46:05 UT

Am Mittwoch, 22. Mai 2013 00:40:29 UTC+2 schrieb Tim Riemenschneider:
Okay, after reading "Git Revision Selection" (
[link]) again, what I really, really meant was
"branch~1", "branch~2" and so on.
But the rest stays the same, rev_parse from git.py only supports
"[link]" and

Re: Show history of file

z...@zoranzaric.de (Zoran Zaric) — Tue, 21 May 2013 22:46:03 UT

With the given information (name mangling) the implementation of a bup specific wrapper for this use case would be a nice exercise for the reader, too.

Thanks,
Zoran

Am 21.05.2013 um 22:29 schrieb Tim Riemenschneider <t.riemenschnei...@detco.de>:

Re: [PATCH 5/5] List unreferenced commits by date

t.riemenschnei...@detco.de (Tim Riemenschneider) — Tue, 21 May 2013 22:40:29 UT

Am Dienstag, 21. Mai 2013 23:57:13 UTC+2 schrieb Tim Riemenschneider:
After looking at git.py a bit longer, I found a working solution.
(The code could maybe get some cleanup:
"rev_parse" uses "read_ref" which uses "list_refs" which uses "git show-ref
-- <given ref>" which does not understand "branch^1" (or "branch@{1}",

Re: [PATCH 5/5] List unreferenced commits by date

t.riemenschnei...@detco.de (Tim Riemenschneider) — Tue, 21 May 2013 21:57:13 UT

Am Donnerstag, 16. Mai 2013 14:20:21 UTC+2 schrieb yungchin:
I'm not sure, can HEAD actually point to a commit (instead of a branch?)
EDIT: Yes it can. However it works not really "out-of-the-box", the first
time you use it, you probably have to write the file "HEAD" yourself.
I tested that some dummy-repository:

Re: Show history of file

t.riemenschnei...@detco.de (Tim Riemenschneider) — Tue, 21 May 2013 20:29:41 UT

I gave it a try, "git log" does not need a working directory.
However, if you don't specify some branches/refs, it (tries to) default to
HEAD, but bup-repositories don't have a HEAD (output of git log: "fatal:
bad default revision 'HEAD'")
But running "git log --follow test -- some/path/in/the/backup", with "test"

Re: Pruning

t.riemenschnei...@detco.de (Tim Riemenschneider) — Tue, 21 May 2013 19:44:18 UT

Since I tested Zorans code a bit (quite some time ago....), I want to take
the opportunity to share some findings:
Since the code is a repack-command, ie. it "removes" from the repository
objects that are not referenced by any branch, the first objective is to
"unreference" the backups that you want to delete.

Re: Show history of file

lelu...@gmail.com (Gabriel Filion) — Mon, 20 May 2013 03:48:38 UT

in general, git log does work well with bup repositories. I'm not quite
sure for specific options for the command though..

although, it seems git log doesn't quite enjoy trying to search for
paths when we don't have a working tree. maybe I'm missing something.

--
Gabriel Filion

Re: Pruning

lelu...@gmail.com (Gabriel Filion) — Mon, 20 May 2013 03:34:24 UT

If you'd like to test it out (either on a _copy_ of your repo, or on a
dummy repository), and maybe if you'd like to improve on it, Zoran's
work is here:

[link]

For now it implements a repack command. When the repack code is
functional and fast enough, we can implement a "delete" command on top