Interrupted backup process

[Alphazo]

Hi,

I'm in a middle of a significant bup save process (83K+ files, 700GB+, stored on an external drive ) to a network share. That should be a good test case for 0.25rc3.

I was wondering how bup would handle sudden hard drive disconnect or loss of network share.

Would it corrupt the repository? Would I be able to restar the processt where it stopped (like rsync)?

Thanks
Alphazo

[Alphazo]

Well it looks like I shouldn't have sent my previous email as my laptop has frozen about ten minutes after I sent it.

Resetarted the index and save and apparently it starts from where it stopped. I will run a bup fsck to make sure.

PS: For the records this is about 694GB, 18950 folders and 65025 files.

[Avery Pennarun]

bup is pretty smart about recovering. It only trusts that a
particular object is on the server if the object is in a git .idx
file. The .idx files are generated only after a pack has been fully
written and then indexed, and the index has been moved into place
atomically. And the bupindex file, while it does contain a list of
"this object has this sha1sum and is not marked dirty" sorts of
entries, is not considered fully trustworthy. bup will make sure the
remote server has a given sha1sum before it trusts the "not dirty"
indicator in the bupindex.

So yes, it should recover fine after a half-completed backup. At most
you'd lose about 1 pack's worth of stuff (about 1GB) that would then be
re-backed-up. Probably you won't even lose that because the bup
server will detect that the client is gone, and finalize that last
pack anyway before exiting.

> --
> You received this message because you are subscribed to the Google Groups
> "bup-list" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to bup-list+u...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Alphazo]

Thanks for the detailled explanation. I'm now facing some weird behaviors.

I did a couple of backup of my external drive on my laptop using the following commands and it worked fine (and fast)

# sudo mount.cifs //server/BACKUPS/bup /mnt/babylon-bup -o guest,uid=1000,gid=100,rw
# export BUP_DIR=/mnt/babylon-bup
# bup index -uv /run/media/alphazo/6a12ef63-f389-4548-8127-e03f6a3a3d66/PHOTOS
Indexing: 83975, done.
bup index -uv /run/media/alpha/6a12ef63-f389-4548-8127-e03f6a3a3d66/PHOTOS  25,09s user 1,92s system 50% cpu 53,336 total
# bup save -n PHOTOS /run/media/alpha/6a12ef63-f389-4548-8127-e03f6a3a3d66/PHOTOS

Then moved that portable HDD to a desktop machine and tried to perform a backup knowing that almost no files have been modified, mostly renamed and moved but maybe 200 out of 80,000.

My portable HDD was not mounted at the same location so I used:

# sudo mount.cifs //server/BACKUPS/bup /mnt/babylon-bup -o guest,uid=1000,gid=100,rw
# export BUP_DIR=/mnt/babylon-bup
# bup index -uv /run/media/john/6a12ef63-f389-4548-8127-e03f6a3a3d66/PHOTOS
Indexing: 83975, done.
bup index -uv /run/media/john/6a12ef63-f389-4548-8127-e03f6a3a3d66/PHOTOS  25,09s user 1,92s system 50% cpu 53,336 total
# bup save -n PHOTOS /run/media/john/6a12ef63-f389-4548-8127-e03f6a3a3d66/PHOTOS

But the process said it was going to take ages to backup the 80,000+ files. I interrupted the process and went back to the original machine that has the former mountpoint but I get the same message.

FYI, I since changed the home folder on my desktop machine to match the one on my laptop.

What went wrong? Am I stuck at saving back all the files?

Thanks

[Rob Browning]

Alphazo <alp...@gmail.com> writes:

> But the process said it was going to take ages to backup the 80,000+ files.
> I interrupted the process and went back to the original machine that has
> the former mountpoint but I get the same message.
>
> FYI, I since changed the home folder on my desktop machine to match the one
> on my laptop.
>
> What went wrong? Am I stuck at saving back all the files?

I haven't had a chance to check your results carefully yet, but I
noticed that you used an external drive. Offhand I wonder if the device
number changed, which caused bup to think all the files might be new.

If that's the problem, see --no-check-device in "bup help index".

--
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

[Alphazo]

Running stat() on both machines showed a different ID (maybe the encrypted external HDD). So I wiped the entire bup directory on the server (CIFS mount) and started again with the --no-check-device.

I then went to the other machine and using the same option was using to index and save. I then renamed one directory to see the increase on the dataset and to my surprise it went up (same files). I renamed it again and during the backup process got a crash:

# bup index -uv --no-check-device   22,93s user 2,49s system 76% cpu 33,345 total
# bup save -n PHOTOS /run/media/alpha/6a84f063-0e89-4147-8127-ab3f6a3a3def/PHOTOS
Reading index: 34, done.
Saving: 100.00% (536372/536372k, 34/34 files), done.     
['/usr/bin/bup', 'midx', '--auto', '--dir', '/mnt/babylon-bup/objects/pack']: returned -7
bloom: adding 1 file (13 objects).
WARNING: 1 errors encountered while saving.
Exception mmap.error: (9, 'Bad file descriptor') in <bound method Reader.__del__ of <bup.index.Reader instance at 0x14c08c0>> ignored
bup save -n PHOTOS   24,35s user 7,63s system 4% cpu 10:46,66 total

I then ran index again followed by a save command a time consuming process started:
# bup save -n PHOTOS /run/media/alpha/6a84f063-0e89-4147-8127-ab3f6a3a3def/PHOTOS
Reading index: 1, done.
Saving: 100.00% (0/0k, 1/1 files), done.   
midx: writing 1.31% (614544/46791888)

What could be the reason for the crash? What is bup doing after the crash?

Thanks

Alphazo