Exception mmap.error in "bup save"

[Jed Brown]

I'm trying to back up a 170 GB volume. I set BUP_DIR to point at my NAS
(cifs mount) because I don't want to keep bup's object store on my
laptop SSD. The following ran for 15 minutes at 15 MB/s compressing
about 14 GB to 3 GB, then crashed. This is a 64-bit Linux system with
10 GB of RAM, using bup-0.25 (and git-1.8.5.2). Is this the "overly
optimistic mmap" problem mentioned in the README (talking about 32-bit
systems)? The file mentioned below is a normal file and can be read
fine. Anything I should do to help debug this?

$ bup save -n home-jed .
Reading index: 2413544, done.
bloom: creating from 2 files (219255 objects).
bloom: adding 1 file (200000 objects).
bloom: creating from 4 files (619255 objects).
bloom: adding 1 file (200000 objects).
bloom: adding 1 file (200000 objects).
bloom: adding 1 file (200000 objects).
/home/jed/usr/emacs/share/emacs/24.3.50/lisp/mh-e/mh-mime.el.gz: [Errno 5] Input/output error
Traceback (most recent call last):
File "/usr/lib/bup/cmd/bup-save", line 365, in <module>
keep_boundaries=False)
File "/usr/lib/bup/bup/hashsplit.py", line 171, in split_to_blob_or_tree
files, keep_boundaries))
File "/usr/lib/bup/bup/hashsplit.py", line 160, in split_to_shalist
for (sha,size,level) in sl:
File "/usr/lib/bup/bup/hashsplit.py", line 114, in split_to_blobs
sha = makeblob(blob)
File "/usr/lib/bup/bup/git.py", line 583, in new_blob
return self.maybe_write('blob', blob)
File "/usr/lib/bup/bup/git.py", line 576, in maybe_write
self._write(sha, type, content)
File "/usr/lib/bup/bup/git.py", line 551, in _write
self.breakpoint()
File "/usr/lib/bup/bup/git.py", line 556, in breakpoint
id = self._end()
File "/usr/lib/bup/bup/git.py", line 640, in _end
obj_list_sha = self._write_pack_idx_v2(self.filename + '.idx', idx, packbin)
File "/usr/lib/bup/bup/git.py", line 671, in _write_pack_idx_v2
assert(count == self.count)
AssertionError
Exception mmap.error: (9, 'Bad file descriptor') in <bound method Reader.__del__ of <bup.index.Reader instance at 0x20d5f38>> ignored

[Greg Troxel]

Jed Brown <j...@jedbrown.org> writes:

> I'm trying to back up a 170 GB volume. I set BUP_DIR to point at my NAS
> (cifs mount) because I don't want to keep bup's object store on my
> laptop SSD. The following ran for 15 minutes at 15 MB/s compressing
> about 14 GB to 3 GB, then crashed. This is a 64-bit Linux system with
> 10 GB of RAM, using bup-0.25 (and git-1.8.5.2). Is this the "overly
> optimistic mmap" problem mentioned in the README (talking about 32-bit
> systems)? The file mentioned below is a normal file and can be read
> fine. Anything I should do to help debug this?

One thing to try would be to use BUP_DIR on the local machine (no cifs)
and to use -d (remote save location) on the save command. That will
separate whether the problem is about dealing with the index vs the save
directory. I believe bup only writes about 1GB packfiles in the save
directory, so you shouldn't be running into issues there.

[Jed Brown]

Greg Troxel <g...@lexort.com> writes:

> Jed Brown <j...@jedbrown.org> writes:
>
>> I'm trying to back up a 170 GB volume. I set BUP_DIR to point at my NAS
>> (cifs mount) because I don't want to keep bup's object store on my
>> laptop SSD. The following ran for 15 minutes at 15 MB/s compressing
>> about 14 GB to 3 GB, then crashed. This is a 64-bit Linux system with
>> 10 GB of RAM, using bup-0.25 (and git-1.8.5.2). Is this the "overly
>> optimistic mmap" problem mentioned in the README (talking about 32-bit
>> systems)? The file mentioned below is a normal file and can be read
>> fine. Anything I should do to help debug this?
>
> One thing to try would be to use BUP_DIR on the local machine (no cifs)
> and to use -d (remote save location) on the save command.

-d is date, but presumably you mean -r.

> That will separate whether the problem is about dealing with the index
> vs the save directory. I believe bup only writes about 1GB packfiles
> in the save directory, so you shouldn't be running into issues there.

It's running, and having BUP_DIR on the local SSD has sped up the
aggregate transfer rate from 15 MB/s to 25 MB/s. Bup seems to have
successfully reused what was written on the first try. I presume this
is recommended usage?

[Jed Brown]

Jed Brown <j...@jedbrown.org> writes:
> It's running, and having BUP_DIR on the local SSD has sped up the
> aggregate transfer rate from 15 MB/s to 25 MB/s. Bup seems to have
> successfully reused what was written on the first try. I presume this
> is recommended usage?

Different error this time:

$ time bup save -r /buffalo/batura/home/jed.bup -n home-jed .
Reading index: 2417876, done.
Receiving index from server: 5601072/5601072, done.
bloom: creating from 1 file (200000 objects).
Receiving index from server: 5601072/5601072, done.

bloom: adding 1 file (200000 objects).

Receiving index from server: 5601072/5601072, done.
bloom: creating from 3 files (600000 objects).
Receiving index from server: 540212/540212, done.
bloom: adding 1 file (19255 objects).
Receiving index from server: 5601072/5601072, done.

bloom: adding 1 file (200000 objects).

Receiving index from server: 5601072/5601072, done.

bloom: adding 1 file (200000 objects).

Receiving index from server: 5601072/5601072, done.

bloom: adding 1 file (200000 objects).

bloom: adding 1 file (199772 objects).
Receiving index from server: 5594688/5594688, done.
bloom: adding 1 file (199772 objects).

bloom: adding 1 file (200000 objects).

Receiving index from server: 5601072/5601072, done.

bloom: adding 1 file (200000 objects).

[Errno 13] Permission denied: '/home/jed/tmp/grub.cfg'
bloom: creating from 10 files (1819027 objects). s/s
Receiving index from server: 5601072/5601072, done.
bloom: creating from 10 files (1819027 objects).

bloom: adding 1 file (200000 objects).

Receiving index from server: 5601072/5601072, done.

bloom: adding 1 file (200000 objects).
bloom: adding 1 file (200000 objects).

Receiving index from server: 5601072/5601072, done.

bloom: adding 1 file (200000 objects).
bloom: adding 1 file (200000 objects).

Receiving index from server: 5601072/5601072, done.

bloom: adding 1 file (200000 objects).
bloom: adding 1 file (200000 objects).

Receiving index from server: 5601072/5601072, done.

bloom: adding 1 file (200000 objects).
bloom: adding 1 file (200000 objects).

Receiving index from server: 5601072/5601072, done.

bloom: adding 1 file (200000 objects).
bloom: adding 1 file (200000 objects).

Receiving index from server: 5601072/5601072, done.

bloom: adding 1 file (200000 objects).
bloom: adding 1 file (200000 objects).

Receiving index from server: 5601072/5601072, done.

bloom: adding 1 file (200000 objects).

bloom: creating from 18 files (3419027 objects).
Receiving index from server: 5601072/5601072, done.
bloom: creating from 18 files (3419027 objects).

bloom: adding 1 file (200000 objects).

Receiving index from server: 5601072/5601072, done.

bloom: adding 1 file (200000 objects).
bloom: adding 1 file (200000 objects).

Receiving index from server: 5601072/5601072, done.

bloom: adding 1 file (200000 objects).

Traceback (most recent call last):

File "/usr/lib/bup/cmd/bup-server", line 206, in <module>
cmd(conn, rest)
File "/usr/lib/bup/cmd/bup-server", line 88, in receive_objects_v2
fullpath = w.close(run_midx=not dumb_server_mode)
File "/usr/lib/bup/bup/git.py", line 654, in close
return self._end(run_midx=run_midx)
File "/usr/lib/bup/bup/git.py", line 638, in _end
f.close()
IOError: [Errno 5] Input/output error

Traceback (most recent call last):
File "/usr/lib/bup/cmd/bup-save", line 365, in <module>
keep_boundaries=False)

File "/usr/lib/bup/bup/hashsplit.py", line 177, in split_to_blob_or_tree
return (GIT_MODE_TREE, maketree(shalist))
File "/usr/lib/bup/bup/git.py", line 588, in new_tree
return self.maybe_write('tree', content)

File "/usr/lib/bup/bup/git.py", line 576, in maybe_write
self._write(sha, type, content)
File "/usr/lib/bup/bup/git.py", line 551, in _write
self.breakpoint()
File "/usr/lib/bup/bup/git.py", line 556, in breakpoint
id = self._end()

File "/usr/lib/bup/bup/client.py", line 313, in _end
return self.suggest_packs() # Returns last idx received
File "/usr/lib/bup/bup/client.py", line 228, in _suggest_packs
self.check_ok()
File "/usr/lib/bup/bup/client.py", line 131, in check_ok
% rv)
bup.client.ClientError: server exited unexpectedly with code 1
2354.383 real 1325.553 user 114.563 sys 61.16 cpu

[Greg Troxel]

Jed Brown <j...@jedbrown.org> writes:

> Greg Troxel <g...@lexort.com> writes:

>> One thing to try would be to use BUP_DIR on the local machine (no
>> cifs) and to use -d (remote save location) on the save command.

> -d is date, but presumably you mean -r.

Sorry, I did. i have scripts to wrap this and thus don't really know
the options :-(

>> That will separate whether the problem is about dealing with the
>> index vs the save directory. I believe bup only writes about 1GB
>> packfiles in the save directory, so you shouldn't be running into
>> issues there.

> It's running, and having BUP_DIR on the local SSD has sped up the
> aggregate transfer rate from 15 MB/s to 25 MB/s. Bup seems to have
> successfully reused what was written on the first try. I presume this
> is recommended usage?

I would certainly recommend not using remote storage for the index, but
I don't know if that's the group's consensus.

I see you still got an error. So I'd suggest using ktruss/ktrace or
some facility which records system call args and return values. The key
question is what actually fails, and whether it should have failed. I
am a bit skeptical of anything having to do with CIFS, although that
probably isn't fair.

[Jed Brown]

Greg Troxel <g...@lexort.com> writes:
> I see you still got an error. So I'd suggest using ktruss/ktrace or
> some facility which records system call args and return values. The key
> question is what actually fails, and whether it should have failed. I
> am a bit skeptical of anything having to do with CIFS, although that
> probably isn't fair.

I ended up sticking with btrfs snapshots which never encountered such
problems. I'm putting my btrfs volumes in encrypted files on the CIFS
share. Meanwhile, I tripped over a manufacturer BIOS bug so I won't be
able to test again on that machine. If I try bup again and encounter
errors, I'll dig deeper. Thanks.