Bup over smb

[carlo...@gmail.com]

Hello,
would it be possible to store a bup repo on a filesystem mounted using cifs? I made a few backup tests, and everything seemed to work just fine, but then I noticed that pruning always results in an errors like this:

keeping all since 2020-03-31-204429
found 502357 objects
expecting to retain about 0.00% unnecessary objects
clearing midx files
clearing bloom filter
clearing reflog
removing unreachable data
deleting pack-4cc5238*.idx
deleting pack-a285d60*.idx
deleting pack-717d6d3*.idx
deleting pack-84e6f86*.idx
preserving live data (0% complete)
removing pack-4cc5238deafc6e6190e583f803190be3dc5cfcd7.idx
removing pack-4cc5238deafc6e6190e583f803190be3dc5cfcd7.pack
removing pack-a285d605e745e2908402db343d560c2db0437153.idx
removing pack-a285d605e745e2908402db343d560c2db0437153.pack
removing pack-717d6d38e7b0199c447a041540430bf004ec3b2d.idx
removing pack-717d6d38e7b0199c447a041540430bf004ec3b2d.pack
removing pack-84e6f86b23233d933f4521e30c87f9521744c09d.idx
removing pack-84e6f86b23233d933f4521e30c87f9521744c09d.pack
Traceback (most recent call last):
File "/usr/lib/bup/cmd/bup-prune-older", line 144, in <module>
verbosity=opt.verbose)
File "/usr/lib/bup/bup/gc.py", line 244, in bup_gc
verbosity)
File "/usr/lib/bup/bup/gc.py", line 212, in sweep
/ float(existing_count) * 100))
File "/usr/lib/bup/bup/gc.py", line 53, in count_objects
idx = git.open_idx(idx_name)
File "/usr/lib/bup/bup/git.py", line 547, in open_idx
f = open(filename, 'rb')
IOError: [Errno 2] No such file or directory: '.../bup_test/objects/pack/pack-84e6f86b23233d933f4521e30c87f9521744c09d.idx'

Backing up again on the same repo seemed to work. I tried many times but I always got errors like this. I did the same test on my ext4 filesystem and everything worked properly. The only information I found about smb is https://groups.google.com/forum/#!msg/bup-list/7xqQbGzh9Tk/o0M3-FI9hygJ, but maybe pruning is not supported?
Thank you for any info.
Regards.

[Greg Troxel]

carlo...@gmail.com writes:

> would it be possible to store a bup repo on a filesystem mounted using
> cifs? I made a few backup tests, and everything seemed to work just fine,
> but then I noticed that pruning always results in an errors like this:

I suspect the snarky view of unix types is that cifs is a filesystem
that almost works. But really I don't know.

Hard to say what's going on. I wonder if this was a file that was just
written, and it hasn't finished being written to storage yet and thus
doesn't appear. A filesystem should present a consistent view even if
not synced. But I'm just wildly guessing here.

You could try to put a sleep(30) in open_idx and see if that helps.

> Backing up again on the same repo seemed to work. I tried many times but I
> always got errors like this. I did the same test on my ext4 filesystem and
> everything worked properly. The only information I found about smb is
> https://groups.google.com/forum/#!msg/bup-list/7xqQbGzh9Tk/o0M3-FI9hygJ,
> but maybe pruning is not supported?

amusing. So six years ago I said:

"using SMB instead should be equivalent, modulo difficult questions
about locking and write ordering that with a single client and no
crashes that you can get away with ignoring."

which was a guess then. I stand by the notion that I am guessing.

[carlo...@gmail.com]

> Hard to say what's going on. I wonder if this was a file that was just
> written, and it hasn't finished being written to storage yet and thus
> doesn't appear. A filesystem should present a consistent view even if
> not synced. But I'm just wildly guessing here.

> You could try to put a sleep(30) in open_idx and see if that helps.

Please, note that I know nothing about the internals of bup (nor git).

By this, you mean that the file pack-84e6f86b23233d933f4521e30c87f9521744c09d.idx is removed and rewritten by the gc operation, and then quickly opened in open_idx?

Thanks for your answer.

[Greg Troxel]

Yes, I mean something that is perhaps sort of like that.

I mean to read the code to find the place where it is doing the open
that fails, and insert a sleep.

It may also be that there is a gc bug; that is labeled dangerous for a
reason (not enough experience to be comfortable).

[carlo...@gmail.com]

> I mean to read the code to find the place where it is doing the open
> that fails, and insert a sleep. 

I placed a 60 seconds sleep before opening and I printed the path of the file about to be opened. The first two idx files were opened properly, the last one (the one that fails), is actually there, but cannot be opened:

luca @ .../bup_test/objects/pack - [] $ ls -lh

total 4,0M

-rwxr-xr-x 0 luca luca 2,8M apr  1 19:40 pack-6651f6288d1d3c734375ca123a6bb685001f539c.idx

-rwxr-xr-x 0 luca luca 1,1M apr  1 19:48 tmp-gc-jMATQN.bloom

luca @ .../bup_test/objects/pack - [] $ hexdump pack-6651f6288d1d3c734375ca123a6bb685001f539c.idx

hexdump: pack-6651f6288d1d3c734375ca123a6bb685001f539c.idx: No such file or directory

after bup fails, the file is removed.

I tried to write a file like this:

dd if=/dev/urandom of=test count=$((5*1024)) bs=1024

and hexdump reads it immediately. Maybe it depends on how it is written?

> It may also be that there is a gc bug; that is labeled dangerous for a
> reason (not enough experience to be comfortable).

Yes, I read about it. I opened this discussion cause I thought it could be a possible improvement or fix for bup.

Thanks for your answers.

[Greg Troxel]

carlo...@gmail.com writes:

>> I mean to read the code to find the place where it is doing the open
>> that fails, and insert a sleep.
>
> I placed a 60 seconds sleep before opening and I printed the path of the
> file about to be opened. The first two idx files were opened properly, the
> last one (the one that fails), is actually there, but cannot be opened:

Good plan about printing and checking.

> luca @ .../bup_test/objects/pack - [] $ ls -lh
> total 4,0M
> -rwxr-xr-x 0 luca luca 2,8M apr 1 19:40 pack-6651f6288d1d3c734375ca123a6bb685001f539c.idx
> -rwxr-xr-x 0 luca luca 1,1M apr 1 19:48 tmp-gc-jMATQN.bloom
> luca @ .../bup_test/objects/pack - [] $ hexdump pack-6651f6288d1d3c734375ca123a6bb685001f539c.idx
> hexdump: pack-6651f6288d1d3c734375ca123a6bb685001f539c.idx: No such file or
> directory
>
> after bup fails, the file is removed.

So that more or less proves that your filesystem is not working
correctly.

I wonder if the issue is that it's been written but close() has not
happened from when it was written, or something.

This could be about your OS support for cifs, vs cifs itself. Is this
linux? How are you mounting?

Do you have an option to mount differently, via fuse, instead of native?

> I tried to write a file like this:
>
> dd if=/dev/urandom of=test count=$((5*1024)) bs=1024
>
> and hexdump reads it immediately. Maybe it depends on how it is written?

Could be. Try

(dd if=/dev/urandom count=$((5*1024)) bs=1024; sleep 120) > test &

and see if you can read it before the command finishes.

>> It may also be that there is a gc bug; that is labeled dangerous for a
>> reason (not enough experience to be comfortable).
>
> Yes, I read about it. I opened this discussion cause I thought it could be
> a possible improvement or fix for bup.

It's great to work on making it work right. I just wanted to caution
you that you are in semi-charted territory; perhaps "gc AND cifs" is
uncharted. Works for anybody else?

> Thanks for your answers.

You're very welcome - figuring these things out helps everybody.

[Luca Carlon]

> This could be about your OS support for cifs, vs cifs itself. Is this
> linux? How are you mounting?

This is a Linux machine, Kubuntu 19.10, mounting samba server running
on another linux server.
I tried two ways. The first one was with these options:
rw,relatime,vers=3.1.1,cache=strict,username=...,uid=1000,forceuid,gid=1000,forcegid,addr=...,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1
then I tried without caches:
rw,relatime,vers=3.1.1,cache=none,username=...,uid=1000,forceuid,gid=1000,forcegid,addr=...,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1

The test with hexdump above was using the second mount option set.

> Do you have an option to mount differently, via fuse, instead of native?

According to what I read online, I guess cifs kernel driver is the
modern typical way of mounting shares. I read about smbnetfs but it
seems to fail on my machine.

> Could be. Try
>
> (dd if=/dev/urandom count=$((5*1024)) bs=1024; sleep 120) > test &
>
> and see if you can read it before the command finishes.

Yes, I can read it properly both while sleep is running and even while
dd is still running. I tried many times and I got no error response
from hexdump, not even with the cache enabled.
Regards.

> >> It may also be that there is a gc bug; that is labeled dangerous for a
> >> reason (not enough experience to be comfortable).
> >
> > Yes, I read about it. I opened this discussion cause I thought it could be
> > a possible improvement or fix for bup.
>
> It's great to work on making it work right. I just wanted to caution
> you that you are in semi-charted territory; perhaps "gc AND cifs" is
> uncharted. Works for anybody else?
>
> > Thanks for your answers.
>
> You're very welcome - figuring these things out helps everybody.

--
Dr. Luca Carlon
Software Engineer