Filesystems without native hard links

[Daniel Distler]

Hi bup-list,

as of bup 0.33.8 fsck uses hard links (see https://github.com/bup/bup/commit/
9c43667f206bd34fde6c57df89c17eb40bf210af). Unfortunately hard links are not
supported on many filesystems, including e.g. FUSE-based filesystems (e.g.,
CryFS) and older formats like FAT and exFAT.

When bup is run on these filesystems, the `os.link` operation fails. This
results in cryptic errors like `OSError: [Errno 34] Numerical result out of
range`, which is not intuitive for the user.

Are there other ways of listing input files for par instead of linking them
into a tmp folder?

Best,
Daniel

[Rob Browning]

Daniel Distler <daniel.dist...@posteo.de> writes:

> Are there other ways of listing input files for par instead of linking them
> into a tmp folder?

I suppose one possible, simpler option for now would be to fall back to
shutil.copy2() or similar when there's a suitable link failure. If you
like, you could adjust fsck.py to try that instead.

> When bup is run on these filesystems, the `os.link` operation fails. This
> results in cryptic errors like `OSError: [Errno 34] Numerical result out of
> range`, which is not intuitive for the user.

That seems odd -- what system/fs? I ask because linux link(2) says that
should be EPERM.

Thanks
--
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

[Greg Troxel]

Rob Browning <r...@defaultvalue.org> writes:

> Daniel Distler <daniel.dist...@posteo.de> writes:
>
>> Are there other ways of listing input files for par instead of linking them
>> into a tmp folder?
>
> I suppose one possible, simpler option for now would be to fall back to
> shutil.copy2() or similar when there's a suitable link failure. If you
> like, you could adjust fsck.py to try that instead.
>
>> When bup is run on these filesystems, the `os.link` operation fails. This
>> results in cryptic errors like `OSError: [Errno 34] Numerical result out of
>> range`, which is not intuitive for the user.
>
> That seems odd -- what system/fs? I ask because linux link(2) says that
> should be EPERM.

I would have said the right error was EOPNOTSUPP. I looked up POSIX,
and it seems that hard links are simply required.

https://pubs.opengroup.org/onlinepubs/9799919799/functions/link.html

But agreed a code change to do something else if they don't work sounds
good.

[Rob Browning]

Greg Troxel <g...@lexort.com> writes:

> I would have said the right error was EOPNOTSUPP.

Right, looks like that's what FreeBSD does.

> But agreed a code change to do something else if they don't work sounds
> good.

I'll probably adjust the code to catch EOPNOTSUPP and EPERM and try
copy2() or similar in either case.

[Aaron M. Ucko]

Rob Browning <rlb-A9c2TQsEE...@public.gmane.org> writes:

> I'll probably adjust the code to catch EOPNOTSUPP and EPERM and try
> copy2() or similar in either case.

Sounds good to me FWIW, but I'd suggest covering additional error codes;
the OP apparently encountered ERANGE (!) and AFS meanwhile disallows
cross-directory links even within a volume (presumably because its ACLs
have directory granularity), albeit with implementation-dependent error
codes: classic OpenAFS reports EXDEV and Linux's kAFS reports EREMOTEIO.
One can also plausibly make a case for EMLINK.

--
Aaron M. Ucko, KB1CJC (amu at alum.mit.edu, ucko at debian.org)
http://www.mit.edu/~amu/ | http://stuff.mit.edu/cgi/finger/?a...@monk.mit.edu

[Daniel Distler]

I did see the ERANGE with CryFS on Fedora Linux 41. This might well be an
issue in CryFS. I guess instead of explicitly failing the link request with
EPERM, EOPNOTSUPP or something similar the call just passes on some error of
some internal call.

But why does bup need the link / copy? Would it work to replace
```
rc = par2(b'create', [b'-n1', b'-c200', b'--', base, pack, idx],
verb_floor=2, cwd=tmpdir)
```
with something like
```
rc = par2(b'create', [b'-n1', b'-c200', b'--', base,
join(b'..', pack), join(b'..', idx)],
verb_floor=2, cwd=tmpdir)
```
to avoid the issue altogether?

[Rob Browning]

"'Aaron M. Ucko' via bup-list" <bup-...@googlegroups.com> writes:

> Sounds good to me FWIW, but I'd suggest covering additional error codes;
> the OP apparently encountered ERANGE (!)

Oh, right. Forgot about that.

> and AFS meanwhile disallows cross-directory links even within a volume
> (presumably because its ACLs have directory granularity), albeit with
> implementation-dependent error codes: classic OpenAFS reports EXDEV
> and Linux's kAFS reports EREMOTEIO. One can also plausibly make a
> case for EMLINK.

Interesting. I suppose it's likely fine to be fairly permissive here.

[Rob Browning]

Daniel Distler <daniel.dist...@posteo.de> writes:

> But why does bup need the link / copy? Would it work to replace
> ```
> rc = par2(b'create', [b'-n1', b'-c200', b'--', base, pack, idx],
> verb_floor=2, cwd=tmpdir)
> ```
> with something like
> ```
> rc = par2(b'create', [b'-n1', b'-c200', b'--', base,
> join(b'..', pack), join(b'..', idx)],
> verb_floor=2, cwd=tmpdir)
> ```
> to avoid the issue altogether?

Unfortunately par2 isn't exactly designed for our purposes in a number
of ways, and here:

$ par2 create -n1 -c200 -- PAR ../README.md ../DESIGN.md
Ignoring out of basepath source file: /home/rlb/src/bup/main/README.md
Ignoring out of basepath source file: /home/rlb/src/bup/main/HACKING.md
You must specify a list of files when creating.