Filesystems without native hard links

[Daniel Distler]

Hi bup-list,

as of bup 0.33.8 fsck uses hard links (see https://github.com/bup/bup/commit/
9c43667f206bd34fde6c57df89c17eb40bf210af). Unfortunately hard links are not
supported on many filesystems, including e.g. FUSE-based filesystems (e.g.,
CryFS) and older formats like FAT and exFAT.

When bup is run on these filesystems, the `os.link` operation fails. This
results in cryptic errors like `OSError: [Errno 34] Numerical result out of
range`, which is not intuitive for the user.

Are there other ways of listing input files for par instead of linking them
into a tmp folder?

Best,
Daniel

[Rob Browning]

Daniel Distler <daniel.dist...@posteo.de> writes:

> Are there other ways of listing input files for par instead of linking them
> into a tmp folder?

I suppose one possible, simpler option for now would be to fall back to
shutil.copy2() or similar when there's a suitable link failure. If you
like, you could adjust fsck.py to try that instead.

> When bup is run on these filesystems, the `os.link` operation fails. This
> results in cryptic errors like `OSError: [Errno 34] Numerical result out of
> range`, which is not intuitive for the user.

That seems odd -- what system/fs? I ask because linux link(2) says that
should be EPERM.

Thanks
--
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

[Greg Troxel]

Rob Browning <r...@defaultvalue.org> writes:

> Daniel Distler <daniel.dist...@posteo.de> writes:
>
>> Are there other ways of listing input files for par instead of linking them
>> into a tmp folder?
>
> I suppose one possible, simpler option for now would be to fall back to
> shutil.copy2() or similar when there's a suitable link failure. If you
> like, you could adjust fsck.py to try that instead.
>
>> When bup is run on these filesystems, the `os.link` operation fails. This
>> results in cryptic errors like `OSError: [Errno 34] Numerical result out of
>> range`, which is not intuitive for the user.
>
> That seems odd -- what system/fs? I ask because linux link(2) says that
> should be EPERM.

I would have said the right error was EOPNOTSUPP. I looked up POSIX,
and it seems that hard links are simply required.

https://pubs.opengroup.org/onlinepubs/9799919799/functions/link.html

But agreed a code change to do something else if they don't work sounds
good.

[Rob Browning]

Greg Troxel <g...@lexort.com> writes:

> I would have said the right error was EOPNOTSUPP.

Right, looks like that's what FreeBSD does.

> But agreed a code change to do something else if they don't work sounds
> good.

I'll probably adjust the code to catch EOPNOTSUPP and EPERM and try
copy2() or similar in either case.

[Aaron M. Ucko]

Rob Browning <rlb-A9c2TQsEE...@public.gmane.org> writes:

> I'll probably adjust the code to catch EOPNOTSUPP and EPERM and try
> copy2() or similar in either case.

Sounds good to me FWIW, but I'd suggest covering additional error codes;
the OP apparently encountered ERANGE (!) and AFS meanwhile disallows
cross-directory links even within a volume (presumably because its ACLs
have directory granularity), albeit with implementation-dependent error
codes: classic OpenAFS reports EXDEV and Linux's kAFS reports EREMOTEIO.
One can also plausibly make a case for EMLINK.

--
Aaron M. Ucko, KB1CJC (amu at alum.mit.edu, ucko at debian.org)
http://www.mit.edu/~amu/ | http://stuff.mit.edu/cgi/finger/?a...@monk.mit.edu

[Daniel Distler]

I did see the ERANGE with CryFS on Fedora Linux 41. This might well be an
issue in CryFS. I guess instead of explicitly failing the link request with
EPERM, EOPNOTSUPP or something similar the call just passes on some error of
some internal call.

But why does bup need the link / copy? Would it work to replace
```
rc = par2(b'create', [b'-n1', b'-c200', b'--', base, pack, idx],
verb_floor=2, cwd=tmpdir)
```
with something like
```
rc = par2(b'create', [b'-n1', b'-c200', b'--', base,
join(b'..', pack), join(b'..', idx)],
verb_floor=2, cwd=tmpdir)
```
to avoid the issue altogether?

[Rob Browning]

"'Aaron M. Ucko' via bup-list" <bup-...@googlegroups.com> writes:

> Sounds good to me FWIW, but I'd suggest covering additional error codes;
> the OP apparently encountered ERANGE (!)

Oh, right. Forgot about that.

> and AFS meanwhile disallows cross-directory links even within a volume
> (presumably because its ACLs have directory granularity), albeit with
> implementation-dependent error codes: classic OpenAFS reports EXDEV
> and Linux's kAFS reports EREMOTEIO. One can also plausibly make a
> case for EMLINK.

Interesting. I suppose it's likely fine to be fairly permissive here.

[Rob Browning]

Daniel Distler <daniel.dist...@posteo.de> writes:

> But why does bup need the link / copy? Would it work to replace
> ```
> rc = par2(b'create', [b'-n1', b'-c200', b'--', base, pack, idx],
> verb_floor=2, cwd=tmpdir)
> ```
> with something like
> ```
> rc = par2(b'create', [b'-n1', b'-c200', b'--', base,
> join(b'..', pack), join(b'..', idx)],
> verb_floor=2, cwd=tmpdir)
> ```
> to avoid the issue altogether?

Unfortunately par2 isn't exactly designed for our purposes in a number
of ways, and here:

$ par2 create -n1 -c200 -- PAR ../README.md ../DESIGN.md
Ignoring out of basepath source file: /home/rlb/src/bup/main/README.md
Ignoring out of basepath source file: /home/rlb/src/bup/main/HACKING.md
You must specify a list of files when creating.

[Tomas Hnyk]

Hello all,

On Saturday, September 20, 2025 at 9:44:47 AM UTC+2 Daniel Distler wrote:

Hi bup-list,

as of bup 0.33.8 fsck uses hard links (see https://github.com/bup/bup/commit/
9c43667f206bd34fde6c57df89c17eb40bf210af). Unfortunately hard links are not
supported on many filesystems, including e.g. FUSE-based filesystems (e.g.,
CryFS) and older formats like FAT and exFAT.

When bup is run on these filesystems, the `os.link` operation fails. This
results in cryptic errors like `OSError: [Errno 34] Numerical result out of
range`, which is not intuitive for the user.

Am I running into this bug when I try to back up to an exFat external harddrive? When I did the backup to my second internal btrfs drive, it completed succesfully. I am using KDE's kup frontend, but reading the logs it seems the problem is with kup. Trying to use the exfat drive, I get this error message:

bup "-d" "/run/media/drew/T5EVO/Zaloha/KDE_Backup-Kup" "fsck" "-g" "-j" "4"
exception: PermissionError(1, 'Operation not permitted')
exception: PermissionError(1, 'Operation not permitted')
exception: PermissionError(1, 'Operation not permitted')
[repeats 300 times]
exception: PermissionError(1, 'Operation not permitted')

Exit code: 99
Kup did not successfully complete the bup backup job: failed to generate recovery info.

 

I use exfat on that harddrive so that it is interoperable with other people's computers, so using a different filesystem is not really feasible. I will patiently wait for a fix if this is indeed this problem, in the meantime I guess I can backup to the other internal btrfs harddrive and just copy a tar archive of bup's backup directory to my exfat file, but obviously, that is a bit annoying :-).
Kind regards

Tomas

 

[Rob Browning]

Tomas Hnyk <toma...@gmail.com> writes:

> as of bup 0.33.8 fsck uses hard links (see
> https://github.com/bup/bup/commit/
> 9c43667f206bd34fde6c57df89c17eb40bf210af).

Right, because par2 1.0 broke the use of symlinks. I suppose we could
try to copy the files if hardlinking fails. That would mean copying up
to about a gigabyte (by default) of data into the temporary directory.

It looked like you mentioned both EPERM and ERANGE. I don't see ERANGE
mentioned in at least Linux's link(2). Was that during fsck, or
elsewhere?

Thanks

[Rob Browning]

Rob Browning <r...@defaultvalue.org> writes:

> Right, because par2 1.0 broke the use of symlinks. I suppose we could
> try to copy the files if hardlinking fails. That would mean copying up
> to about a gigabyte (by default) of data into the temporary directory.

Oh, right, this is already the case in main, just not in 0.33.x. I
might take a look at backporting the fallback, but it should be "fixed"
by 0.34, regardless.

> It looked like you mentioned both EPERM and ERANGE. I don't see ERANGE
> mentioned in at least Linux's link(2). Was that during fsck, or
> elsewhere?

Nevermind --- already addressed this in the changes in main.

[Rob Browning]

If the platform/filesystem doesn't appear to support hard links, fall
back to copy2(). See also:

commit daa6bf3eeabf1f50630b17b0cbbbdcbbef59c90e
fsck: switch from symlinks to hard links for par2 sandbox

Thanks to Daniel Distler for reporting the issue and Greg Troxel,
and Aaron M. Ucko for help with the errno values.

Signed-off-by: Rob Browning <r...@defaultvalue.org>
Tested-by: Rob Browning <r...@defaultvalue.org>
(adapted from commit 337173a9f20fd9aa94c0da348e43d671fecc3d79)
---

Proposed for 0.33.x.

lib/bup/cmd/fsck.py | 27 ++++++++++++++++++++++++---
note/0.33.x.md | 15 +++++++++++++++
2 files changed, 39 insertions(+), 3 deletions(-)
create mode 100644 note/0.33.x.md

diff --git a/lib/bup/cmd/fsck.py b/lib/bup/cmd/fsck.py
index abdcbe2d..9d5cd937 100644
--- a/lib/bup/cmd/fsck.py
+++ b/lib/bup/cmd/fsck.py
@@ -4,7 +4,8 @@ from subprocess import PIPE
from tempfile import mkdtemp
from binascii import hexlify
from os.path import join
-import glob, os, subprocess, sys
+from shutil import copy2
+import errno, glob, os, subprocess, sys

from bup import options, git
from bup.compat import argv_bytes
@@ -86,6 +87,16 @@ def par2(action, args, verb_floor=0, cwd=None):
cmd.extend(args)
return run(cmd, cwd=cwd)

+_unable_to_link = set()
+_unable_to_link.add(getattr(errno, 'EMLINK', None))
+_unable_to_link.add(getattr(errno, 'EOPNOTSUPP', None)) # freebsd
+_unable_to_link.add(getattr(errno, 'EPERM', None)) # linux
+_unable_to_link.add(getattr(errno, 'ERANGE', None)) # cryfs
+_unable_to_link.add(getattr(errno, 'EREMOTEIO', None)) # kafs (cross-directory)
+_unable_to_link.add(getattr(errno, 'EXDEV', None)) # openafs (cross-directory)
+_unable_to_link.discard(None)
+_unable_to_link = frozenset(_unable_to_link)
+
def par2_generate(stem):
parent, base = os.path.split(stem)
# Work in a temp_dir because par2 was observed creating empty
@@ -94,8 +105,18 @@ def par2_generate(stem):
with temp_dir(dir=parent, prefix=(base + b'-bup-tmp-')) as tmpdir:
idx = base + b'.idx'
pack = base + b'.pack'
- os.link(join(tmpdir, b'..', idx), join(tmpdir, idx))
- os.link(join(tmpdir, b'..', pack), join(tmpdir, pack))
+ for entry in idx, pack:
+ entry_src = join(tmpdir, b'..', entry)
+ entry_dst = join(tmpdir, entry)
+ copy_instead = False
+ try:
+ os.link(entry_src, entry_dst)
+ except OSError as ex:
+ if not ex.errno in _unable_to_link:
+ raise
+ copy_instead = True
+ if copy_instead:
+ copy2(entry_src, entry_dst)

rc = par2(b'create', [b'-n1', b'-c200', b'--', base, pack, idx],
verb_floor=2, cwd=tmpdir)

if rc == 0:
diff --git a/note/0.33.x.md b/note/0.33.x.md
new file mode 100644
index 00000000..022b1a67
--- /dev/null
+++ b/note/0.33.x.md
@@ -0,0 +1,15 @@
+Notable changes in 0.33.x (incomplete)
+======================================
+
+Bugs
+----
+
+* As noted in 0.33.8, `bup fsck` switched from symlinks to hardlinks
+ to accommodate an incompatible change in `par2` 1.0's behavior. To
+ allow the use of filesystems without hardlinks, `bup` now copies the
+ input files if hardlinking fails.
+
+Thanks to (at least)
+====================
+
+...
--
2.47.3

[Rob Browning]

Rob Browning <r...@defaultvalue.org> writes:

> Proposed for 0.33.x.

Decided to go ahead; pushed to 0.33.x.

[Tomas Hnyk]

On Monday, January 19, 2026 at 2:12:13 AM UTC+1 Rob Browning wrote:

Rob Browning <r...@defaultvalue.org> writes:

> Proposed for 0.33.x.

Decided to go ahead; pushed to 0.33.x.

Thank you a lot, cannot wait for a release :-). I am very glad that it is fixed!

Tomas

 

[Rob Browning]

Rearrange client initialization so we can check that it's choosing the
expected index-cache ids. If these were to inadvertently change, the
client would end up duplicating the entire index-cache for a given
host.

Signed-off-by: Rob Browning <r...@defaultvalue.org>
Tested-by: Rob Browning <r...@defaultvalue.org>

---

Proposed for 0.33.x as a backstop, and we'll also transfer the tests
to main and fix the backward incompatibilities that has revealed.

lib/bup/client.py | 35 ++++++++++++++++++++---------------
test/int/test_client.py | 24 ++++++++++++++++++++++++
2 files changed, 44 insertions(+), 15 deletions(-)

diff --git a/lib/bup/client.py b/lib/bup/client.py
index 621c8b6f..18ecdc4c 100644
--- a/lib/bup/client.py
+++ b/lib/bup/client.py
@@ -67,27 +67,32 @@ def parse_remote(remote):
return b'ssh', rs[0], None, rs[1]


+def _remote_parts(reverse, remote):
+ if not reverse:
+ scheme, host, port, path = parse_remote(remote)
+ else:
+ assert not remote, remote
+ scheme, host, port, path = parse_remote(reverse + b':')
+ # The b'None' here matches python2's behavior of b'%s' % None == 'None',
+ # python3 will (as of version 3.7.5) do the same for str ('%s' % None),
+ # but crashes instead when doing b'%s' % None.
+ cache_id = re.sub(br'[^@\w]', b'_',
+ b'%s:%s' % (b'None' if host is None else host,
+ b'None' if path is None else path))
+ return scheme, host, port, path, cache_id
+
+
class Client:
def __init__(self, remote, create=False):
self.closed = False
self._busy = self.conn = None
self.sock = self.p = self.pout = self.pin = None
try:
- is_reverse = environ.get(b'BUP_SERVER_REVERSE')
- if is_reverse:
- assert(not remote)
- remote = b'%s:' % is_reverse
- (self.protocol, self.host, self.port, self.dir) = parse_remote(remote)
- # The b'None' here matches python2's behavior of b'%s' % None == 'None',
- # python3 will (as of version 3.7.5) do the same for str ('%s' % None),
- # but crashes instead when doing b'%s' % None.
- cachehost = b'None' if self.host is None else self.host
- cachedir = b'None' if self.dir is None else self.dir
- self.cachedir = git.repo(b'index-cache/%s'
- % re.sub(br'[^@\w]',
- b'_',
- b'%s:%s' % (cachehost, cachedir)))
- if is_reverse:
+ reverse = environ.get(b'BUP_SERVER_REVERSE')
+ self.protocol, self.host, self.port, self.dir, cache_id = \
+ _remote_parts(reverse, remote)
+ self.cachedir = git.repo(b'index-cache/' + cache_id)
+ if reverse:
self.pout = os.fdopen(3, 'rb')
self.pin = os.fdopen(4, 'wb')
self.conn = Conn(self.pout, self.pin)
diff --git a/test/int/test_client.py b/test/int/test_client.py
index 40580b33..3d7b67b6 100644
--- a/test/int/test_client.py
+++ b/test/int/test_client.py
@@ -157,3 +157,27 @@ def test_remote_parsing():

with pytest.raises(client.ClientError):
client.parse_remote(b'http://asdf.com/bup')
+
+
+def test_remote_cache_ids():
+ def cid(reverse, remote):
+ scheme, host, port, path, cid = client._remote_parts(reverse, remote)
+ return cid
+ with pytest.raises(AssertionError):
+ assert cid(b'x', b'y')
+ with pytest.raises(TypeError):
+ assert cid(None, None)
+ # remotes
+ assert cid(None, b'') == b'None_'
+ assert cid(None, b':') == b'None_'
+ assert cid(None, b'-') == b'None__'
+ assert cid(None, b'h:') == b'h_'
+ assert cid(None, b':p') == b'None_p'
+ # reverses
+ with pytest.raises(TypeError): # currently same as None, None case
+ assert cid(b'', None)
+ assert cid(b':', None) == b'None__'
+ assert cid(b'-', None) == b'None_'
+ assert cid(b'x:', None) == b'x__'
+ assert cid(b':x', None) == b'None_x_'
+ assert cid(b'xy', None) == b'xy_'
--
2.47.3