fsck is broken by par2cmdline v1.0.0

[Nathaniel]

The recently released par2cmdline version 1.0.0 breaks bup fsck -g  because par2 create  now intentionally does not process symlinks. This is discussed in their Github:

https://github.com/Parchive/par2cmdline/issues/164

Bup has been using temporary symlinks for par2 redundancy generation, to avoid the issue where par2 creates empty redundancy files if it gets canceled, for about a year:

https://groups.google.com/g/bup-list/c/XIL8Axz3qXQ/m/5ZNatZ5yAgAJ

This means that bup fsck generation no longer works with par2cmdline version 1. This means bup  on Archlinux, for instance, is broken today, as Arch provides par2cmdline v1.0.0 in its repos. The question is, what is the proper way to resolve the issue? We could update bup to no longer rely on symlinks... we could try to upstream a patch to par2cmdline that adds a flag that allows symlinks...

Any other ideas? Thoughts?

--
Nathaniel Hourt

[Nathaniel]

Upon further reflection, I realize another option is to drop the dependency on par2cmdline.

This may sound overcorrective on its face, but I'm not certain that it is. Looking at the linked par2cmdline issue, it seems clear that the project is in trouble. The lead contributor and, it seems, maintainer of the repo said directly in that issue's conversation that he's long since lost interest in the project, and the decision to unconditionally disregard symlinks is, frankly, absurd and inappropriately cursory.

It seems to me that, from Bup's perspective, the "correct" fix is to upstream a flag to par2cmdline, since par2's new behavior makes no sense... but that sounds like a difficult sell if the maintainer isn't interested... and even if the patch were accepted, it also feels like kicking the can down the road. If it happened once, it's likely to happen again, so that makes it a questionable choice for a dependency.

So I'll propose another option, which is to drop the par2cmdline dependency in favor of a library RS-code implementation (a library will hopefully have a more stable API). There are several RS-code libraries for Python available, with one example being reedsolo, which is packaged for many popular distros, and seems to see a suitable commit frequency.

I don't personally have any skin in this game except that I'm thinking bup is a great idea and I'd like to rely on it, so I'd like it to be reliable. I'm not sure par2cmdline will facilitate that — I suspect it's going to become increasingly fragile without a motivated maintainer.

--
Nathaniel Hourt

[Johannes Berg]

On Mon, 2025-06-30 at 00:45 -0500, 'Nathaniel' via bup-list wrote:
> Upon further reflection, I realize another option is to drop the dependency on par2cmdline.
>

[snip]

None of your reflections are really wrong in any way, but for now bup is
creating hardlinks instead of symlinks already:

https://github.com/bup/bup/commit/daa6bf3eeabf1f50630b17b0cbbbdcbbef59c90e



Rob has also been .. I want to say "tempted" (?) .. to write some
recovery code directly (e.g. using isa-l [1]), but it's not that simple
really, and there probably are a lot of more important things to do
(such as a long-planned index overhaul :-) .)

[1] https://github.com/intel/isa-l

johannes

[Nathaniel Hourt]

Well, I forgot to check master before writing an email. That was silly!

Nevertheless, it sounds like a better solution is desired and moving to
a different RS-coding backend is already under consideration. I could
take a look at it while those more familiar with bup focus on higher
priorities. It seems like it should be mostly self-contained; a
reasonable beginner issue.

Is there anything I should know going in? Also, how desirable is a
solution based on ISA-L? I can play around with benchmarks, but if it's
not dramatically faster than a native Python solution (ISA-L's Python
bindings do not include the RS-coding), then a Python solution is likely
to be simpler and more maintainable. I can play around with some
benchmarks to see.

—
Nathaniel

[Johannes Berg]

On Mon, 2025-06-30 at 11:16 -0500, Nathaniel Hourt wrote:
>
> Well, I forgot to check master before writing an email. That was silly!

No worries.

> Nevertheless, it sounds like a better solution is desired and moving to
> a different RS-coding backend is already under consideration. I could
> take a look at it while those more familiar with bup focus on higher
> priorities. It seems like it should be mostly self-contained; a
> reasonable beginner issue.

Indeed it shouldn't be all that hard from an integration perspective.

> Is there anything I should know going in? Also, how desirable is a
> solution based on ISA-L?

I suspect ISA-L was more "it's in Debian (old?)stable and looks fast",
but not really otherwise particularly special.

> I can play around with benchmarks, but if it's
> not dramatically faster than a native Python solution (ISA-L's Python
> bindings do not include the RS-coding), then a Python solution is likely
> to be simpler and more maintainable. I can play around with some
> benchmarks to see.

True. Chances are a _pure_ python solution is much slower, but could
still be worth trying. Also 'reedsolo' includes cython and C versions,
so it shouldn't be _particularly_ slower I guess. Not even in Debian
stable though, but I guess in this case arguably there you - still -
have even the older version of par2cmdline, so you just couldn't migrate
now to any newer solution (unless you manually install it.)

johannes

[Rob Browning]

Johannes Berg <joha...@sipsolutions.net> writes:

> I suspect ISA-L was more "it's in Debian (old?)stable and looks fast",
> but not really otherwise particularly special.

It (offhand) seemed plausible[1], but would of course require work, and
then we're responsible for it. If we did end up doing this, I'd
definitely want to include very solid (perhaps generative, randomized)
testing.

[1] ...maybe well maintained, possibly fast for common architectures, and
importantly, has effectively no additional dependencies. Not that
additional dependencies are forbidden, but I much prefer to avoid
them when feasible.

I got as far as starting to understand the library, some of the
underlying concepts, in part via
https://web.eecs.utk.edu/~jplank/plank/papers/CS-96-332.pdf, and vaguely
contemplating a data format, but that's it, and I'm still not sure I
think it'd be a good idea for us to make this our problem, i.e. I think
the bar for doing this ourselves should be moderately high, and probably
more than one of us (likely me included) would need to become fairly
competent in the domain -- I'm not yet, and not certain I want to sign
up for doing so right now.

I'd prefer to just switch to some well maintained tool (or higher level
library), but as you suggest, I'd likely also want it to be "readily
available", e.g. in Debian stable at least, etc.

So -- still not sure, but par2cmdline (with the hardlink fix) does still
work, and is fairly easy to find, so as you say, our time might be
better spent elsewhere for now.

--
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

[Renaud Pacalet]

I just encountered the same issue. As far as I understand the fix is not yet available in a stable release. Shall I switch to main (and take the risk of using an unstable version) or wait until the next stable release (and take the risk of some missing par2 files)?

[Rob Browning]

Renaud Pacalet <ren...@pacalet.fr> writes:

> I just encountered the same issue. As far as I understand the fix is not
> yet available in a stable release. Shall I switch to main (and take the
> risk of using an unstable version) or wait until the next stable release
> (and take the risk of some missing par2 files)?

I'd stick with 0.33.x for now. That branch also has the fix, and 0.33.8
should be released soon. I'd generally avoid main for anything that
matters, and/or that won't always be available by other means.

[Rob Browning]

Renaud Pacalet <ren...@pacalet.fr> writes:

> Thanks Rob. I updated from 0.33.7 to 0.33.9 and it fixed this issue.

Glad to hear it.