Feasibility of zfec replacing par2
Mid Magic
n erasure-coded chunks, partly because it's extremely fast, and partly
because it's a generic erasure-coding library. What would be the
feasibility of building a zfec-based backend to replace the par2
redundancy mechanism zoranzaric mentioned in his 28c3 talk?
Mid Magic
suggesting someone else do work that I would like to see represented
in the project. :) I'm just looking for some overall suggestions, or
comments on where to start and/or whether I should even make the
attempt (as in the case where, for example, par2 is too inextricably
linked to the internals of bup to be easily replaced.)
Avery Pennarun
> I prefer using zfec to split blocks and/or files out to multiple k-of-
> n erasure-coded chunks, partly because it's extremely fast, and partly
> because it's a generic erasure-coding library. What would be the
> feasibility of building a zfec-based backend to replace the par2
> redundancy mechanism zoranzaric mentioned in his 28c3 talk?
>
> suggesting someone else do work that I would like to see represented
> in the project. :) I'm just looking for some overall suggestions, or
> comments on where to start and/or whether I should even make the
> attempt (as in the case where, for example, par2 is too inextricably
> linked to the internals of bup to be easily replaced.)
All the par2-related code in bup is entirely contained in
cmd/fsck-cmd.py. You could quite easily write an alternative that
would use zfec instead of / as an alternative to / in addition to
par2.
I've heard in a couple of places that zfec is much faster and just as
reliable as par2. Given that, the only reasons *not* to use zfec are
probably just:
1) It's nonstandard and therefore may have unknown flaws
2) It's less easily available (eg. not in Debian-stable yet)
I'm not actually concerned about (1), as par2 is just as no-name (and
last time I checked, Debian has a special par2 patch that makes it not
crash - not so inspiring).
I worry more about #2, but since this is an optional part of bup
anyway, it's not really critical. The most important part is that we
give people easy instructions for how to get it, and make sure bup
works without it (as it currently works without par2). Sounds like
python-zfec is in Debian testing/unstable now, so that's a good sign.
For simplicity, I think i would fall somewhat on the side of replacing
par2 with zfec in bup, rather than continuing to support both.
Have fun,
Avery
Zandr Milewski
> On Fri, Dec 30, 2011 at 6:35 PM, Mid Magic <midm...@gmail.com> wrote:
>> I prefer using zfec to split blocks and/or files out to multiple k-of-
>> n erasure-coded chunks, partly because it's extremely fast, and partly
>> because it's a generic erasure-coding library.
Small world. I was the ops guy at Allmydata, which is where Tahoe (and
thus zfec) was written.
> I've heard in a couple of places that zfec is much faster and just as
> reliable as par2.
The nature of Tahoe means there's a LOT of integrity checking going on.
I don't recall seeing any integrity failures that were attributable to
problems in zfec, given sufficient shares to satisfy k.
> 1) It's nonstandard and therefore may have unknown flaws
It's a young implementation, but there's no new science here. It's
Reed-Solomon under the hood. I worked with some of the new science
('rateless' or 'fountain' codes) at a previous gig... that stuff was
non-standard for sure. :)
> 2) It's less easily available (eg. not in Debian-stable yet)
True, but it's been in Ubuntu universe since Lucid.
> Sounds like
> python-zfec is in Debian testing/unstable now, so that's a good sign.
Ah, didn't know that. That's encouraging.
I can reach out to the Tahoe devs if we need help... let me know.
-Zandr
Gabriel Filion
> I worry more about #2, but since this is an optional part of bup
> anyway, it's not really critical. The most important part is that we
> give people easy instructions for how to get it, and make sure bup
> works without it (as it currently works without par2). Sounds like
> python-zfec is in Debian testing/unstable now, so that's a good sign.
about this point, if the package is in sid/testing, maybe we could
convince the package maintainer to try and get it to squeeze-backports.
it wouldn't be so difficult to reach from stable then.
--
Gabriel Filion
Avery Pennarun
Works for me :)
Avery
Avery Pennarun
> Thanks for the discussion, folks. I really think this kind of thing
> (combined with a fuse interface) is what a lot of people like myself
> are just quietly searching for, and currently getting along with
> stopgap solutions. It was a good talk at 28c3, I enjoyed it.
You know the fuse interface already exists, right? :)
Avery
Gabriel Filion
hmpf, well it's a bit longer than I expected, I wanted to say "hey, look
at that URL, the package is in!", but it's not there yet... but just you
wait! :P
I looked up who were the package maintainers of python-zfec in Debian in
order to contact them and ask if they could push to squeeze-backports,
and I happened to know one of them ;) So I asked the maintainer I knew
about pushing the package and he let out a "hah! easy!" and the package
was submitted 10mins later, heh.
The package is currently awaiting approval from squeeze-backports and
should make it in soonish.
--
Gabriel Filion