Any update on Python 3?

[Tadej Janež]

Hi,

I'm the maintainer of bup package in Fedora.

Today, I had to remove bup-web functionality from Fedora 31+ [1] since
starting with Fedora 31, Tornado doesn't support Python 2 anymore.
Hence, bup-web which requires it, can't be packaged anymore.

Python 2 will receive no upstream support from Jan 1, 2020 onwards [2].
Fedora plans to retire the python2 package in Fedora 32 which should
come out around May 2020 [3].

According to the Fedora Python 2 Dropping Database, bup's dependencies
already support Python 3 [4], except for fuse-python (python-fuse)
which needs to be updated to the latest upstream which has support for
Python 3 [5].

Being a package maintainer, I'm curious if you have any plans/timeline
on implementing support for Python 3?

Best regards,
Tadej


[1]
https://src.fedoraproject.org/rpms/bup/c/b14777d7e8f5ff196e43afbb0ab50e4c2c7a4749?branch=master
[2] https://www.python.org/dev/peps/pep-0373/#id4
[3] https://fedoraproject.org/wiki/Changes/RetirePython2
[4] https://fedora.portingdb.xyz/pkg/bup/
[5] https://github.com/libfuse/python-fuse/releases

[Julien Goodwin]

On 10/6/19 7:04 pm, Tadej Janež wrote:
> According to the Fedora Python 2 Dropping Database, bup's dependencies
> already support Python 3 [4], except for fuse-python (python-fuse)
> which needs to be updated to the latest upstream which has support for
> Python 3 [5].

The dependencies might, but the internal SWIG doesn't. I did a bunch of
fixes last year reducing most of the incompatabilities other than that
(although I think one got dropped somewhere, possibly I never mailed it).

I did have a look at it, but ultimately stumbled on an incompatibility
in the internal python mmap library not having a specific interface
(can't remember specifically what). Others had requested this from
python core but had no luck.

Rob was going to look it this, he may have made further progress.

[Rob Browning]

Tadej Janež <tad...@nez.si> writes:

> Being a package maintainer, I'm curious if you have any plans/timeline
> on implementing support for Python 3?

So... I'm sure we'll do "something", but I'm not sure what that
something will be yet -- and whether or not all of the dependencies
support Python 3 is only part of the problem.

The bigger problem, assuming I don't misunderstand the current state, is
Python's string-related decisions, with the fundamental one being the
initial insistence that paths/users/groups/acls/etc. are unicode.
They're not.

And while we've made some progress with respect to Python 3
compatibility, and I think Python itself may be, by necessity, backing
away from its initial "everything is unicode" position, two of what I
suspect are the biggest issues remain:

- making sure that either all the relevant Python 3 apis can return
binary strings for users/groups/paths/acls/etc., or that we've
written our own C wrappers that do.

- finding and adjusting all of the relevant bup code to make sure it
never uses the new unicode default strings, or that it always uses
the more recent "byte smuggling" support in a safe way (with the
attendant inefficiency). (This may be ugly.)

In addition, we'll need to come up with a clear idea of how we want to
handle formatting paths output diagnostically -- logging, progress
messages, etc. (e.g. imagine we need to print a path that's a set of
bytes that's unrepresentable in the current encoding).

Also worth noting that without care (and solid test (data) coverage), I
suspect we could end up with a Python 3 bup that appears to work, but
doesn't actually save/restore data correctly all the time.

In any case, I've been poking at this off and on, and if I haven't
missed something, the amount of time (the extent of which is still
unknown) that the Python upstream's decisions have effectively taken
from us is disheartening.

And while I'm not seriously proposing anything, the situation is
frustrating enough that over the past while, I've toyed with a handful
of alternative bup implementations just to see what they might look like
(and how they might perform).

Anyway, I suspect we'll end up with a Python 3 bup, and if I end up
being the one to implement most of it, I'd love to have it more or less
working by the end of the year, but I haven't really focused on it yet.

Hope this helps
--
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

[Tadej Janež]

Hi Rob,

thanks for a thorough and sincere reply.

On Tue, 2019-07-30 at 01:08 -0500, Rob Browning wrote:
>
> The bigger problem, assuming I don't misunderstand the current state,
> is
> Python's string-related decisions, with the fundamental one being the
> initial insistence that paths/users/groups/acls/etc. are unicode.
> They're not.

Huh, I don't envy you all this bytes/unicode mess.

> And while we've made some progress with respect to Python 3
> compatibility, and I think Python itself may be, by necessity,
> backing
> away from its initial "everything is unicode" position, two of what I
> suspect are the biggest issues remain:
>
> - making sure that either all the relevant Python 3 apis can return
> binary strings for users/groups/paths/acls/etc., or that we've
> written our own C wrappers that do.

I guess this would be the nicest solution (upstream API support for
binary strings), but is the one you have the least influence on.

> Also worth noting that without care (and solid test (data) coverage),
> I
> suspect we could end up with a Python 3 bup that appears to work, but
> doesn't actually save/restore data correctly all the time.

Yes, this will be tricky since backup software is special in this
regard.

> In any case, I've been poking at this off and on, and if I haven't
> missed something, the amount of time (the extent of which is still
> unknown) that the Python upstream's decisions have effectively taken
> from us is disheartening.

I can imagine.

> And while I'm not seriously proposing anything, the situation is
> frustrating enough that over the past while, I've toyed with a
> handful
> of alternative bup implementations just to see what they might look
> like
> (and how they might perform).

I'm curious about these alternative bup implementations. In which
language were they written? Did you make any changes to the
architecture?

> Anyway, I suspect we'll end up with a Python 3 bup, and if I end up
> being the one to implement most of it, I'd love to have it more or
> less
> working by the end of the year, but I haven't really focused on it
> yet.

Sure, I completely understand.

Thanks for taking the time and answering my email!

Tadej

[Rob Browning]

Tadej Janež <tad...@nez.si> writes:

> I'm curious about these alternative bup implementations. In which
> language were they written? Did you make any changes to the
> architecture?

Well, I wouldn't call any of it anything approaching an
"implementation", but among the things I've toyed with (in part I used
this as an excuse to investigate some things I was interested in
anyway):

- a rust version of various bits of bup; among other things, a
primitive join, and some expirimentation into creating "normal"
shared libs using rust, i.e. a libfoo.so that just happens to be
written in rust, etc.

- migrating more of bup to C (which we may have to do anyway for py3,
i.e. if nothing else, we'll likely need more byte-array returning
syscall wrappers)

- a clojure/jvm version of various bits of bup, using a "quit on idle"
server to avoid clojure/jvm's longer startup times. Command line
invocations just connect to it via an fs socket, and hand their
stdin/stdout/stderr to the server over the socket (only recently
learned that trick) so that the server can masquerade as the client
with "no" IO overhead for those streams. The client just waits for
the server to say it's finishe and exits with the appropriate
status, and the client can be written in whatever you like.

This implementation has the advantage of *solid* concurrency
support, etc., but of course all the jvm disadvantages (varying
depending on the audience/situation).

The server approach also has the potential advantages of making it
easier to coordinate parallel access to the repository without
lockfiles or staleness/liveness questions, and making it easier to
unify (share) VFS caching, etc. But it's not a panacea.

- a guile version of various bits of bup -- similar advantages to the
clojure/jvm version, but unlike clojure/jvm (and many other
languages) it's a very posix, thread, C, friendly platform, and I
suspect it's fairly widely deployed (due to among other things, make
and gdb support). Though we'd need to make sure to exclude all the
mmapped regions from the conservative GC's purview (for which
there's an api). (On the tangential benefit front, this bit of
poking around resulted in the discovery and eventual upstream
resolution of a binary IO issue that improved guile's throughput by
orders of magnitude.)

> Thanks for taking the time and answering my email!

Certainly -- just took me a while.