Deprecate and remove/relocate explain_pickle module?

[Erik Bray]

There is a cool module in sage called sage.misc.explain_pickle, which
is useful for helping to understand and debug how non-trivial objects
in Sage are pickled and unpickled.

However, it's a big, fairly complicated module which carries quite a
bit of technical debt with it, and relatively little of it is specific
to Sage (the few bits that are could be added on as a wrapper). It
could in theory be useful outside of Sage as well. Furthermore, it's
not actually *used* by anything in Sage; it's a debugging tool. And
probably one that's not used as much anymore since (I believe) a lot
of the deepest pickle-related issues in Sage have been worked out? I
could be wrong about that though--maybe it's used more than I know.

I ask about it because in the Python 3 of Sage it's even of less
value. Although it could be made to work on Python 3, it only
supports the "version 2" pickle protocol from Python 2. Since Python
3.4 we're now up to protocol version 4, and there's a protocol version
5 in the works [1].

For this reason I think it should be deprecated from Sage (or at least
importing it from sage.misc). If someone wants to update it to
support new pickle protocol versions (I don't), it would be worth
putting in a new repository where people can still contribute to it.
If not, I think it should just be sunsetted and sent off with the
praise it deserves.

E


[1] https://www.python.org/dev/peps/pep-0574/

[Jeroen Demeyer]

It must be said that I found this tool quite valuable. When unpickling
goes wrong, it is typically not easy to find out why and explain_pickle
does help with that.

That being said, if nobody wants to maintain it, it has to go (with
praise, as you said)...

[Erik Bray]

For sure; I wanted to be clear that I don't think it isn't valuable.
But I don't necessarily think it belongs in sagelib itself, and can
mostly be extracted out as a separate package with its own repository.
That way, at least it won't be entirely lost to the sands of time, and
someone who has a need for it and is sufficiently motivated to update
it for newer Python versions can pick it up again. We could even
still keep sage.misc.explain_pickle to provide the Sage-specific bits
as a wrapper around a more general library. That is work that I am
willing to do (I'm just not motivated enough to update it beyond that,
at least at the moment).

[Nils Bruin]

On Friday, September 21, 2018 at 1:46:47 AM UTC-7, Erik Bray wrote:

There is a cool module in sage called sage.misc.explain_pickle, which
is useful for helping to understand and debug how non-trivial objects
in Sage are pickled and unpickled.

I am adding another testimonial  on the usefulness of the routine. On several occasions I was in a pickle (sorry for the pun), which could be solved immediately by explain_pickle. it was a big time-saver that it was available *right there*, because I wasn't looking forward to having to hunt down a third party package that may or may not help me. I also think that with multiprocessing, serialization remains very important to support for (hopefully) all data structures in sage. That means that whenever a new structure pops up, errors and debugging will be inevitable. The problems in serialization are often triggered by non-local interactions, so I would not expect that problems with serialization in sage will become rarer (in fact, I expect them to become more complex due to more intricate interactions).

I would be a little sad to see explain_pickle go.

[Erik Bray]

Per my reply to Jeroen, what if it were still "there" in
sage.misc.explain_pickle, but the meat of it were moved out to a
separate module (which could still be included in
sage-the-distribution, as at least an optional package if not a
standard package)?

My reasoning here is not so much that it isn't useful (which I think
it clearly is), but rather that it's a pretty general tool that
deserves to live outside Sage. And it's also in need of maintenance,
which eventually will make it a blocker for Sage, which is not so good
given that it can clearly exist outside of Sage itself.

[Matthias Koeppe]

+1 on splitting it out as a standalone Python package. With a bit of luck, a new set of contributors for it can be found that way. 

[Dima Pasechnik]

+1 from me too.

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.

[Erik Bray]

Ok, thanks for the feedback. This isn't a high priority right now,
but it was just one my mind at the moment so I wanted to put the idea
out there. I'll plan to split it out to a stand-alone package when I
get there (probably it will be one of the last things to do before we
can claim full Python 3 support).