Pickling classes (not class instances)

[Nicolas M. Thiery]

Dear Sage developers,

Here is a question I posted on comp.lang.python on January 10th
without getting any answer yet. It's about class pickling which is the
main show stopper for the new category framework.

Comments, or suggestions of persons to contact are very very welcome!

Best regards,
Nicolas

------------------------------------------------------------------------------

Purpose of this e-mail:
-----------------------

How to customize how a class (not an instance of a class!!!) is pickled?

Example:
==============================================================================
class metaclass(type):
def __new__(mcs, name, bases, dict):
print " running metaclass.__new__"
return type.__new__(mcs, name, bases, dict)
#
def __reduce_class__(self):
print "reducing a class"
# do the real work

c = metaclass("foo", (object,), dict())

import copy_reg
copy_reg.pickle(metaclass, metaclass.__reduce_class__)

pickle.dumps(c)
---------------------------------------------------------------------------
PicklingError Traceback (most recent call
last)
...
PicklingError: Can't pickle <class '__main__.foo'>: it's not found as
__main__.foo
==============================================================================

Context:
--------

I am working on the Sage project (www.sagemath.org), and more
precisely on the category infrastructure. The code is building lots of
classes on the fly by composing preexisting classes by inheritance
(for the curious, see the doc of the class Category in
http://sage.math.washington.edu:2144/file/1567cea09170/categories-nt....).

Since those classes are built on the fly, they cannot be pickled with
the default mechanism of name lookup. A proper pickling would be to
rebuild the class anew. Nothing problematic, except for the question
above.

Discussion:
-----------

It sounds like copy_reg would be the natural way to go (as in the
example above). However, its documentation suggests that it explicitly
is not intended for pickling classes, e.g. first paragraph of:

http://docs.python.org/library/copy_reg.html#module-copy_reg

is:

The copy_reg module provides support for the pickle and cPickle
modules. The copy module is likely to use this in the future as
well. It provides configuration information about object
constructors which are not classes. Such constructors may be factory
functions or class instances.

And indeed, looking up at the code of pickle (l. 289-299 of pickle.py)
(and similarly in cpickle), the copy-reg dispatch is explicit bypassed
for metaclasses:

# Check for a class with a custom metaclass; treat as regular
class
try:
issc = issubclass(t, TypeType)
except TypeError: # t is not a class (old Boost; see SF
#502085)
issc = 0
if issc:
self.save_global(obj)
return

# Check copy_reg.dispatch_table
reduce = dispatch_table.get(t)

Which causes the failure above.

Is there a specific reason for this restriction?

Would it be thinkable to move up the copy reg dispatch before the
metaclass treatment in pickle and cPickle? I did it locally, and it
fixed my problem.

If not, what would be the right way to achieve this?

Many thanks in advance!

Best regards,
Nicolas
--
Nicolas M. Thiéry "Isil" <nth...@users.sf.net>
http://Nicolas.Thiery.name/

[Carl Witty]

On Thu, Feb 12, 2009 at 2:42 PM, Nicolas M. Thiery
<Nicolas...@u-psud.fr> wrote:
> Is there a specific reason for this restriction?

No idea... :)

> Would it be thinkable to move up the copy reg dispatch before the
> metaclass treatment in pickle and cPickle? I did it locally, and it
> fixed my problem.

We probably wouldn't want to patch this directly in our Python,
because that would make it a lot harder for Tim Abbott to package Sage
for Debian, etc.; but if nobody else can come up with a better
solution, we could fork cPickle. I mean, we could copy cPickle.c into
the Sage library, rename it as sagePickle.c, and make whatever changes
we wanted to that copy.

Carl

[Nicolas M. Thiery]

Dear Carl,

I just posted my patch doing this on trac: #5985

Reviews welcome!

Best,