Persist a class (not an instance)

[Kent Johnson]

Is there a way to persist a class definition (not a class instance, the actual class) so it can be restored later? A naive approach using pickle doesn't work:

>>> import pickle
>>> class Foo(object):
... def show(self):
... print "I'm a Foo"
...
>>> p = pickle.dumps(Foo)
>>> p
'c__main__\nFoo\np0\n.'

Hmm, doesn't look too promising. In a new interpreter:

>>> import pickle
>>> p='c__main__\nFoo\np0\n.'
>>> Foo = pickle.loads(p)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "C:\Python24\lib\pickle.py", line 1394, in loads
return Unpickler(file).load()
File "C:\Python24\lib\pickle.py", line 872, in load
dispatch[key](self)
File "C:\Python24\lib\pickle.py", line 1104, in load_global
klass = self.find_class(module, name)
File "C:\Python24\lib\pickle.py", line 1140, in find_class
klass = getattr(mod, name)
AttributeError: 'module' object has no attribute 'Foo'

The idea is to persist classes that are created and modified at runtime.

Thanks,
Kent

[Sybren Stuvel]

Kent Johnson enlightened us with:

> Is there a way to persist a class definition (not a class instance,
> the actual class) so it can be restored later?

From the docs:

"Similarly, classes are pickled by named reference, so the same
restrictions in the unpickling environment apply. Note that none of
the class's code or data is pickled [...]"

Sybren
--
The problem with the world is stupidity. Not saying there should be a
capital punishment for stupidity, but why don't we just take the
safety labels off of everything and let the problem solve itself?
Frank Zappa

[Kent Johnson]

Sybren Stuvel wrote:
> Kent Johnson enlightened us with:
>
>>Is there a way to persist a class definition (not a class instance,
>>the actual class) so it can be restored later?
>
>
> From the docs:
>
> "Similarly, classes are pickled by named reference, so the same
> restrictions in the unpickling environment apply. Note that none of
> the class's code or data is pickled [...]"

OK that confirms that pickle won't work. Is there another approach that will?

Kent

[Sybren Stuvel]

Kent Johnson enlightened us with:

> OK that confirms that pickle won't work. Is there another approach
> that will?

Well, since the classes are created at runtime as well, you could
compile them using the appropriate API and call exec() on them.

[David Wahler]

Kent Johnson wrote:
> Is there a way to persist a class definition (not a class instance,
> the actual class) so it can be restored later? A naive approach
> using pickle doesn't work:

[snip]

> The idea is to persist classes that are created and modified at runtime.

I couldn't resist the challenge, so I decided to take a crack at it. My
code is below. (I hope it's OK to post it even though it's a bit on the
long side.) So far, it seems to work OK; the biggest caveat to be aware
of is that functions' global context is not preserved.

My approach was to use pickle's __reduce__ protocol to store functions
and classes. Of course, you can't modify the built-in function and
classobj types, so I subclassed Pickler to override them. The advantage
of this is that you don't need an extension to the pickling data
format, and you can use the standard unpickler. (The custom module
still needs to have been imported, as it adds the classobj type to
__builtins__.)

Unfortunately, I'm not sure how to go about making it work for
new-style classes. It would seem to involve messing with dictproxy and
descriptor objects, and that's getting me into more unfamiliar
territory.

I'm sure there's a better way to do this; this seemed like "the
simplest thing that could possibly work".

-- David

#####################################################
# code_pickle.py

import sys, copy_reg, pickle, new, marshal, types, StringIO

# Needed to unserialize old-style classes
sys.modules['__builtin__'].classobj = new.classobj

# from http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/439096
def get_cell_value(cell):
return type(lambda: 0)(
(lambda x: lambda: x)(0).func_code, {}, None, None, (cell,)
)()

def func_constructor(name, code, defaults, closure):
return new.function(marshal.loads(code), globals(), name,
defaults, closure)

class CodePickler(pickle.Pickler):
def __init__(self, *args, **kwargs):
pickle.Pickler.__init__(self, *args, **kwargs)
self.dispatch = self.dispatch.copy()
self.dispatch[types.ClassType] = CodePickler.do_class
self.dispatch[types.FunctionType] = CodePickler.do_function

def save(self, ob, *args, **kwargs):
print ob
pickle.Pickler.save(self, ob, *args, **kwargs)

def do_class(self, ob):
if ob in (types.__dict__.values()):
self.save_global(ob)
else:
args = (ob.__name__, ob.__bases__, ob.__dict__)
self.save_reduce(type(ob), args)

def do_function(self, ob):
if ob == func_constructor:
self.save_global(ob)
else:
if ob.func_closure:
closure = tuple(map(get_cell_value, ob.func_closure))
else:
closure = None
args = (ob.func_name, marshal.dumps(ob.func_code),
ob.func_defaults, closure)
self.save_reduce(func_constructor, args)

def dumps(ob):
s = StringIO.StringIO()
CodePickler(s).dump(ob)
return s.getvalue()

# Example:
#
# import code_pickle
# class Foo:
# def show(self):
# print "Foo!"
#
# s = code_pickle.dumps(Foo)
# --------------------------------------
# import code_pickle, pickle
# Foo = pickle.loads(s)
# Foo().show
#####################################################

[Phillip J. Eby]

David Wahler wrote:
> Kent Johnson wrote:
> > Is there a way to persist a class definition (not a class instance,
> > the actual class) so it can be restored later? A naive approach
> > using pickle doesn't work:
> [snip]
> > The idea is to persist classes that are created and modified at runtime.
>
> I couldn't resist the challenge, so I decided to take a crack at it. My
> code is below. (I hope it's OK to post it even though it's a bit on the
> long side.) So far, it seems to work OK; the biggest caveat to be aware
> of is that functions' global context is not preserved.
>
> My approach was to use pickle's __reduce__ protocol to store functions
> and classes. Of course, you can't modify the built-in function and
> classobj types, so I subclassed Pickler to override them. The advantage
> of this is that you don't need an extension to the pickling data
> format, and you can use the standard unpickler. (The custom module
> still needs to have been imported, as it adds the classobj type to
> __builtins__.)
>
> Unfortunately, I'm not sure how to go about making it work for
> new-style classes. It would seem to involve messing with dictproxy and
> descriptor objects, and that's getting me into more unfamiliar
> territory.
>
> I'm sure there's a better way to do this; this seemed like "the
> simplest thing that could possibly work".

This is actually pretty sweet. It seems to me that you'll be fine with
new-style classes if you just save dict(ob.__dict__) instead of trying
to save __dict__ directly, as that'll get rid of the dictproxy part.
There's no generic way to save descriptors, as far as I know, but you
can always register reducers for specific types, like property, and
user-defined descriptor classes are likely to be picklable anyway.

As for functions' global context, you could look to see if there's a
__name__ present, in which case you can save a reference to that
module's __dict__. Otherwise, simply pickle the func_globals as-is.
Some folks might just want to do that anyway, if the code isn't
actually being loaded from a module.

Of course, the classic caveat regarding pickles and security applies to
all this. That is, pickles and security don't mix. If you want one,
you can't really get the other. ;-)