Object serialization: transfer from a to b (non-implemented code on b)

[Gabriel Rossetti]

Hello everyone,

I am trying to serialize a function, class, etc and transfer it, have it
unserialized and used. The thing is that this code is not defined on the
receiving side and thus it does not work. I tried the following tests:

Terminal A:

>>> import pickle
>>> def test(): pass
...
>>> pickle.dumps(test)
'c__main__\ntest\np0\n.'
>>>

Terminal B:

>>> import pickle
>>> pickle.loads('c__main__\ntest\np0\n.')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.5/pickle.py", line 1374, in loads
return Unpickler(file).load()
File "/usr/lib/python2.5/pickle.py", line 858, in load
dispatch[key](self)
File "/usr/lib/python2.5/pickle.py", line 1090, in load_global
klass = self.find_class(module, name)
File "/usr/lib/python2.5/pickle.py", line 1126, in find_class
klass = getattr(mod, name)
AttributeError: 'module' object has no attribute 'test'

Ok, so pickle needs the code to be defined on both sides, so I tried
marshal:

Terminal A:

>>> import marshal
>>> marshal.dumps(test)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: unmarshallable object

Ok, as the docs say, marshal is quite limited, no functions or user
classes can be marshalled, I did get it to work like this though :

>>> import inspect
>>> marshal.dumps(inspect.getmembers(test, inspect.iscode)[0][1])
'c\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00C\x00\x00\x00s\x04\x00\x00\x00d\x00\x00S(\x01\x00\x00\x00N(\x00\x00\x00\x00(\x00\x00\x00\x00(\x00\x00\x00\x00(\x00\x00\x00\x00s\x07\x00\x00\x00<stdin>t\x04\x00\x00\x00test\x01\x00\x00\x00s\x00\x00\x00\x00'

ok, but can I unmarshal it?

Terminal B:

>>>
marshal.loads('c\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00C\x00\x00\x00s\x04\x00\x00\x00d\x00\x00S(\x01\x00\x00\x00N(\x00\x00\x00\x00(\x00\x00\x00\x00(\x00\x00\x00\x00(\x00\x00\x00\x00s\x07\x00\x00\x00<stdin>t\x04\x00\x00\x00test\x01\x00\x00\x00s\x00\x00\x00\x00')
<code object test at 0xb7591578, file "<stdin>", line 1>

ok, it seams to work...

>>>
marshal.loads('c\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00C\x00\x00\x00s\x04\x00\x00\x00d\x00\x00S(\x01\x00\x00\x00N(\x00\x00\x00\x00(\x00\x00\x00\x00(\x00\x00\x00\x00(\x00\x00\x00\x00s\x07\x00\x00\x00<stdin>t\x04\x00\x00\x00test\x01\x00\x00\x00s\x00\x00\x00\x00')()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'code' object is not callable

ok, logical, it's a code object...

>>>
eval(marshal.loads('c\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00C\x00\x00\x00s\x04\x00\x00\x00d\x00\x00S(\x01\x00\x00\x00N(\x00\x00\x00\x00(\x00\x00\x00\x00(\x00\x00\x00\x00(\x00\x00\x00\x00s\x07\x00\x00\x00<stdin>t\x04\x00\x00\x00test\x01\x00\x00\x00s\x00\x00\x00\x00'))

ok, this works, not super pretty, but it works

now user objects?

>>> class A(object): pass
...
>>> a = A()
>>> marshal.dumps(a)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: unmarshallable object

ok, expected...

>>> inspect.getmembers(a, inspect.iscode)
[]

ok, not cool, can't use the previous method...

Ok, now a word about what I'm trying to do, I'm trying to run python
code in a new process, like multiprocessing does, but without
multiprocessing. I'm trying to serialise the code because I 'd like to
do something like:

pseudo-code, serialize/unserialize/process are made up:

def newProcess(func, *args, **kwargs):
func = serialize(func)
args = serialize(args)
kwargs = serialize(kwargs)
process("python -c 'import serialize, unserialize;print
serialize(unserialize(\'%s\')(*unserialize(\'%s\'),
**unserialize(\'%s\')))'" % (func, args, kwargs))
#read result from stdout

I'm greatly simplifying this because I'm using a framework, but that's
the basic idea.

thanks,
Gabriel

[Paul Rubin]

Gabriel Rossetti <gabriel....@arimaz.com> writes:
> I am trying to serialize a function, class, etc and transfer it

You mean the actual code? You have to use marshal rather than pickle,
the Python versions have to be the same at both ends, and you better
have some kind of authenticated transport to stop malicious code from
getting accepted and run by the receiving end.

[Gabriel Rossetti]

Yes, but I wasn't able to marshal/unmarshal them correctly as shown in
my examples

[Andreas Löscher]

The marshal module can be used to dump the Code of an Function (or
anything else), like you have done. If you have the Code, you can
reconstruct the function.

>>> import types
>>> import marshal
>>> def a(): pass
...
>>> s=marshal.dumps(a.__code__)
>>> f=types.FunctionType(marshal.loads(s), {})
>>> f
<function a at 0x7f6308a66de8>

The second parameter is the global scoop of the function.

If you want to marshal a class, things get a little more complicated,
because these consist of many code objects and you also need the code
to assemble them together.

There are two easy ways I can think of right now:

1.
Send the string of the code you want to transfer:
>>> s="""class A: pass"""

and then:
>>> exec(s) in globals()

2.
Encapsule the stuff you want to send in a function.

Best,
Andreas

[Gabriel Rossetti]

Thanks Andreas, I'll give that a try. I didn't really want to do the
string method which is why I was trying to serialize.

best,
Gabriel

[Andreas Löscher]

Am Mittwoch, den 14.04.2010, 12:37 +0200 schrieb Gabriel Rossetti:

You can compile the string befor you send it in an code object. This way
you also don't need to compile it twice if you execute ist two times.

>>> s="""class A: pass"""
>>> co=compile(s, "foo.py", "exec")

>>> exec(marshal.loads(marshal.dumps(co))) in globals()

But be aware, that the string method may be the most save method for
doing this. Python is not ByteCode compatible between it's versions.
But if I understand you right, this is not an issue for you.

Best,
Andreas

[Gabriel Rossetti]

Andreas Löscher wrote:
> Am Mittwoch, den 14.04.2010, 11:33 +0200 schrieb Gabriel Rossetti:
>
>> Paul Rubin wrote:
>>
>>> Gabriel Rossetti <gabriel....@arimaz.com> writes:
>>>
>>>
>>>> I am trying to serialize a function, class, etc and transfer it
>>>>
>>>>
>>> You mean the actual code? You have to use marshal rather than pickle,
>>> the Python versions have to be the same at both ends, and you better
>>> have some kind of authenticated transport to stop malicious code from
>>> getting accepted and run by the receiving end.
>>>
>>>
>> Yes, but I wasn't able to marshal/unmarshal them correctly as shown in
>> my examples
>>
>
> The marshal module can be used to dump the Code of an Function (or
> anything else), like you have done. If you have the Code, you can
> reconstruct the function.
>
>
>>>> import types
>>>> import marshal
>>>> def a(): pass
>>>>
> ...
>
>>>> s=marshal.dumps(a.__code__)
>>>> f=types.FunctionType(marshal.loads(s), {})
>>>> f
>>>>
> <function a at 0x7f6308a66de8>
>

What version of python do you have? If I try your code above I get :

>>> import types
>>> import marshal
>>> def a(): pass
...
>>> s=marshal.dumps(a.__code__)

Traceback (most recent call last):
File "<stdin>", line 1, in <module>

AttributeError: 'function' object has no attribute '__code__'

[Andreas Löscher]

> >>>> import types
> >>>> import marshal
> >>>> def a(): pass
> >>>>
> > ...
> >
> >>>> s=marshal.dumps(a.__code__)
> >>>> f=types.FunctionType(marshal.loads(s), {})
> >>>> f
> >>>>
> > <function a at 0x7f6308a66de8>
> >
>
> What version of python do you have? If I try your code above I get :
>
> >>> import types
> >>> import marshal
> >>> def a(): pass
> ...
> >>> s=marshal.dumps(a.__code__)
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> AttributeError: 'function' object has no attribute '__code__'
>

I used Version 2.6 for this. __code__ is just an alias for func_code.

>>> def a(): pass
...
>>> s=marshal.dumps(a.func_code)

This should allways work, unless the implementation of the function
object changes. (I have tested it under 2.4, 2.5 and 2.6, but it should
also work in further versions)

Best,
Andreas

[Gabriel Rossetti]

Yes, it works, thank you,
Gabriel

[Daniel Fetchinson]

> I am trying to serialize a function, class, etc and transfer it, have it
> unserialized and used.

You might want to look at pyro: http://pyro.sourceforge.net/ and also
picloud: http://www.picloud.com/

HTH,
Daniel

--
Psss, psss, put it down! - http://www.cafepress.com/putitdown