serialize and deserialize objects in Pypy

[Yicong Huang]

Hi,

Are there any good methods to serialize Pypy object to C++ char* and deserialize to Pypy object from  C++ char*?

Our scenario like this:

We have at least two C++ process running, and each process embeded Pypy to execute some functions.

And we woule like to reuse Pypy object from one process to another process.

For primitive objects, int/char/bool/double etc., we could use cffi to pass objects.

Pypy object -> c++ process 1 -> c++ process 2 -> Pypy object

But for complex objects, e.g. list/array/dict/tuple, are there any methods?

[Yicong Huang]

Hi Maciej,

Could you provide some details on how to use cffi buffers? (a simple demo code or documents)

I tried the below code but it did not work.

#python list

x=[1,2,3,4]

#intend to convert to c++ char*, but failed

p=ffi.from_buffer(x)

#intend to covert back to python list, but failed

y=ffi.buffer(p)

Thanks!

-Ethan

On Fri, Aug 7, 2015 at 6:24 PM, Maciej Fijalkowski <fij...@gmail.com> wrote:

you can use cffi buffers (e.g. cffi char*) that you manipulate from
python using wrappers. They would behave (sort of) like python objects
and the wrapper code is really not a performance penalty in most cases

> _______________________________________________
> pypy-dev mailing list
> pypy...@python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>

[Yicong Huang]

The problem is we might need to support several kinds of collection objects: list, dict, tuple.

In addition, we might not know what is exact type of objects stored in the collections. A list might store int, string, double.

So it might involve a lot of work to write the python wrapper.

Do you have any good ideas?

On Fri, Aug 7, 2015 at 7:46 PM, Maciej Fijalkowski <fij...@gmail.com> wrote:

you really can't convert it to the C stuff like that. You need to do:

foo = ffi.new("int[]", 13)
foo[0] = 12
foo[1] = 55

etc.

You can write nice python wrapper around those things too, if the data
is too bare bone.

[Maciej Fijalkowski]

yes, you need a serializer protocol, like JSON (or protobuf or
anything). You generally need to express stuff in a way C can
understand and read it in python

[Yicong Huang]

In general, we don't need to parse or process Pypy object in C code, but only transfer the object from one Pypy enviroment to another Pypy enviroment.

I am thinking could we have the alternavtive easier way to do this, like PyObject in Python?

Could we get the address of the Pypy Object and the length of the object, and memcpy the content from Pypy to C++ and then from C++ to Pypy?

[Armin Rigo]

Hi Yicong,

On 7 August 2015 at 15:13, Yicong Huang <hengh...@gmail.com> wrote:
> Could we get the address of the Pypy Object and the length of the object,
> and memcpy the content from Pypy to C++ and then from C++ to Pypy?

No. And if you try the same thing on CPython with PyObjects, it would
crash in 99% of the cases too.


A bientôt,

Armin.