encoding/decoding pyglet objects

[greenmoss]

Hello,

It seems that encoding and then decoding pyglet objects does not work.
I'm guessing it's due to ctypes embedded in the objects, since
conventional pickling complains about these. Here is a sample of how
jsonpickle doesn't work with pyglet:

>>> import pyglet
>>> image = pyglet.resource.image('dot.png')
>>> import jsonpickle
>>> encoded = jsonpickle.encode(image)
>>> decoded = jsonpickle.decode(encoded)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/
lib/python2.7/site-packages/jsonpickle/__init__.py", line 325, in
decode
return j.restore(json.decode(string))
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/
lib/python2.7/site-packages/jsonpickle/unpickler.py", line 120, in
restore
value = self.restore(v)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/
lib/python2.7/site-packages/jsonpickle/unpickler.py", line 120, in
restore
value = self.restore(v)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/
lib/python2.7/site-packages/jsonpickle/unpickler.py", line 120, in
restore
value = self.restore(v)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/
lib/python2.7/site-packages/jsonpickle/unpickler.py", line 66, in
restore
return self._pop(self._objs[obj[tags.ID]])
IndexError: list index out of range



Here is the same thing, using pickle:

>>> import pickle
>>> pickled = pickle.dumps(image)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/
lib/python2.7/pickle.py", line 1374, in dumps
Pickler(file, protocol).dump(obj)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/
lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/
lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/
lib/python2.7/pickle.py", line 419, in save_reduce
save(state)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/
lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/
lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/
lib/python2.7/pickle.py", line 663, in _batch_setitems
save(v)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/
lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/
lib/python2.7/pickle.py", line 419, in save_reduce
save(state)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/
lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/
lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/
lib/python2.7/pickle.py", line 663, in _batch_setitems
save(v)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/
lib/python2.7/pickle.py", line 306, in save
rv = reduce(self.proto)
ValueError: ctypes objects containing pointers cannot be pickled



So the question here is: is the jsonpickle code intended to encode
objects such as the above example, that even pickle doesn't support?
If not, should jsonpickle fail with an explicit error message during
encode, instead of with a mysterious "list index out of range" error
during decoding?

[John Paulett]

To me the interesting thing here is that jsonpickle happily encodes
the data without indicating to the user that the data is not actually
stored (pickle fails on the encode, jsonpickle fails on the decode).
This is something we may want to correct.

Do you have any ideas on methods for supporting pyglet / ctypes data?

Something that would probably have to occur is the support of binary
data (e.g. jsonpickle will not serialized a File object currently).

> --
> You received this message because you are subscribed to the Google Groups "jsonpickle" group.
> To post to this group, send email to jsonp...@googlegroups.com.
> To unsubscribe from this group, send email to jsonpickle+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/jsonpickle?hl=en.
>
>

[greenmoss]

I am unfamiliar with pyglet's code, so I don't personally have any
ideas about supporting pyglet and ctypes data. I have only been able
to dig up previous mailing list posts regarding pickling pyglet data,
which seem to indicate that it requires manual workarounds, eg:

http://comments.gmane.org/gmane.comp.python.pyglet.user/163

I did notice that the standard methods of excluding attributes
(__getstate__ and __setstate__) don't seem to have an effect in
jsonpickle.

[David Aguilar]

On Fri, Jul 08, 2011 at 06:16:14AM -0700, greenmoss wrote:
> I am unfamiliar with pyglet's code, so I don't personally have any
> ideas about supporting pyglet and ctypes data. I have only been able
> to dig up previous mailing list posts regarding pickling pyglet data,
> which seem to indicate that it requires manual workarounds, eg:
>
> http://comments.gmane.org/gmane.comp.python.pyglet.user/163
>
> I did notice that the standard methods of excluding attributes
> (__getstate__ and __setstate__) don't seem to have an effect in
> jsonpickle.

Hmm that should work. We have another method, too,
which is registering handler objects.

Can you show me an example where __getstate__ and
__setstate__ do not work as expected?

I pushed a fix that makes the decoding work for the
JSON produced when encoding a pyglets.resource.image().

It's in my "refs" branch:

https://github.com/davvid/jsonpickle/tree/refs

I added warnings on decode to let you know that something went
wrong. We might want to improve this
(maybe have a way to disable warnings?)

Can you please try it out and let me know if it looks good?

I'll need a little more time to write a test case.

I'm looking through the python stdlib to see if there's a
class that we can use that exhibits this same problem
so we can add a test to cover it. If I can't find something
there I might add a conditional test that loads a common
library from /usr/lib64 using ctypes. If anybody has any
suggestions on what would make a good test object please
let me know (criteria: dependency-free and either part
of the python stdlib or a common enough library that it
would exist on both stock OS X and Linux installations).

But, yeah, it'll probably take me a bit longer to write
a good test case. This definitely needs a test.

> So the question here is: is the jsonpickle code intended to encode
> objects such as the above example, that even pickle doesn't support?
> If not, should jsonpickle fail with an explicit error message during
> encode, instead of with a mysterious "list index out of range" error
> during decoding?

We do our best, but there are cases where we cannot restore the
object. Things like database cursors, C extension modules --
that sorta thing.

The idea is that encode() can still be useful in these
situations because the JSON can still be useful to others
even if the object cannot be restored.

What we're doing is warning on decode(). That makes sense to me
since it is in-line with the idea that jsonpickle is not the only
consumer of encode()'s output.
--
David