All,

While trying to write game state, I found that objects containing
pyglet sprites, etc would not pickle due to embedded ctypes. For
example:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/
lib/python2.7/pickle.py", line 1374, in dumps
    Pickler(file, protocol).dump(obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/
lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/
lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/
lib/python2.7/pickle.py", line 419, in save_reduce
    save(state)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/
lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/
lib/python2.7/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/
lib/python2.7/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/
lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/
lib/python2.7/pickle.py", line 419, in save_reduce
    save(state)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/
lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/
lib/python2.7/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/
lib/python2.7/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/
lib/python2.7/pickle.py", line 306, in save
    rv = reduce(self.proto)
ValueError: ctypes objects containing pointers cannot be pickled

I have worked around this by manually excluding/reconstituting pyglet
images and sprites from my objects using __getstate__ and
__setstate__. The downside for me is the extra work/code that is
required. Are there any plans to make pyglet objects amenable to
pickling?

I am not sure I see a strong case for needing to pickle sprite objects. The
OpenGL data/state that would need to be reconstructed on unpickling is
significant enough to be troublesome (i.e. can't unpickle without a valid
OpenGL context).

I would generally recommend that you separate game data from graphics data,
and design your engine such that you can reload the graphics data based on
the game data.

--
Tristam MacDonald
System Administrator, Suffolk University Math & CS Department
http://swiftcoder.wordpress.com/

On Jul 14, 11:53 am, Tristam MacDonald <swiftco...@gmail.com> wrote:

Let me preface my response by saying that I'm inexperienced, so I
could well be taking a suboptimal approach. That being the case, the
most logical way to handle game and graphics data for an in-game
object would IMHO be to attach it to a single python object. This in
turn leads to manual getstate/setstate overrides, extra code, etc.

Is there some other way that people handle this type of situation?

well you could save the buildup to the instruction file of a kind,
where you would save how to reconstruct the screen later, and then
just do it.
Also, do not use pickle, its insecure.

2011/7/14 greenmoss <ktygoo...@yoderhome.com>:

On 14 July 2011 19:33, greenmoss <ktygoo...@yoderhome.com> wrote:

The preferred way to do this is to have one python object that contains a
sprite PROPERTY. In design, this is it:

  class Object:
      def __init__(self,properties):
          #set whatever you need except graphics
          self.sprite = Sprite(graphics properties)
          #set graphics properties

And either have 'wrappers' for the functions you need, for example:

      def draw(self,options):
          self.sprite.draw(options)

Or just make sure you're calling the SPRITE'S functions, and not your game
objects. Separating graphics and game code is the basis for any graphics
coding more advanced than a hello world program, that you would intend to
develop further.

If you still have trouble with the design/practice of it, reply and I will
make a tutorial detailing the process as much as possible.

That is certainly not an unreasonable way to structure a game, but as you
have become aware, it has various drawbacks when it comes to serialisation.

I prefer to maintain a very strict Model-View-Controller separation in my
code, to allow serialisation to be restricted to only data,  which is fully
separated from behaviour.

That statement is at best misleading - there is no such thing as a 'secure'
way to write data to the filesystem. Even were one to apply encryption, the
program would need to have the decryption key in order to read it again, and
that key could be trivially read from the source file or even from the
running program in memory.

The only reason that the python documentation issues the following warning:
"*The pickle module is not intended to be secure against erroneous or
maliciously constructed data. Never unpickle data received from an untrusted
or unauthenticated source*", is that it is possible to hand-craft a file
that will crash the unpickle process. In other words, only unpickle data
that you had previously pickled yourself.

--
Tristam MacDonald
System Administrator, Suffolk University Math & CS Department
http://swiftcoder.wordpress.com/

The way I tend to handle this kind of thing is that
the game object doesn't contain a direct reference to
the graphics data. Instead it contains some piece of
identifying data, such as an integer or string, and
a global dict is used to map this to the associated
graphics data.

If you like, you can hide this lookup behind a property,
so that the object appears to have the graphics data
as an attribute.

--
Greg

No, it's somewhat worse than that: it's possible to construct
a pickle that will call an arbitrary class or function in your
application with arbitrary arguments. It's conceivable that a
malicious opponent could use this to pwn your machine.

So you need to treat untrusted pickles with the same level of
caution as untrusted executables.

--
Greg

On Jul 14, 10:04 pm, Greg Ewing <greg.ew...@canterbury.ac.nz> wrote:

Greg,

What do you use instead of pickle for saving and/or serializing game
state for later reuse?

On Jul 14, 11:53 am, Tristam MacDonald <swiftco...@gmail.com> wrote:

After reading the responses in this thread relating how to handle
saving pyglet game state using pickle, the general idea seems to be
"don't do it that way", with various implementations of how to save
state while avoiding pickling pyglet objects. The solution I have been
using, eg getstate/setstate has so far worked for me, so this is
probably what I will continue to use.

However, while hopefully not sounding too presumptuous, it strikes me
that in theory and in practice a framework such as pyglet tries to
make it easier for game developers by providing convenience methods
for common back-end chores. Gracefully handling pickle and unpickle
requests seems to me compatible with this goal[1]. Am I
misunderstanding the aims of the pyglet project?

[1] Note that I'm making no claims or demands about implementation.
Instead I am opining that it is *desirable*. Rebuttals are of course
welcome.

Actually, try this simple string as an input to pickle:
pickle.loads("cos\nsystem\n(S'ls ~'\ntR.")

2011/7/14 Tristam MacDonald <swiftco...@gmail.com>:

On Jul 15, 2:57 am, Greg Ewing <greg.ew...@canterbury.ac.nz> wrote:

A nice side effect if this sort of change is that your game state and
your controller classes become completely decoupled from your
renderer.

If it is indeed a common need, then yes, I think it would be desirable to
handle it.

I am not actually clear on what the semantics of pickling OpenGL objects
looks like. Do we:

a) pickle the file name, and hope that the file is still present when we
unpickle to reload the texture from, or
b) readback the actual data from the graphics card, and pickle that?

Also keep in mind that we need to support pickling vertex data if we support
pickling textures.

Further question: is is even legal to pickle objects that will fail to
unpickle depending on external state (missing image files, or
missing/incorrect OpenGL context).

--
Tristam MacDonald
System Administrator, Suffolk University Math & CS Department
http://swiftcoder.wordpress.com/

On Jul 15, 7:01 pm, Tristam MacDonald <swiftco...@gmail.com> wrote:

My suggestion would have been to write a pyglet 'pickle' method, that
actually saves the current window/context states AND the state of the
objects given to it, in a pyglet-like fashion. These objects would be
any subclass of AbstractImage, Sprite, VertexList, etc. It would,
possibly, actually PICKLE the object, but on unpickling, besides
restoring any properties they have, pyglet would recreate the windows/
contexts in their saved state (or at least try to, and fail gracefully
if it can't), then return the list of objects as it was given, with
the objects recreated, reinitialised, and linked to the new contexts/
windows.

This would, however, involve a LOT of work, and for very customised
objects, it might fail to initialise them, etc. In the case of missing
resources, a simple graceful fail would be appropriate.

Thinking about it, it doesn't seem like a lot of work for the simple
objects, and if objects would provide their own _unpyckle_ methods for
different initialisation and linking, it should work. The prototypes
would be like this:

  def pyckle(self, objs):
      #create anything to store the data we're about to pickle, a db,
file, whatever is best
      #save any context states, flags we need, sizes, resources etc
      #      not save the resources themselves, but what they refer to
(filename, possibly)
      for obj in objs:
          #call its _pyckle_ method which will pack the stuff it needs
and return them. Then, we save that.
          #it's the developer's own responsibility to make their
personal subclass pyckle and unpyckle methods work together, so that
the data from pyckle works in unpyckle.

  def unpyckle(self, file):
     #read everything in
     #look at context data, start creating context as it was, add
flags, resize, add resources,
     #   any failing on this means a graceful fail
     #for every object in the data, we create a base object from the
class it was of (maybe actually python pickle that class so we have an
easy copy)
     #  then we call unpyckle(data_of_object,new_context), which
should: set any variables in the data, anything else that needs to be
done, then link to the new_context.

It seems feasible, and useful, at the least. What other graphics
library has the ability to save to a file, then resume just as it was?
(There are probably a few, come to think of it.)

I use JSON with explicit serialize/deserialize methods. Explicit is
better than implicit.

On Jul 15, 9:55 am, greenmoss <ktygoo...@yoderhome.com> wrote:

On Thu, Jul 14, 2011 at 10:04 PM, Greg Ewing <greg.ew...@canterbury.ac.nz>wrote:

I think this is a much better way to say it. The advice "don't use pickle,
it's insecure" makes about as much sense to me as "don't use include, it's
insecure", since including a maliciously crafted file can also pwn your
machine. Basically, only include files you trust, and only unpickle files
that you trust. I use pickle for serializing game objects all the time. I
agree with the OP that this is a good use case to support. JMHO.

-Christopher

Do you have an example you could show?

On Jul 15, 4:01 pm, Steve <dior...@gmail.com> wrote:

Since the best way to deal with pickling of pyglet objects
in saved games is usually to avoid attempting to pickle them
in the first place, it's hard to see how pyglet could do
anything to make this more convenient.

I stand by my belief that you should redesign your game
architecture so that there are no direct references from
your game logic objects to your graphics objects.

Take a piece of paper and draw a line down the middle.
Write the heading "User Interface and Graphics Objects"
on the left, and "Game State Objects" on the right.
Draw boxes representing the various objects on their
respective sides, with arrows between them representing
references.

Now, make sure that arrows that cross the dividing line
point *only* from the left side to the right side. If you
ever find yourself wanting to create an arrow from the
right side to the left side, take a deep breath, count to
ten and think of some other way to make the association.

--
Greg

I am a bear of little brain, but this sounds very tricky and not at all
desirable. There is a massive amount of information implied here, and you
absolutely *don't* want to be restoring most of it when you restore the
(x,y) and other model state of a single game object. It implies that you'd
be restoring all the opengl state repeatedly, once for every game object you
load. What if the state saved with one object contradicted that stored with
another? What if you're restoring on a different machine, so the same state
information isn't valid?

I may be wrong but I think these ideas of pickling Sprites are totally
swimming against the current. It is much easier and more useful to only
pickle simple value-type objects (e.g. integers and strings), and
re-construct any required Sprites by calling the Sprite constructor as usual
when you restart the application.

+1

Uh, again, I didn't mean actual pickling, just something HAND-MADE, that has
the same end-point. And it doesn't sound tricky at all, really. What you'd
be doing is saving the current state of the context (any enabled bits, any
disabled bits, any loaded resources, etc), restoring them (if possible, if
not, fail gracefully with a 'could not complete' message). Each game object
needs to have its hand-made pickling routine. You create a sprite? You also
add a pickle and unpickle method to it that saves what YOU want it to. By
default, for example in a sprite, you'd be saving the image filename, and
any attributes in the pickle method. In the unpickle method, it'd recreate a
Sprite class with the same filename, then set the attributes.

If _anything_ can't be done, contradicts, isn't valid, it fails with a
simple error. This isn't any different to python's pickling. It's up to you
to use it properly, it isn't a fix-all solution.

This was just an idea, but after seeing this response I'm thinking of
actually writing it up just to prove that it's not that complex.

On 19 July 2011 09:50, Jonathan Hartley <tart...@tartley.com> wrote:

On Tue, Jul 19, 2011 at 2:50 AM, Jonathan Hartley <tart...@tartley.com>wrote:

want to serialise the absolute minimum amount of data required to
reconstruct your current program state.

In other words, anything that your program has loaded from somewhere, can be
loaded again - no need to serialise it. Particularly for resources like
sprites, if your program has loaded the sprites in the first place, it
plainly knows how to load them again. Serialising sprites is just going to
duplicate your existing loading functionality, while bloating your save
files.
--
Tristam MacDonald
System Administrator, Suffolk University Math & CS Department
http://swiftcoder.wordpress.com/

Hey. Thanks for explaining further.

I don't think it helps to do your serialisation by hand, instead of using
pickle - the same objections hold.

The things I infer that you are talking about serialising - sprite content
and opengl state - these are not part of your application's model state. To
save them and then restore them along with your game state is conflating two
very different things, and all the objections I stated above, together with
plenty of others, will cause no end of problems.

Suppose the opengl state contains information about the open window, such as
it's size, or screen resolution, or the display it is on. When the user
changes this state, by moving the window to another display, they do not
want this change to be reset when they reload their game!

Suppose the user saves game on one computer, but then reloads it on another
computer. The saved opengl state might not even be valid.

Suppose they update the application, which includes a nice graphical
makeover which improves the appearance of the sprites. Then they load their
old save game, and this reloads the old ugly sprites again.

I don't want to be patronising, but I think you might benefit from reading
about the model-view-controller pattern. I'm not usually a big 'patterns'
kind of person, but this one seems especially pertinent here.

Sorry to be so negative! But those are my thoughts. Obviously I wish you
nothing but success with whatever strategy you choose to go forward with.
Cheers!

Uh. I'm not talking about serialisation. I used the word 'pickle'
since that's what it all started with, but at one point, the talk
diverged to a way to save the current windows and objects and restore
them. Serialising or not. You aren't 'serialising' unless you choose
to in this. This is probably my fault for still saying pickle, etc.

Let me try and re-explain with different words, so that I can possibly
get the point across. For every object, like a Sprite, you'd have a
method like such:

def save(self,file):
  #to the file, do the following:
  #  save self.x
  #  save self.y
  #  ...
  #  save self.image_path

This is pseudo-code. You don't need to save to a file, you don't need
to serialise everything. There are lots of ways to save these things,
the only important thing is that the 'saved' things need to persist
until load is called. For ease of example, I'll assume this somehow
saves everything to a file.

def load(self,file):
  #from the file, load the following:
  #  load x into tmpx
  #  load y into tmpy
  #  ...
  #  load path into tmppath
  #create a NEW sprite, with the init options we have saved, tmpx,
tmpy, etc.
  #set any other things we've saved but can't set while we initialise,
like special tex coords, blending, depends what you set.
  #replace the current object or return the new object.

These two methods are defined for every object you want to be able to
save. This includes a window. A window would do this:

def save(self,file):
  #into file:
  #  save enabled bits
  #  save disabled bits
  #  save current size
  #  ...
  #  save whatever else we need

def load(self, file):
  #from file:
  #  load everything into temporary vars
  #create a NEW window/context
  #set the enabled bits in that new window to those we used to have.
  #disabled bits, resize it, etc.
  #replace the current instance or return the new one.

If you have a way of getting what screen the window's on, and you WANT
it to remember what screen it was on, it's a matter of also saving/
loading that. If you don't want to, you just omit saving/loading that.

The global pyglet.save() method would take a list of objects that
it'll call obj.save() on, then store (in this example) which file
they're in into a 'manager' file. The global pyglet.load() method
would read the manager file, and one-by-one load each file and load()
the object into creation. Last, it'll create the saved window/context,
then go through each object, and make sure it's pointing at the new
context where it needs to. If at ANY point, it gives an error,
everything stops and the error is reported.

Just to be clear, this is NOT a way of saving/loading your game, nor
is it a good way to theoretically. This is where the discussion
started, but I kind of went off on a tangent. This is meant to be a
way to be able to close and restore the actual graphics themselves
(generally on the same machine, but similar ones will do fine).
Literally, you'd be able to close, then when you start it again,
EVERYTHING would be as you left it (provided you make sure you save
every thing you need and load them back properly). It's a way of
'pickling' the actual graphics objects. If you serialise the objects
and store them, you could say you're actually pickling them.

That in mind, if you make sure you overwrite the default save/load
functions, and only use sprite/window/etc subclasses for your game
logic, it would work just fine as saving/loading your game, up until
the point where you try it on a machine where restoring the settings
wouldn't work. (At that point, it's a matter of handling it sanely, by
making sure the load() function looks where it failed and ignores
loading that specific bit. This would crash horrendously on resources
missing, but would work for, say, some type of antialiasing not
working).

Hope this clears up some things, thanks for the replies.

On Jul 20, 12:30 am, Jonathan Hartley <tart...@tartley.com> wrote:

You're missing the point: attempting to save OpenGL state
in a game save file is a wrongheaded thing to be doing in
the first place, IMO.

--
Greg