pickling generators
Paul Boehm
we're developing a python based adventure game engine for a
fangame effort, and we'd like to use generator based lightweight threads,
to let each of our objects be scripted mostly independently of each other,
without all of the hassles of strictly event oriented coding.
our problem, however, is that we can't properly do savegames with this design,
as generator instances are not pickle- and unpickleable.
we're now looking for options, and advice, on the feasability
of getting support for this into future python versions, either by learning
how to do it, and implementing it ourselves, or by finding someone willing
to help us out.
thanks a lot,
paul
Gerrit Holl
> our problem, however, is that we can't properly do savegames with this design,
> as generator instances are not pickle- and unpickleable.
True.
> we're now looking for options, and advice, on the feasability
> of getting support for this into future python versions, either by learning
> how to do it, and implementing it ourselves, or by finding someone willing
> to help us out.
I have written some code to do this. It assumes:
* When the same arguments are passed to the generators function, it
yields the same values,
* No value is yielded twice, or it is yielded forever,
* Requires Python 2.3.
My __getstate__ contains:
if isinstance(d[attr], types.GeneratorType):
g = d[attr]
try:
# dropwhile includes, so this is correct
latest = g.next()
except StopIteration: # no need for pickling
pass
else:
fm = g.gi_frame
# XXX: is this safe?
func = fm.f_globals[inspect.getframeinfo(fm)[2]]
argnames, _, _, argvalues = inspect.getargvalues(fm)
args = tuple([argvalues[a] for a in argnames])
d[attr] = (func, args, latest)
My __setstate-_ contains:
if isinstance(val, tuple) and len(val) == 3 and callable(val[0]):
func, args, latest = val
def f(x): return x != latest
g = func(*args)
newg = itertools.dropwhile(f, g)
setattr(self, key, newg)
It may me possible to do this better, however; I'm eager to hear about that!
You can also use a class iterator with a __getstate__ and __setstate__ to
solve this problem.
yours,
Gerrit.
--
230. If it kill the son of the owner the son of that builder shall be
put to death.
-- Hammurabi, Code of Law
--
Asperger Syndroom - een persoonlijke benadering:
http://people.nl.linux.org/~gerrit/
Het zijn tijden om je zelf met politiek te bemoeien:
http://www.sp.nl/
Paul Boehm
> I have written some code to do this. It assumes:
>
> * When the same arguments are passed to the generators function, it
> yields the same values,
> * No value is yielded twice, or it is yielded forever,
> * Requires Python 2.3.
this is not suitable for our engine, we need to HARD restore state,
not try to recreate it.
thanks for sharing your code and ideas though!
paul
Terry Reedy
"Paul Boehm" <pa...@soniq.net> wrote in message
news:mailman.104834151...@python.org...
> hi,
>
> we're developing a python based adventure game engine for a
> fangame effort, and we'd like to use generator based lightweight
threads,
> to let each of our objects be scripted mostly independently of each
other,
> without all of the hassles of strictly event oriented coding.
>
> our problem, however, is that we can't properly do savegames with
this design,
> as generator instances are not pickle- and unpickleable.
Have you looked at Stackless? Don't know if tasklets can be pickled.
tjr
Oren Tirosh
> hi,
>
> we're developing a python based adventure game engine for a
> fangame effort, and we'd like to use generator based lightweight threads,
> to let each of our objects be scripted mostly independently of each other,
> without all of the hassles of strictly event oriented coding.
>
> our problem, however, is that we can't properly do savegames with this design,
> as generator instances are not pickle- and unpickleable.
>
> of getting support for this into future python versions, either by learning
> how to do it, and implementing it ourselves, or by finding someone willing
> to help us out.
Stackless python has picklable execution state somewhere in its roadmap
for exactly this kind of application. I'm afraid it won't be available
soon enough for your purposes. I know that Christian Tismer is looking
for sponsors for his work on Stackless.
Oren
Jeremy Bowers
> this is not suitable for our engine, we need to HARD restore state, not
> try to recreate it.
Ahhh, there really isn't any such thing. The very act of pickling and
unpickling is "recreating state". It's not like it just loads a binary
image into memory.
A lower-tech, but significantly more bullet-proof, is to instantiate a
class around each generator and store all data the generator might call
"local" in that class, then pickle the class normally. Your generators are
quite likely either glorified loops, or initialization + glorified loops,
and this can be made to work, with minimal additional state variables to
help the generators over the parts they should skip. (Sometimes you can
also write non-destructive initializers (if not hasattr(self, "name"):
self.name = "initial name") so that even if the generators re-do that part
the first time they run after unpickling it won't hurt anything.)
It may not seem initially like it will work, but it can almost certainly
be made to do so with some creativity. (I'd even expect that you'd reap
some unexpected benefits in other areas as a result of such careful
refactoring.)
Frankly, I'm surprised that the solution Gerrit Holl posted works at all.
I wouldn't expect anything better to come along anytime soon. You're going
to have to do some work to make this go. Bleh. ;-)
Christian Tismer
I think I can do some support for this. With generators
in their current shape, it would be even possible without Stackless,
although the latter would give you more flexibility. Tasklets are
more flexible than generators, but generators might do what you need.
If you're interested, let's talk about details.
Thanks Oren for your support.
ciao - chris
--
Christian Tismer :^) <mailto:tis...@tismer.com>
Mission Impossible 5oftware : Have a break! Take a ride on Python's
Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/
14109 Berlin : PGP key -> http://wwwkeys.pgp.net/
work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776
PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04
whom do you want to sponsor today? http://www.stackless.com/
Christian Tismer
...
> Have you looked at Stackless? Don't know if tasklets can be pickled.
Will be, sooner or later. The more often you ask, the sooner :-)