Python Pickle Error - "Can't pickle <class 'sqlalchemy.orm.session.Session'>"
Lynton Grice
I am writing a little persistant Queue class/s using sqlalchemy and
sqlite. All is going fine with the development but now I have reached
the stage where I would like to have each request spawned as a new
process (from Python's muliprocessing module).
The error I am getting is: "Can't pickle <class
'sqlalchemy.orm.session.Session'>" it's not the same object as
sqlalchemy.orm.session.Session
pickle.PicklingError:
In my one class that is the closest to SQLite I have the following in
the INIT method:
def __init__(self, queueConnection):
self._queue = queueConnection.queue
self._queueName = queueConnection.queue.name
self._engine = queueConnection.engine
self._sessionmaker = sessionmaker(bind=self._engine)
self._session = self._sessionmaker()
mapper(_Message, self._queue)
And if the calling client called the "SEND" method to send a message
to the queue...the code is as follows:
def send(self, message, timeout=10, callback=None):
start_pipe, end_pipe = Pipe()
start_pipe.send(message)
p = Process(target=self._queue_proxy.send, args=
(end_pipe,)).start()
result = p.join(timeout = timeout)
if result == False:
return "Timeout"
else:
return start_pipe.recv()
The exception is raised when the following line of code runs above:
p = Process(target=self._queue_proxy.send, args=(end_pipe,)).start
()
NOTE: The "self._queue_proxy" object above is essentailly the object
that holds the reference to the "SessionMaker" above.
Does anyone have any idea how I can get around this Pickling error?
Any help will be greatly appreciated ;-)
Lynton
Antoine Pitrou
> Does anyone have any idea how I can get around this Pickling error?
Different ways to make your class picklable are described here:
http://docs.python.org/library/pickle.html#pickling-and-unpickling-normal-class-instances
You could for example write __getstate__ and __setstate__ methods such
that the session / sessionmaker objects don't get pickled when pickling
your class, and are recreated on the fly when unpickling.
Regards
Antoine.
Michael Bayer
> Hi there,
>
> I am writing a little persistant Queue class/s using sqlalchemy and
> sqlite. All is going fine with the development but now I have reached
> the stage where I would like to have each request spawned as a new
> process (from Python's muliprocessing module).
The result of "sessionmaker()" is a class. Its defined at the module
level and should be globally present in memory, and automatically
available to any child processes that are spawned thereafter. It's not
something that's really appropriate to be passing around even within a
single process in most cases - its part of your application's globally
available structure.
Additionally, when you use multiprocessing, any arguments passed across
the boundary of a process are pickled, sent over a pipe, and then
unpickled on the other side. For this reason, it is critical that the
arguments passed between parent/child are minimized in terms of their
size. This is in the multiprocessing docs:
http://docs.python.org/library/multiprocessing.html#programming-guidelines
. So I would also argue that you really shouldn't be passing *any*
significant state across process boundaries. I would pass only small
easily-packageable bits of state that are just enough for the child
process to reconstruct what it needs from the application's global state
as well as the database. Things like classes, sessions, engines,
connections and transactions are not candidates for sending over a pipe.
> --
>
> You received this message because you are subscribed to the Google Groups
> "sqlalchemy" group.
> To post to this group, send email to sqlal...@googlegroups.com.
> To unsubscribe from this group, send email to
> sqlalchemy+...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/sqlalchemy?hl=en.
>
>
>
Lynton Grice
now...
Lynton
On Dec 18, 7:19 pm, "Michael Bayer" <mike...@zzzcomputing.com> wrote:
> Lynton Grice wrote:
> > Hi there,
>
> > I am writing a little persistant Queue class/s using sqlalchemy and
> > sqlite. All is going fine with the development but now I have reached
> > the stage where I would like to have each request spawned as a new
> > process (from Python's muliprocessing module).
>
> The result of "sessionmaker()" is a class. Its defined at the module
> level and should be globally present in memory, and automatically
> available to any child processes that are spawned thereafter. It's not
> something that's really appropriate to be passing around even within a
> single process in most cases - its part of your application's globally
> available structure.
>
> Additionally, when you use multiprocessing, any arguments passed across
> the boundary of a process are pickled, sent over a pipe, and then
> unpickled on the other side. For this reason, it is critical that the
> arguments passed between parent/child are minimized in terms of their
> size. This is in the multiprocessing docs:http://docs.python.org/library/multiprocessing.html#programming-guide...
> >http://groups.google.com/group/sqlalchemy?hl=en.- Hide quoted text -
>
> - Show quoted text -