Re: [rpyc] ValueError: pickling is disabled

[Tomer Filiba]

See allow_pickle at http://rpyc.sourceforge.net/api/core_protocol.html#api-protocol

https://github.com/tomerfiliba/rpyc/blob/master/rpyc/core/protocol.py#L547

I have no idea where you got  thread safety issues from

On Oct 8, 2012 9:02 PM, "samovar" <mrai...@gmail.com> wrote:

Running as a service and trying to write a pickle file on the remote, I get:

ValueError: pickling is disabled

I read that pickling is disabled because it is not thread safe.

Of course it's not thread safe. No one who is capable to use Rpyc would think it is.
Static class data and @classmethod are not thread safe either. Should these also be disallowed? There not.

I will use a locking mechanism to deal with thread safety issues for pickle. I should be able to do this.
Making it totally unavailable is wrong.

Am I wrong?

[ben]

yet, when i try to make a deepcopy of an object after allowing pickling, i get the following error:

File "/auto/sw_tools/root/lib/python2.7/copy.py", line 182, in deepcopy
    rv = reductor(2)
  File "/auto/sw_tools/root/lib/python2.7/site-packages/rpyc-3.2.1-py2.7.egg/rpyc/core/netref.py", line 176, in __reduce_ex__
    return pickle.loads, (syncreq(self, consts.HANDLE_PICKLE, proto),)
  File "/auto/sw_tools/root/lib/python2.7/site-packages/rpyc-3.2.1-py2.7.egg/rpyc/core/netref.py", line 71, in syncreq
    return conn.sync_request(handler, oid, *args)
  File "/auto/sw_tools/root/lib/python2.7/site-packages/rpyc-3.2.1-py2.7.egg/rpyc/core/protocol.py", line 433, in sync_request
    raise obj
cPickle.PicklingError: Can't pickle <type 'thread.lock'>: attribute lookup thread.lock failed

(I am running a Treaded server...)

???

[Tomer Filiba]

how about you post a code snippet ???

-----------------------------------------------------------------

Tomer Filiba  tomerfiliba.com

[Tomer Filiba]

and even though you're rude, try looking at rpyc.classic.obtain/deliver

https://github.com/tomerfiliba/rpyc/blob/master/rpyc/utils/classic.py#L251

-----------------------------------------------------------------

Tomer Filiba  tomerfiliba.com

[ben]

Hi Tomer,

please accept my appology, was not trying to be rude.

thanks for the reply, the obtain/deliver methods looks really promising, although i am unable to get them to work as yet (possibly because i am not using the classic mode but rather the service mode)

on the server i am creating a list of objects and trying to pass them back to the client, and be able to have some form of access to them after the connection has been terminated.

connection code:

  @contextmanager
  def connection_to_root(self):
    conn = connect(self._agent_host, self._agent_port,
                         config = {"allow_all_attrs" : True,
                                   'allow_pickle' : True})
    conn.root.setup(*self._args, **self._kwargs)
    yield conn.root.root
    conn.close()

from the client, i then use code like:
  with Client(ip).connection_to_root() as c:
    return [SomeLocalClass(x) for x in xrange(5)]

Obviously, as soon as i leave the context, my "weakly referenced" object are no longer available. and when i try to deepcopy them, i get the error above with the lock.

how would i use the obtain/deliver to get what i need, everything i passed to it so far as the connection complains that it does not have the modules attribute required to execute the following line:
return conn.modules["rpyc.lib.compat"].pickle.loads(pickle.dumps(localobj))

again, appologies for the rudeness

[ben]

Hi Tomer,

any suggestions regarding my previous post?

thanks

[Tomer Filiba]

> how would i use the obtain/deliver to get what i need, everything i passed to it so far as the connection complains that it does not have the modules attribute required to execute the following line:

> return conn.modules["rpyc.lib.compat"].pickle.loads(pickle.dumps(localobj))

well, it won't work because your connection object doesn't have the modules attribute (it's implemented by the SlaveService).

you have two options: 

* derive your class from SlaveService (which may be a security risk, if you're not using a secure, local network, but otherwise it's fine)

* do the pickling yourself: 

    * make sure that allow_pickle is enabled on *both sides* (the code snippet you gave seems to configure it only on the client, so be sure to configure the server as well)

    * just import pickle and do pickle.loads(pickle.dumps(x)) where x is your netref object

    * be sure that the classes you wish to recreate on the client exist there (e.g., if the server returns an object of type MyClass, you have to have MyClass on the client as well) -- that's how pickle works

reading through your explanation, however, it seems you're not "using rpyc right". rpyc objects are not meant to survive the connection terminating -- they rely on a long-living connection.

what you seem to require is "plain old RPC". you can do it with rpyc, of course, but it's an overkill.

in your case, and using rpyc, i'd just have a function that returns "raw data", e.g., dicts/lists/strings/ints, and pass it pickled.

then i'd load this data on the client and process it there, creating classes to represent it, etc.

hope this helps,

-tomer

-----------------------------------------------------------------

Tomer Filiba  tomerfiliba.com