A way to use pickle instead of brine

[Fruch]

I want to rule out a bug in rpyc while working with multiple POSIX
threads,
is there any command switch for a connection to use pickle instead of
brine ?

[Fruch]

my initial problem is that an async from the server side is being
called
from two different thread simultaneously.
(it's a C callback that is wrapped by SWIG.)

so a new question (that might solve my initial problem)
is brine.dump thread-safe ?
and if it's not why it's outside of the _sendlock ?

def _send(self, msg, seq, args):
data = brine.dump((msg, seq, args))
self._sendlock.acquire()
try:
self._channel.send(data)
finally:
self._sendlock.release()

i'll try to put it inside tomorrow to see if that solve the problem

[tomer filiba]

brine is thread safe. it's almost purely functional ( http://en.wikipedia.org/wiki/Purely_functional ).

the problems you experience are most likely not related to serialization, but to python thread handling being screwed up from the bottom.
of course there's no "command switch" to replace brine with pickle. rpyc is a library, not a command line tool.

if you really think it's related to brine, try patching up brine.py with something like the following:

[ core/brine.py ]
def dump(obj):
    return pickle.dumps(obj)

def load(data):
    return pickle.loads(data)

that should be a pain-free replacement of brine, so you can try it out.


hope that helps,
-tomer

An NCO and a Gentleman

[Fruch]

I was probably barking at the wrong tree,
here is the trackback of the failure:

Exception in thread Thread-1:
Traceback (most recent call last):
File "/NDS/bin/py/lib/python2.6/threading.py", line 525, in
__bootstrap_inner
File "/NDS/bin/py/lib/python2.6/threading.py", line 477, in run
File "/NDS/bin/py/lib/python2.6/site-packages/rpyc/utils/server.py",
line 110, in _authenticate_and_serve_client
File "/NDS/bin/py/lib/python2.6/site-packages/rpyc/utils/server.py",
line 127, in _serve_client
File "/NDS/bin/py/lib/python2.6/site-packages/rpyc/core/
protocol.py", line 312, in serve_all
File "/NDS/bin/py/lib/python2.6/site-packages/rpyc/core/
protocol.py", line 304, in serve
File "/NDS/bin/py/lib/python2.6/site-packages/rpyc/core/
protocol.py", line 274, in _dispatch
File "/NDS/bin/py/lib/python2.6/site-packages/rpyc/core/
protocol.py", line 233, in _dispatch_request
File "/NDS/bin/py/lib/python2.6/site-packages/rpyc/core/
protocol.py", line 161, in _send_reply
File "/NDS/bin/py/lib/python2.6/site-packages/rpyc/core/
protocol.py", line 150, in _send
File "/NDS/bin/py/lib/python2.6/site-packages/rpyc/core/brine.py",
line 276, in dump
File "/NDS/bin/py/lib/python2.6/site-packages/rpyc/core/brine.py",
line 162, in _dump
File "/NDS/bin/py/lib/python2.6/site-packages/rpyc/core/brine.py",
line 156, in _dump_tuple
File "/NDS/bin/py/lib/python2.6/site-packages/rpyc/core/brine.py",
line 162, in _dump
File "/NDS/bin/py/lib/python2.6/site-packages/rpyc/core/brine.py",
line 155, in _dump_tuple
TypeError: an integer is required

and this I saw this strange thing
def _dispatch_request(self, seq, raw_args):
try:
handler, args = raw_args
args = self._unbox(args)
res = self._HANDLERS[handler](self, *args)
except KeyboardInterrupt:
raise
except:
t, v, tb = sys.exc_info()
self._last_traceback = tb
if t is SystemExit and
self._config["propagate_SystemExit_locally"]:
raise
self._send_exception(seq, t, v, tb)
else:
self._send_reply(seq, res)

maybe my python isn't that solid, but what is an else got to do with
try & except ?

On Mar 10, 8:22 pm, tomer filiba <tomerfil...@gmail.com> wrote:
> brine is thread safe. it's almost purely functional (http://en.wikipedia.org/wiki/Purely_functional).
>
> the problems you experience are most likely not related to serialization,
> but to python thread handling being screwed up from the bottom.
> of course there's no "command switch" to replace brine with pickle. rpyc is
> a library, not a command line tool.
>
> if you really think it's related to brine, try patching up brine.py with
> something like the following:
>
> [ core/brine.py ]
> def dump(obj):
>     return pickle.dumps(obj)
>
> def load(data):
>     return pickle.loads(data)
>
> that should be a pain-free replacement of brine, so you can try it out.
>
> hope that helps,
> -tomer
>
> An NCO and a Gentleman
>

[Alex Grönholm]

This is perfectly valid Python. An "else" block in such case is executed
if and only if an exception has NOT been raised in the try block. If an
exception is raised in the else block, it will not be handled in the
parent try block's except blocks.

[Fruch]

I've found a solution, for some reason i was using rpyc.async around
my callback function
it was working most of the time, but in the specific timing it wasn't

I remove the async decorator, and no it's working find

BTW I really like rpyc,

while i was reading stream.py
I've noticed you missed an important thing,
errno EINTR should retry too,

retry_errnos = set([errno.EAGAIN, errno.EINTR])
if hasattr(errno, "WSAEWOULDBLOCK"):
retry_errnos.add(errno.WSAEWOULDBLOCK)
if hasattr(errno, "WSAEINTR"):
retry_errnos.add(errno.WSAEWOULDBLOCK)

and anther thing:
itertools.count is not tread-safe

doing something like this might be safer.
(python that need to use c extension should count on the GIL too
much)

def synchronized_iterator(f):
class iterator(object):
def __init__(self,f):
self.iter = f()
self.lock = RLock()
def __iter__(self):
return self
def next(self):
self.lock.acquire()
try:
return self.iter.next()
finally:
self.lock.release()
return iterator(f)

[tomer filiba]

what makes you think itertools.count is not thread safe?

it's implemented in c, and thus is inherently thread safe, since only the thread holding the GIL can execute at any given point (and it's not an IO-bound operation, so it doesn't release the GIL). afaict, itertools.count *is* thread safe.

see http://svn.python.org/view/python/tags/r264/Modules/itertoolsmodule.c?revision=75707&view=markup and grep for count_next

-tomer

An NCO and a Gentleman