Support for refs/STM

[Leo Franchi]

Hi guys,

I just found this project and it looks exciting! 

I'm looking around for ways to contribute, and stumbled on an email from May saying that parts of the STM/ref/LockedTransaction code still needed to be ported from the JVM. Is that still the case? I'm interested in taking a crack at it if so :) 

cheers,

leo

[Eric Shull]

Hi Leo,

I don't think that's been tackled yet. Have at it!

Eric

[Leo Franchi]

Hi guys,

So I've added support for refs + transactions in my clojure-py branch here:

https://github.com/lfranchi/clojure-py/tree/refs

It's a pretty direct port of the STM from clojure itself, mostly contained in Ref.py and LockingTransaction.py. There are also some pretty comprehensive tests in ref-tests.py, and threaded-transaction-tests.py. Some immediate things that come to mind:

Clearly the massive value that the STM provides clojure is a bit less beneficial as we're stuck with the GIL. On the other hand, clojure-py needs some sort of ref/stm system if it wants to retain even mildly compatible with clojure. That said, I imagine there are some more intelligent ways to go about leveraging the STM while not losing all semblance of concurrency. I took a stab at using multiprocessing to distribute functions across processes, but I think the level of magic required to get that to work transparently for functions + closures + shared state is a bit too complex for me. I'd be interested to chat about alternatives if anyone has some cool ideas---I saw Timothy has some cool plans for distributed concurrency/threadlets + generators, as well.

Anyway, I'll be making pull request after a few cleanups/squashed, and any and all comments/reviews are appreciated. 

cheers,

leo

[Antony Lee]

Hi,
Looks very nice, but I kind of dislike the SharedLock piece of code it depends on (or at least I would rework it to get rid of the dependency on exc_string).  I know very little about multithreaded code so what I say may be worthless, but I believe that most of the functionality can be written very simply (nontested code, but you probably get the idea).
Also, have you looked at what PyPy proposes in term of STM?  That may be where the real money is... (again, I know little abot that, just wondering.)
Antony

class SLock(object):
    def __init__(self):
        self._lock = Lock()
        self._counter = 0 # -1: write lock; >0: read lock
        self._not_share_locked = Event()
        self._not_share_locked.set()
        self._atomic_lock = Lock()

    def acquire_shared(self):
        with self._atomic_lock:
            with self._lock:
                self._not_share_locked.clear()
                self._counter += 1

    def release_shared(self):
        with self._atomic_lock:
            self._counter -= 1
            if self._counter == 0:
                self._not_share_locked.set()

    def acquire(self):
        while True:
            self._not_share_locked.wait()
            with self._atomic_lock:
                if self._not_share_locked.is_set():
                    self._lock.acquire()
                else:
                    continue

    def release(self):
        with self._atomic_lock:
            self._lock.release()
            self._counter = 0

2012/10/30 Leo Franchi <lfra...@kde.org>

[Leo Franchi]

Hey,

I can extract the exc_string dep from SharedLock, it's mostly used for formatting of debugging and the like. However, it's a bit tricker for the rest: I need it to be re-entrant (so locking it from the same thread does not block), and I also need to support arbitrary timeouts---blockAndBarge waits .1s for the lock to be free, otherwise retries. I figured a thread-safe implementation of those two features on top of a standard read/write mutex might have taken more time than the rest of  the STM stuff. 

Regarding the PyPy STM, I know it's planned but I haven't really had a chance to dig into it. Would be cool to be able to tie into that when/if running on PyPy. That said, it seems like there are some blockers that hold clojure-py back from running on PyPy atm? Or at least in compiling down to RPython byte code? I haven't really  been on top of that discussion.

cheers,

leo

[Antony Lee]

For the reentrant lock part, does a thread have to be able to hold both
a shared and a non-shared lock at the same time, or is it just the
non-shared lock that has to be reentrant? In the latter case,
self._lock = RLock() should be enough. In the former case, some extra
work may be needed, yes.

Adding a timeout is probably not too hard, especially as Event already
supports timeouts (as well as Python 3's version of Lock, but well...)

In any case if you think the implementation of SharedLock you used is
the best way to go then so be it, I just want to avoid adding huge
pieces of code to the repo when simpler solutions would be enough.

I haven't followed too much the PyPy side of things either, probably
should do some reading before saying nonsense in my emails :-)

Best,

Antony