sharing data between threads

[Tamas]

Hi,

I have never done thread programming (and I do not know if I
understand it well).
I would like to create a dictionary object initiated from a database
at the startup of my cherrypy application.
After that I would like to read and modify this dict obj from the
application.

I think something like this could work, but I do not even know how to
test/check it :-)

(pseudo code; missing try/except and other stuff to make attention to
avoiding blocking)
#***************************
class MyApp:
@exposed
def myfunct:
global glbLock
global glbDict
glbLock.acquire()
glbDict['key1'] = 'something'
glbLock.release()

#***************************
glbLock = threading.Lock()
glbDict = read_from_db()
cherrypy.quickstart( MyApp())

Thanks for your halp in advance,
Tamas

[Robert Brewer]

That really, definitely, absolutely needs to read:

glbLock.acquire()
try:
glbDict['key1'] = 'something'
finally:
glbLock.release()

...or you will get deadlocks. Just be ready for the "trial by fire" of
sharing state across threads. You'll think you understand it all many
times and be proved wrong by irate customers each time. But it's the
only way to learn the pitfalls and it's worth it. :) Writing *lots* of
tests helps ease the pain.

Robert Brewer
fuma...@aminus.org

[Dirk Rothe]

If you guard all your globals during write operations with locks you
should be on the safe side.
I think read operations should fine without locking - but I'm not sure
here.

--dirkse

[Asim]

You should check out the source code for two Python packages for two
possible approaches.

Firstly, get shove (http://pypi.python.org/pypi/shove/), install it
(from the site, or using easy_install). Then check out the
"__init__.py" file in the root of the shove package, and you'll find
this synchronized decorator:

def synchronized(func):
'''Decorator to lock and unlock a method (Phillip J. Eby).

@param func Method to decorate
'''
def wrapper(self, *__args, **__kw):
self._lock.acquire()
try:
return func(self, *__args, **__kw)
finally:
self._lock.release()
wrapper.__name__ = func.__name__
wrapper.__dict__ = func.__dict__
wrapper.__doc__ = func.__doc__
return wrapper

shove uses this decorator interally to achieve the same objective
you're getting at. You could just use shove for your threaded
application; check out the "tests" subdirectory in the package for
usage instructions. however, you can check out the "/store/bsddb.py"
file to see how the "synchronized" decorator is used, e.g.:

class BsdStore(SyncStore):
'''Class for Berkeley Source Database Store.'''
...
@synchronized
def __getitem__(self, key):
return super(BsdStore, self).__getitem__(key)
...

(The @ symbol is how decorators are used in Python)

A second approach is to use the "bsddb" module directly within Python
2.5 (and only 2.5, not 2.4). This is because of the following notice
(http://docs.python.org/lib/module-bsddb.html):

"The following is a description of the legacy bsddb interface
compatible with the old python bsddb module. Starting in Python 2.5
this interface should be safe for multithreaded access. The bsddb.db
API is recommended for threading users as it provides better control."

I'm currently trying to figure out how to use bsddb's native threading
support within cherrypy. When I finish this effort I'll gladly share
my work.

For now, you should check out tests that come with Python 2.5 (in my
Windows distribution, the useful test cases are in side \Python25\Lib
\bsddb\test\test_thread.py). Within this test file you'll find,
amongst others, a "ConcurrentDataStoreBase" class, which describes
multiple-reader single-writer access to the bsddb database, and a
"ThreadedTransactionsBase" class for multiple-reader multiple-writer
transaction access to a bsddb database.

Here is a reference to the Python bsddb3 module:

http://pybsddb.sourceforge.net/bsddb3.html

And here is a reference to the C++ version of bsddb, which may be
helpful in describing functionality not covered in the Python
reference:

http://www.oracle.com/technology/documentation/berkeley-db/db/ref/toc.html

I've written a lot in an unstructured fashion, but if you'll notice
both my suggestions use bsddb. My takehome message is that shove uses
bsddb in a simple fashion to serialize all accesses to the bsddb
database using decorators. Using bsddb natively will give you greater
performance and more configurability at the price of complexity.

HTH,
Asim

[gregjor]

Another technique: use a one-entry Queue to serialize access to the
shared data structure.

---
import Queue

queue = Queue.Queue(1)
...
# init global dict
queue.put({"foo": 1, "bar": 2})

...
# get a copy of the global dict
gd = queue.get()
d = gd.copy()
queue.put(gd)
if (d["foo"] == 1): ...

# update the global dict
gd = queue.get()
gd.update({"bar": 43, "baz": 99})
queue.put(gd)
---

This works because queue.get() blocks until it can get the shared dict
object from the queue, so only one thread can "have" the shared dict
at any time. Any code that uses queue.get() MUST use queue.put() to
put the shared dict back into the queue.

This should all be wrapped up in a module that has the queue as a
private module-level object, and function to encapsulate access to the
shared dict, with exception handling, e.g.

def updateGlobalDict(d):
gd = queue.get()
try:
gd.update(d)
finally:
queue.put(gd)


Depending on the application needs this may or may not be more clear
than acquiring/releasing a lock around access to a global variable. In
either case I strongly suggest encapsulating all access to the shared
resource/data structure in functions that manage the lock or queue,
and putting all of that in a module, so you aren't scattering lock
acquire/release or queue get/put all over your code.

Greg Jorgensen

[Kelly Black]

Thanks Greg!

Kelly