pymongo + cherrypy + multithreading

[marc]

hi guys,

i've recently recognized, that using pymongo with cherrypy and
multithreading (about 10 threads) results in strange failures.

e.g. this error occurs using python and gridfs. running the same code
in a single thread solves the problem, but shouldn't be a solution
because i need cherrypy running with multiple threads.

--- code ---
import gridfs
from pymongo.connection import Connection


db = Connection().test
fs = gridfs.GridFS(db)

file = fs.open('test.txt', 'w', 'files')
file.content_type = 'text/plain'
file.write('hello')
file.close()
--- /code ---

--- snip ---
Thu Jul 2 17:54:06 Assertion failure type() == BinData db/jsobj.h
326
0x100030186 0x1000373f5 0x1000c42c5 0x1000bda1f 0x100060fe7
0x100065bbc 0x1000e2413 0x1000e404c 0x100122736 0x100128314
0x7fff82120e8b 0x7fff82120d4d
0 mongod 0x0000000000030186
_ZN5mongo12sayDbContextEPKc + 246
1 mongod 0x00000000000373f5
_ZN5mongo8assertedEPKcS1_j + 309
2 mongod 0x00000000000c42c5
_ZN5mongo10CmdFileMD53runEPKcRNS_7BSONObjERSsRNS_14BSONObjBuilderEb +
2101
3 mongod 0x00000000000bda1f
_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERSt18basic_stringstreamIcSt11char_traitsIcESaIcEERNS_10BufBuilderERNS_14BSONObjBuilderEbi
+ 2767
4 mongod 0x0000000000060fe7
_ZN5mongo11runCommandsEPKcRNS_7BSONObjERSt18basic_stringstreamIcSt11char_traitsIcESaIcEERNS_10BufBuilderERNS_14BSONObjBuilderEbi
+ 55
5 mongod 0x0000000000065bbc
_ZN5mongo8runQueryERNS_7MessageERSt18basic_stringstreamIcSt11char_traitsIcESaIcEE
+ 5516
6 mongod 0x00000000000e2413
_ZN5mongo13receivedQueryERNS_10DbResponseERNS_7MessageERSt18basic_stringstreamIcSt11char_traitsIcESaIcEEb
+ 515
7 mongod 0x00000000000e404c
_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERK11sockaddr_in
+ 2092
8 mongod 0x0000000000122736
_ZN5mongo10connThreadEv + 710
9 mongod 0x0000000000128314
thread_proxy + 132
10 libSystem.B.dylib 0x0000000082120e8b
_pthread_start + 316
11 libSystem.B.dylib 0x0000000082120d4d
thread_start + 13
--- snap ---

i've also tried using the pymongo connection with a poolsize of 10 and
registering a thread-wide global connection. but this doesn't help.
maybe pymongo is not threadsafe or i'm too stupid using it :)

do you have any solutions running pymongo in multithreaded
environments?

thanks in advance,

cheers marc

[Voltron]

Hmm thats strange, I never noticed this problem. I am using Mongodb
also with CherryPY, but not using GridFS. I asked the same question a
while back and I was assured that the driver is thread - safe

[marc]

hi voltron,

thanks for your answer. good to hear that it works for you with
cherrypy. maybe i have sth. in my code that interferences with
mongodb.i'll have closer look at it.

enjoy your weekend,
cheers marc

[Michael Dirolf]

the PyMongo driver should be completely thread-safe. i've done a bit
of testing to see if i could reproduce an error while using gridfs in
a threaded environment, but haven't managed to break anything yet.

if you're able to get a reproducible test case that fails that would
be great.

[Michael Dirolf]

I spoke too soon - looks like I can reproduce this threading issue
with gridfs. Thanks for the report!

[Michael Dirolf]

just pushed a fix that should make gridfs thread-safe. please let us
know if there are more issues with this.

thanks again,
mike

[marc]

hi mike,

thanks for your investigations and the fix. unfortunately this hasn't
fixed the issue. my setup where i use gridfs is a bit complicated.

i've constructed a testcase with the setup i'm using in my
environment:

this is how it works. cherrypy accepts a request, executes serverside
javascript code and returns the result. the javascript has access to
mongodb through a wrapper function. jscore and spidermonkey are to
show that both js engines have problems with mongodb. to run the
testcase, please install cherrypy, pyjscore and python-spidermonkey.
sorry for the inconvenience :)

* http://code.google.com/p/pyjscore/
* http://github.com/davisp/python-spidermonkey/tree/master

if you are using mac os x and python 2.6, here are the precompiled js
engine modules i've uploaded:
http://rapidshare.com/files/252952385/jscore.tar.bz2.html
http://rapidshare.com/files/252952571/spidermonkey.tar.bz2.html

(yeah rapidshare hosts open source warez :P)

here is the code: http://pastebin.com/f3e2cf774

cherrypy will open a server on 127.0.0.1:8000. there are 5 tests
included.

http://localhost:8000/test_jscore -> eval javascript using jscore and
mongodb
http://localhost:8000/test_jscore_regular -> demo to show that jscore
is capable of multithreading

http://localhost:8000/test_spidermonkey -> eval javascript using
spidermonkey and mongodb
http://localhost:8000/test_spidermonkey_regular -> demo to show that
spidermonkey is capable of multithreading

http://localhost:8000/test_native -> run mongodb test without
javascript

both, test_jscore and test_spidermonkey freeze my cherrypy instance
with more than one concurrent request. to test the setup i'm using ab:

ab -c 5 -n 1000 http://127.0.0.1:8000/test_jscore -> freezes
ab -c 5 -n 1000 http://127.0.0.1:8000/test_jscore_regular -> works
ab -c 5 -n 1000 http://127.0.0.1:8000/test_spidermonkey -> freezes
ab -c 5 -n 1000 http://127.0.0.1:8000/test_spidermonkey_regular ->
works
ab -c 5 -n 1000 http://127.0.0.1:8000/test_native -> works
etc...

thank you very much in advance, you guys rock!

cheers marc

[Michael Dirolf]

i've spent a while looking into this issue. i think that the issue
might be that the JS libraries aren't threadsafe, or at least not when
they're used in this way. here is an example that doesn't use pymongo
or cherrypy that deadlocks:
http://mongodb.pastebin.com/m2ecdfebb

thoughts?

[marc]

hi mike,

thanks for getting into this issue. i'm sorry that i've thought
mongodb was the reason for the deadlock. i've forwarded your testcase
to the python-spidermonkey developer.

cheers marc

On Jul 7, 11:06 pm, Michael Dirolf <m...@10gen.com> wrote:
> i've spent a while looking into this issue. i think that the issue  
> might be that the JS libraries aren't threadsafe, or at least not when  
> they're used in this way. here is an example that doesn't use pymongo  
> or cherrypy that deadlocks:http://mongodb.pastebin.com/m2ecdfebb
>
> thoughts?
>
> On Jul 7, 2009, at 6:10 AM, marc wrote:
>
>
>
>
>
> > hi mike,
>
> > thanks for your investigations and the fix. unfortunately this hasn't
> > fixed the issue. my setup where i use gridfs is a bit complicated.
>
> > i've constructed a testcase with the setup i'm using in my
> > environment:
>
> > this is how it works. cherrypy accepts a request, executes serverside
> > javascript code and returns the result. the javascript has access to
> > mongodb through a wrapper function. jscore and spidermonkey are to
> > show that both js engines have problems with mongodb. to run the
> > testcase, please install cherrypy, pyjscore and python-spidermonkey.
> > sorry for the inconvenience :)
>
> > *http://code.google.com/p/pyjscore/

> > *http://github.com/davisp/python-spidermonkey/tree/master

>
> > if you are using mac os x and python 2.6, here are the precompiled js
> > engine modules i've uploaded:
> >http://rapidshare.com/files/252952385/jscore.tar.bz2.html
> >http://rapidshare.com/files/252952571/spidermonkey.tar.bz2.html
>
> > (yeah rapidshare hosts open source warez :P)
>
> > here is the code:http://pastebin.com/f3e2cf774
>
> > cherrypy will open a server on 127.0.0.1:8000. there are 5 tests
> > included.
>

> >http://localhost:8000/test_jscore-> eval javascript using jscore and
> > mongodb
> >http://localhost:8000/test_jscore_regular-> demo to show that jscore
> > is capable of multithreading
>
> >http://localhost:8000/test_spidermonkey-> eval javascript using
> > spidermonkey and mongodb
> >http://localhost:8000/test_spidermonkey_regular-> demo to show that

> > spidermonkey is capable of multithreading
>

> >http://localhost:8000/test_native-> run mongodb test without

> > javascript
>
> > both, test_jscore and test_spidermonkey freeze my cherrypy instance
> > with more than one concurrent request. to test the setup i'm using ab:
>

> > ab -c 5 -n 1000http://127.0.0.1:8000/test_jscore-> freezes
> > ab -c 5 -n 1000http://127.0.0.1:8000/test_jscore_regular-> works
> > ab -c 5 -n 1000http://127.0.0.1:8000/test_spidermonkey-> freezes
> > ab -c 5 -n 1000http://127.0.0.1:8000/test_spidermonkey_regular->
> > works
> > ab -c 5 -n 1000http://127.0.0.1:8000/test_native-> works