DB Connection as Singleton

[wsantos]

Im using motor, and put the db in a base handler to get db for all
RequestHandler, but i want the connection system wide. how i can
accomplish this ? can i get application from server ? is ok to have
the db in options singleton ?



regards.

[A. Jesse Jiryu Davis]

Put this at module scope:

db = motor.MotorClient().my_database

You could stick that in a db.py and then "from db import db" wherever you want it. Or, put it in the application:

tornado.web.Application(urls, db=db)

Then every RequestHandler gets the same db instance in self.settings['db'].

--
You received this message because you are subscribed to the Google Groups "Tornado Web Server" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-tornad...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Ben Darnell]

On Sat, Jun 8, 2013 at 11:19 AM, A. Jesse Jiryu Davis <je...@emptysquare.net> wrote:

Put this at module scope:

db = motor.MotorClient().my_database

You could stick that in a db.py and then "from db import db" wherever you want it. Or, put it in the application:

tornado.web.Application(urls, db=db)

Then every RequestHandler gets the same db instance in self.settings['db'].

Any of these will work.  The application setting is probably the easiest, although personally I tend to make an explicit singleton like IOLoop.instance().  Wherever you put the singleton, I recommend accessing it through a method or property so you can change it easily if you need to (e.g. https://github.com/facebook/tornado/blob/master/demos/blog/blog.py#L67-69).

There are some caveats with a singleton database connection.  The first of course is that it's generally not safe to use the singleton object from multiple threads.  A related but more subtle issue is that you can't do anything asynchronous during a transaction, or else something else might use the connection while you're waiting for the asynchronous result.  This means you must commit any transaction during the same callback that started it, and you can't yield from a coroutine during the transaction.  Finally, if you have tornado fork multiple processes, the connection must not be created until after the fork.  

-Ben

[A. Jesse Jiryu Davis]

Agreed. Motor is not thread-safe, and if you fork, MotorClients must be created after the fork. OTOH we don't have transactions in MongoDB so that's not an issue. =)

[Yutong Zhao]

Sorry to necro a thread, redis-py claims to be thread safe (https://github.com/andymccurdy/redis-py#thread-safety):


"Redis client instances can safely be shared between threads. Internally, connection instances are only retrieved from the connection pool during command execution, and returned to the pool directly after. Command execution never modifies state on the client instance."


Does this mean I can safely initialize a redis client before the fork, and use the same client across all my child processes?

[Ben Darnell]

On Tue, May 27, 2014 at 9:05 PM, Yutong Zhao <prot...@gmail.com> wrote:

Sorry to necro a thread, redis-py claims to be thread safe (https://github.com/andymccurdy/redis-py#thread-safety):


"Redis client instances can safely be shared between threads. Internally, connection instances are only retrieved from the connection pool during command execution, and returned to the pool directly after. Command execution never modifies state on the client instance."


Does this mean I can safely initialize a redis client before the fork, and use the same client across all my child processes?

Thread-safety and fork-safety are entirely separate issues.  Thread-safety usually doesn't matter much in Tornado because everything is multiplexed onto one thread.  Things that open a persistent client network connection are very rarely fork-safe, so I would not expect to be able to initialize a redis client before the fork.

-Ben

 

For more options, visit https://groups.google.com/d/optout.

[Eric B]

This peaqued my interest, just wanted to share further educational material: http://linuxjournal.com/content/three-ways-web-server-concurrency

[Yutong Zhao]

Interesting, thanks! It turns out the ConnectionPool object in redis-py is also fork-safe. (check rev. 2.4.12 https://github.com/andymccurdy/redis-py/blob/master/CHANGES). No wonder things haven't blown up yet.