efficient connection pooling in python

[David Montgomery]

Hi,

In python I currently have about 10 databases that I have to connect
to in python.

The below is my naive approach.

r_segment = redis.StrictRedis(host='localhost', port=6379, db=0)
r_creative = redis.StrictRedis(host='localhost', port=6379, db=1)

r_segment.set('a','b')
r_creative.set('c','d')

I understand that there is a concept of connection pooling that looks
something like this.

pool = redis.ConnectionPool(host='localhost', port=6379, db=0)
r = redis.Redis(connection_pool=pool)

So..how does the python code look like to efficiently connect to the
databases and to select among them as efficiently as possible?

Thanks

[Josiah Carlson]

Create a pool for each of your database configurations, and stuff them
into some module...

# redis_connections.py
import redis

testing = redis.ConnectionPool(..., db=15)
queues = redis.ConnectionPool(..., db=1)
cache = redis.ConnectionPool(..., db=3)

# end redis_connections.py

Then when you want a connection of a particular type, you just use:
conn = redis_connections.<type>.get_connection(None)
When you are done:
redis_connections.<type>.release(conn)

If you feel like making it a bit more explicit, you can use a context manager:

# begin python code
import contextlib

import redis_connections

@contextlib.contextmanager
def redis_conn(config):
pool = getattr(redis_connections, config)
conn = pool.get_connection(None)
yield conn
pool.release(conn)

# end python code

That will generally work pretty well.

Because I usually have a few threads going, because the connection
pooling didn't exist when I was first using Python with Redis, and
because I don't trust myself to explicitly release every connection, I
actually create a new connection to every database from every thread,
and use an instance of threading.local() to keep it all separate. I
then have function called get_connection(config) to get the right
connection, but I use it via a combination decorator/context manager
everywhere.

Regards,
- Josiah

[Josiah Carlson]

I forgot to describe how you use the context manager, if you aren't
familiar with them:

with redis_conn('testing') as conn:
# do stuff with your connection
# out here, it's already been released back into the pool

Regards,
- Josiah

[Andy McCurdy]

This is actually more complicated than necessary. redis-py client objects and connection pools are designed to be threadsafe. And creating a client instance without explicitly specifying a connection pool creates one for you by default. Therefore, all you need to do is:

# redis_connections.py

cache = redis.StrictRedis(..., db=1)

testing = redis.StrictRedis(..., db=2)

...

# end redis_connections.py

# Then if you application:

import redis_connections

redis_connections.cache.set('foo', 'bar')

...

redis_connection.testing.get('bing')

--

You received this message because you are subscribed to the Google Groups "Redis DB" group.

To post to this group, send email to redi...@googlegroups.com.

To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.