Re: me confused! Multiple redis instances, one redis-py connection pool?

[Josiah Carlson]

You are missing a few things:

* The default connection pool does not really support sharding. You need to subclass redis.ConnectionPool to either take a shard_hint argument, or to do its calculations internally on the passed "keys" argument. (I don't know if it's documented)

* When you do subclass it, you're going to need to be careful with managing your different connections to the different shards, as none of that is built for you

* By default, connections aren't actually created until you need them

* If you think that you're going to be able to detect when a Redis connection dies so as to do multi-machine concurrent writes, you're going to have a bad time[1]

[1] Multi-machine commit is a PITA to get right (made worse by Redis' inability to rollback partial changes). Heck, I'm not sure that I could get it right the first time without spending several hours reading articles before starting.

Regards,

 - Josiah

On Sat, Jun 8, 2013 at 6:37 PM, BigFanOfAnonymity <mali...@gmail.com> wrote:

Hi there,

I have an application that requires multiple redis instances for presharding.  Data will be distributed evenly by a simple hash.  There will be a couple operations on the database that require several instances which I plan to execute asynchronously with gevent, and a couple more that will only involve one instance. The data is only useful as a whole, so if one instance becomes unavailable (in the process of failing over) everything needs to stop until it is available again.

I've done a lot of reading, but I am still confused about what a ConnectionPool is for.  Most examples seem to use a single connection pool to connect to multiple DBs but not multiple instances.  Working along the lines that I would be subclassing the connectionpool I tried writing a quick python script like this:

class HashedConnectionPool(redis.ConnectionPool):
        pass

pool = HashedConnectionPool(max_connections=4)

r1 = redis.Redis(connection_pool=pool, port=6379)
r2 = redis.Redis(connection_pool=pool, port=6380)
r3 = redis.Redis(connection_pool=pool, port=6381)
r4 = redis.Redis(connection_pool=pool, port=6382)

.. But after this is done, _available_connections within the ConnectionPool is still empty.  What am I missing here?  Thanks!

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.