The python redis client, redis.py, uses hiredis for parsing, which gives itself considerable performance advantage over some other async redis libraries.
However, redis.py is blocking, which goes against Tornado's design. To make it work better with Tornado, the author of redis.py has suggested something on the redis google group:
To make redis-py work well with Tornado, I believe all you'd have to do is
> implement your own Connection class, using whatever async socket calls > Tornado provides. Take a look at redis-py/redis/connection.py. With > redis-py, you can easily tell the client what connection class you want to > use in your app by passing the "connection_class" argument to either a > ConnectionPool, or to the base Redis class itself.
However, it seems like no one has implemented this. Can someone point some direction about how we can do this for Tornado?
There's toredis, which uses hiredis parser. I made small fork of original
toredis with some bug fixes and disabled connection pooling, as I needed
PUB/SUB: https://github.com/mrjoes/toredis/
There are no examples and documentation and it follows original redis spec
for most of the commands. If there's need for docs/examples, I can make
some.
On Thu, Nov 1, 2012 at 4:28 PM, Chris <bhp...@gmail.com> wrote:
> The python redis client, redis.py, uses hiredis for parsing, which gives
> itself considerable performance advantage over some other async redis
> libraries.
> However, redis.py is blocking, which goes against Tornado's design. To
> make it work better with Tornado, the author of redis.py has suggested
> something on the redis google group:
> To make redis-py work well with Tornado, I believe all you'd have to do is
>> implement your own Connection class, using whatever async socket calls
>> Tornado provides. Take a look at redis-py/redis/connection.py. With
>> redis-py, you can easily tell the client what connection class you want to
>> use in your app by passing the "connection_class" argument to either a
>> ConnectionPool, or to the base Redis class itself.
> However, it seems like no one has implemented this. Can someone point some
> direction about how we can do this for Tornado?
Is it easy to switch the connection pooling back on? Or make it configurable? At the moment, I don't need PUB/SUB, so connection pooling would be nice!
Some small examples of using it would be great, as I find hardly any in the original toredis repository. Thanks.
If you're using redis blocking operations, like BLPOP, you still need
separate connection, which you can create and use it for that operation.
If you're not using blocking operations, you just use one connection for
your application.
I will add pooling on top, but don't have any deadline.
On Thu, Nov 1, 2012 at 5:59 PM, Chris <bhp...@gmail.com> wrote:
> Is it easy to switch the connection pooling back on? Or make it
> configurable? At the moment, I don't need PUB/SUB, so connection pooling
> would be nice!
> Some small examples of using it would be great, as I find hardly any in
> the original toredis repository. Thanks.
Hi serge, i guess you're the best person to confirm this: i think connection pooling would benefit multiple sockjs-tornado clients. In other words, instead of having one redis connection for each connected client, they share a connection pool. Would that work?
Redis is single-threaded, it can't run more than one operation at the same
time. So, if you only use operations that return results immediately (GET,
SET, PUBLISH, etc), it does not matter if you open one connection to redis
or use connection pool.
However, it makes difference if you have redis cluster and distribute
requests between two servers. For this, connection pool will be more
efficient, as it will manage connections internally and you don't have to
worry about managing them yourself.
If you use blocking operations, SUBSCRIBE, etc - you will need per-client
redis connection.
To summarise: it is perfectly fine to open one redis connection to serve
all SockJS clients as long as they don't use any blocking commands (BLPOP,
BRPOP, SUBSCRIBE, etc).
On Thu, Nov 1, 2012 at 6:44 PM, Chris <bhp...@gmail.com> wrote:
> It depends why you need connection pooling.
> Hi serge, i guess you're the best person to confirm this: i think
> connection pooling would benefit multiple sockjs-tornado clients. In other
> words, instead of having one redis connection for each connected client,
> they share a connection pool. Would that work?
> To summarise: it is perfectly fine to open one redis connection to serve > all SockJS clients as long as they don't use any blocking commands (BLPOP, > BRPOP, SUBSCRIBE, etc).
Ok, as I am only using commands like sadd, hset etc, and pipelining (I don't think that is blocking), I think i'm good to use a single redis connection for all connected sockjs clients. To confirm, this means that I create a class-level variable such as 'redis_conn' inside my SockJSConnection class, which will then be shared by all connected SockJS clients afterwards.
It also uses a monkey-patching to add a @task decorator to the toredis.commands.RedisCommandsMixin methods. Equivalent decorator looks to be added into the 3.X.X version of tornado 3, and I'll add it to the toredis source code in future. So I suggest you to use it too.
