Making a connection class of redis.py for Tornado

[Chris]

The python redis client, redis.py, uses hiredis for parsing, which gives itself considerable performance advantage over some other async redis libraries.

However, redis.py is blocking, which goes against Tornado's design. To make it work better with Tornado, the author of redis.py has suggested something on the redis google group:

To make redis-py work well with Tornado, I believe all you'd have to do is implement your own Connection class, using whatever async socket calls Tornado provides. Take a look at redis-py/redis/connection.py. With redis-py, you can easily tell the client what connection class you want to use in your app by passing the "connection_class" argument to either a ConnectionPool, or to the base Redis class itself.

However, it seems like no one has implemented this. Can someone point some direction about how we can do this for Tornado? 

 

[Serge S. Koval]

There's toredis, which uses hiredis parser. I made small fork of original toredis with some bug fixes and disabled connection pooling, as I needed PUB/SUB: https://github.com/mrjoes/toredis/

There are no examples and documentation and it follows original redis spec for most of the commands. If there's need for docs/examples, I can make some.

Serge.

[Chris]

Is it easy to switch the connection pooling back on? Or make it configurable? At the moment, I don't need PUB/SUB, so connection pooling would be nice!

Some small examples of using it would be great, as I find hardly any in the original toredis repository. Thanks.

[Serge S. Koval]

It depends why you need connection pooling.

If you're using redis blocking operations, like BLPOP, you still need separate connection, which you can create and use it for that operation.

If you're not using blocking operations, you just use one connection for your application.

I will add pooling on top, but don't have any deadline.

Small example how to use: https://github.com/mrjoes/toredis/blob/master/tests/test_handler.py

Serge.

[Chris]

It depends why you need connection pooling.

Hi serge, i guess you're the best person to confirm this: i think connection pooling would benefit multiple sockjs-tornado clients. In other words, instead of having one redis connection for each connected client, they share a connection pool. Would that work?

[Serge S. Koval]

Hi,

Redis is single-threaded, it can't run more than one operation at the same time. So, if you only use operations that return results immediately (GET, SET, PUBLISH, etc), it does not matter if you open one connection to redis or use connection pool.

However, it makes difference if you have redis cluster and distribute requests between two servers. For this, connection pool will be more efficient, as it will manage connections internally and you don't have to worry about managing them yourself.

If you use blocking operations, SUBSCRIBE, etc - you will need per-client redis connection.

To summarise: it is perfectly fine to open one redis connection to serve all SockJS clients as long as they don't use any blocking commands (BLPOP, BRPOP, SUBSCRIBE, etc). 

Serge.

[Chris]

To summarise: it is perfectly fine to open one redis connection to serve all SockJS clients as long as they don't use any blocking commands (BLPOP, BRPOP, SUBSCRIBE, etc). 

Ok, as I am only using commands like sadd, hset etc, and pipelining (I don't think that is blocking), I think i'm good to use a single redis connection for all connected sockjs clients. To confirm, this means that I create a class-level variable such as 'redis_conn' inside my SockJSConnection class, which will then be shared by all connected SockJS clients afterwards.

[Andrew Grigorev]

Hi!

I work on a more advanced version of the toredis, with a proper
connection pooling and pythonic API: https://github.com/ei-grad/toredis

Usage example: https://bitbucket.org/eigrad/passage/src/master/models.py

It also uses a monkey-patching to add a @task decorator to the
toredis.commands.RedisCommandsMixin methods. Equivalent decorator looks
to be added into the 3.X.X version of tornado 3, and I'll add it to the
toredis source code in future. So I suggest you to use it too.

[Andrew Grigorev]

01.11.2012 18:39, Serge S. Koval пишет:

> If there's need for docs/examples, I can make some.

It would be cool.