Redis client for tornado

[Andrew Grigorev]

Hello, the Tornado community :-).

I just looked at tornado-redis code, and terrified:

1. It use blocking function socket.connect to establish TCP connection
to the Redis server.
2. It's recommendded (as followed from demos) to use a single db
connection for all requests. It is terrible if you using redis
transactions in your code. And from other side, it is bad to make a new
connection for every request, even if we didn't had to have the problem
with blocking "connect" function.
3. It goes wrong way by defining statically all redis commands in Client
class. This is ugly and not natural for such a dynamic language, as Python.

There are a good points, how to fix that problems in other redis library
for tornado:

1. Use the tornado IOStream correctly.
2. Implement a connection pooling mechanism, which (in trivial case)
would delegate a separate redis connection to every request or (in the
ideal case) would monitor the usage of transactions and using redis
request pipelining persistently for single requests, switching between
free established connections in round-robin.
3. Use the metaprogramming techniques to populate the Client class with
all redis commands, having proper docstrings, processing arguments and
return values correctly.

So, here I go. Join to me, if you like to use redis in tornado. It is
probably the best storage to use with it, as it is simple and fast,
exactly like the tornado :-).

The interesting thing - I started to implement all of these from
scratch, and named the repository "toredis". Then, when I thought that
it is time to push the initial code to the Github, I accidently found
the existing toredis project by @joshmarshall on github, which already
partically applies to specifed points :-). Thanks, Josh. Do you have any
plans to evolve the toredis?

The original toredis repo: https://github.com/joshmarshall/toredis
My toredis repo: https://github.com/ei-grad/toredis

Comments, suggestions and other participation in toredis development are
welcome :-).

PS, join to the redis-db@ discussion
https://groups.google.com/forum/?fromgroups=#!topic/redis-db/WwIWZkfKf7A:

-------- Исходное сообщение --------
Тема: OBJECT and SHOWLOG commands in commands.json and docs
Дата: Tue, 25 Sep 2012 20:57:46 +0400
От: Andrew Grigorev <and...@ei-grad.ru>
Кому: redi...@googlegroups.com



Hello.

I am working on toredis, a redis client for tornado, which using
metaprogramming to construct the Client class from commands.json file. I
have a several questions about redis docs and that file:

1. Is it a good idea to use it for redis clients in dynamic languages?
What is the status of that file?

2. I want to add the "return" field to commands.json commands, is this
reasonable?

3. Don't you think, that it is strange to have a single command for
OBJECT REFCOUNT|ENCODING|IDLETIME subcommands in docs, while there are
different DEBUG OBJECT|SEGFAULT commands? The second approach looks
better for me. Also, there is a SHOWLOG command, joining three
subcommands in one, too. Are there any chances for pull-request,
splitting OBJECT command and SHOWLOG command to different commands, to
be merged in redis-docs? If not, then what if them would be splitted at
least in commands.json? Basically, this question is a blocker for the
second question, because subcommands of that commands return different
values.

Thank you for your attention.

--
Andrew

[Frank Smit]

Hello Grigorev,

Interesting to see that you generate the methods from a JSON file.
Saves a lot of work. I'm curious how it's going to work out with all
the Redis commands. For example: some commands accept key/value pairs.
The nicest way to throw that data to the command in the form of a
dictionary and not a list. Anyway, looks interesting!

I've been (on and off) working on a Redis client myself too. It has
connection pooling and a tiny bit of C. :)
https://github.com/FSX/akane

Regards,
Frank

[Shane Spencer]

Keep in mind you can also use webdis.io and the AsyncHTTPClient.

- Shane

[Andrew Grigorev]

HTTP gives a huge overhead comparing to Redis protocol, which in fact is
much simplier then constructing resource URLs for webd.is. Using webd.is
is normal for JavaScript applications in browser, but it is bad idea to
use redis over http from tornado. You could better then use raw
IOStream+Redis, then AsyncHTTPClient+Webd.is :-).

25.09.2012 23:56, Shane Spencer О©╫О©╫О©╫О©╫О©╫:

>>> -------- О©╫О©╫О©╫О©╫О©╫О©╫О©╫О©╫ О©╫О©╫О©╫О©╫О©╫О©╫О©╫О©╫О©╫ --------
>>> О©╫О©╫О©╫О©╫: OBJECT and SHOWLOG commands in commands.json and docs
>>> О©╫О©╫О©╫О©╫: Tue, 25 Sep 2012 20:57:46 +0400
>>> О©╫О©╫: Andrew Grigorev <and...@ei-grad.ru>
>>> О©╫О©╫О©╫О©╫: redi...@googlegroups.com

>>>
>>>
>>>
>>> Hello.
>>>
>>> I am working on toredis, a redis client for tornado, which using
>>> metaprogramming to construct the Client class from commands.json file. I
>>> have a several questions about redis docs and that file:
>>>
>>> 1. Is it a good idea to use it for redis clients in dynamic languages?
>>> What is the status of that file?
>>>
>>> 2. I want to add the "return" field to commands.json commands, is this
>>> reasonable?
>>>
>>> 3. Don't you think, that it is strange to have a single command for
>>> OBJECT REFCOUNT|ENCODING|IDLETIME subcommands in docs, while there are
>>> different DEBUG OBJECT|SEGFAULT commands? The second approach looks
>>> better for me. Also, there is a SHOWLOG command, joining three
>>> subcommands in one, too. Are there any chances for pull-request,
>>> splitting OBJECT command and SHOWLOG command to different commands, to
>>> be merged in redis-docs? If not, then what if them would be splitted at
>>> least in commands.json? Basically, this question is a blocker for the
>>> second question, because subcommands of that commands return different
>>> values.
>>>
>>> Thank you for your attention.
>>>
>>> --
>>> Andrew
>>>
>>>
>>>
>>>

--
Andrew

[Shane Spencer]

I actually use my own async redis socket functions in most of my
tornado apps.. it's simple and to the point.

I agree that webdis has some overhead but I like that it's a
routable/proxyable solution that dumps out whatever format you want as
well. Very useful IMHO from both javascript and daemon processes.
I'd like to think the overhead would only be substantial after around
0.5 seconds of full throttle benchmarking.

- Shane

On Tue, Sep 25, 2012 at 12:16 PM, Andrew Grigorev <and...@ei-grad.ru> wrote:
> HTTP gives a huge overhead comparing to Redis protocol, which in fact is
> much simplier then constructing resource URLs for webd.is. Using webd.is is
> normal for JavaScript applications in browser, but it is bad idea to use
> redis over http from tornado. You could better then use raw IOStream+Redis,
> then AsyncHTTPClient+Webd.is :-).
>

> 25.09.2012 23:56, Shane Spencer пишет:

>>>> -------- Исходное сообщение --------
>>>> Тема: OBJECT and SHOWLOG commands in commands.json and docs
>>>> Дата: Tue, 25 Sep 2012 20:57:46 +0400
>>>> От: Andrew Grigorev <and...@ei-grad.ru>

>>>> Кому: redi...@googlegroups.com

[Andrew Grigorev]

Hi Smit, thanks for reply.

Your akane looks interesting - implementing the response parsing in C
could give some boost, I thought to add something like that in toredis
later. IMO probably we should look into the hiredis for such stuff.
Can't determine on the first look, does akane use its code or not...
Does it?

About argument/response parsing - I believe, it could be neatly
implemented in Python using metaprogramming, but I didn't have time to
look in that subject yet. But I am sure, it would be pythonic. To be
more specific, for example, HSET command execution should look as
something like this:

class Handler(web.RequestHandler):

@web.asynchronous
def post(self):
c = redis.Client()
c.hset('key', fieldname='value', callback=self.on_set)
# may be c.hset('key', {'fieldname': 'value'}, callback),
# but not c.hset('key', 'fieldname', 'value',
callback=callback), of course

def on_set(self, response):
self.finish('done')

Also, I thought about implementing a more pythonic blocking API for
Redis (redis-py is a pile of poo too, absolutely like tornado-redis),
based on proxy pattern. Something like this:

>>> c = redis.Client()
>>> c.list('key')[10] = 'value'
>>> print c.list('key')[10] # equal to c.lindex('key', 10)
value
>>> c.hash('key2')['asd']

and even like this:

>>> class ShopItem(redis.Hash):
... pass
...
>>> pony = ShopItem('unicorn-pony')
>>> pony.cost = 100
>>> print redis.hget('ShopItem:unicorn-pony', 'cost')
100
>>> redis.hset('ShopItem:unicorn-pony', color='pink')
>>> print pony.color
'pink'
>>> pony.auto_commit = False
>>> pony.cost = 1000
>>> print redis.hget('ShopItem:unicorn-pony', 'cost')
100
>>> pony.color = 'blue'
>>> pony.commit()
>>> print redis.hmget('ShopItem:unicorn-pony', 'cost', 'color')
[1000, 'blue']

May be such part of the library could be implemented even simplier, than
metaprogramming part. But such API is not supposed to be usable in
Tornado as it is asynchronous, and I am not interested now in
implementing a blocking library for redis. Althought, I do not exclude
the possibility of that some similar API, usable in asynchronous style,
could exist. It only needs to come up :-). Does anyone has any ideas?
But in any case, asynchronous execution of all Redis commands in
pythonic way have to be implemented first.

25.09.2012 23:38, Frank Smit пишет:

--
Andrew

[Andrew Grigorev]

Forgot to write about connection pool.

Your implementation looks good, probably I'll steal some bits of its
code :-). Thanks.

26.09.2012 01:16, Andrew Grigorev пишет:

[Andrew Grigorev]

26.09.2012 01:16, Andrew Grigorev пишет:

> Hi Smit, thanks for reply.
>
> Your akane looks interesting - implementing the response parsing in C
> could give some boost, I thought to add something like that in toredis
> later. IMO probably we should look into the hiredis for such stuff.
> Can't determine on the first look, does akane use its code or not...
> Does it?
>
> About argument/response parsing - I believe, it could be neatly
> implemented in Python using metaprogramming, but I didn't have time to
> look in that subject yet. But I am sure, it would be pythonic. To be
> more specific, for example, HSET command execution should look as
> something like this:
>
> class Handler(web.RequestHandler):
>
> @web.asynchronous
> def post(self):
> c = redis.Client()
> c.hset('key', fieldname='value', callback=self.on_set)
> # may be c.hset('key', {'fieldname': 'value'}, callback),
> # but not c.hset('key', 'fieldname', 'value',
> callback=callback), of course
>

Hm. Although perhaps then I got a little excited. "c.hset('key',
'fieldname', 'value')" is acceptable form, probably. I somehow switched
to HSET from HMSET in my mind :-). I want to say that HMSET, of course,
should accept a dict parameter, or use **kwargs values. Probably even
just a single dict is better.

[Frank Smit]

Yes, it uses hiredis for reply parsing, but I want to make my own
reply parser some time. Just to see if it works out. :)