Tornado and redis clients --- How to make redis-py to work in a non blocking manner

[David Montgomery]

Hi,

I am using tornando and redis with the torandoredis client from
https://github.com/leporo/tornado-redis/tree/master/tornadoredis.

The client is painfully slow due to the fact that no tornado clients
for redis use hiredis. redis-py is way faster but is blocking. Are
there any tricks that I can use such as gevent threads or something
like that will allow me to use redis-py but yet not be blocking?

Below is my typical code for redis calls. How would I modify to use
gevent threads for redis-py?


class MyHandler(tornado.web.RequestHandler):
@tornado.web.asynchronous
@tornado.gen.engine
def get(self):
self.push='test'
yield tornado.gen.Task(r_main_writes.lpush,'global',self.push)
#most recent time user has seen add
self.set_header('Content-Type', 'test/plain')
self.write('test')
self.finish()

Thanks

[Jon Parise]

On Mon, Jun 4, 2012 at 7:49 PM, David Montgomery
<davidmo...@gmail.com> wrote:

> I am using tornando and redis with the torandoredis client from
> https://github.com/leporo/tornado-redis/tree/master/tornadoredis.
>
> The client is painfully slow due to the fact that no tornado clients
> for redis use hiredis.  redis-py is way faster but is blocking.  Are
> there any tricks that I can use such as gevent threads or something
> like that will allow me to use redis-py but yet not be blocking?

If you're otherwise happy with tornadoredis (aside from its parsing
speed), maybe you could take a swing at adding hiredis support to it.
That feels like a more direct (and perhaps "easier") approach to your
particular performance problem.

[Didip Kerabat]

I've deployed several production app using blocking redis-py. One of the app serves full HTML cache from it. redis-py is so much, ridiculously, faster than MySQL I no longer care about it.

Try profile your model calls with hotshot, you will see 99 problems but redis-py won't be one of them.

Another perspective, when tornado first came out, FriendFeed didn't even care of making asynchronous MySQL client. I'd say don't worry about it.

- Didip -

Sent from my iPad

[David Montgomery]

Yes....I might have to and will def enter a new frontier by doing
so.... least for me. It is confirmed by another developer that adding
hiredis improves performance but not production quality. But would
like to exhaust all options before doing so.

[Frank Smit]

I'm currently working on a Redis client that'll use C to assemble and
parse data. It's not finished yet, I'm currently working on reply
parsing. Maybe you find it interesting. :)

https://github.com/FSX/akane

[David Montgomery]

My hero! When do you expect it to be stable? I am very eager to use it.

Aloha,

David

[David Montgomery]

Also, I use a password for redis. Will the connection allow for auth?

On Tue, Jun 5, 2012 at 3:58 PM, Frank Smit <fr...@61924.nl> wrote:

[Frank Smit]

Auth isn't implemented yet. Feel free to make a ticket to ensure I
don't forget it. ;)

I don't have a date set for a stable release, because the time I can
spend on it is not stable. The "roadmap" is as follows:

1. C-based reply parser. (in progress)
2. Implement commands and unit tests. (in progress)
3. Implement pipelining.

I haven't looked into pub/sub, Lua scripting and transactions, but
those will be included too.


On Tue, Jun 5, 2012 at 10:14 AM, David Montgomery

[Arek Bochinski]

@Didip

"Try profile your model calls with hotshot, you will see 99 problems but redis-py won't be one of them."

-this was one of the best comments I've read on this list. *brushes off shoulder* ^_^

[David Montgomery]

I have tried all redis clients for tornado and cyclone and for my
needs way too slow. Main culprit is not using hiredis. I have a c10K
problem and my requirement is to server a response in less than 120
ms. A blocking redis-py is faster than a non blocking burkova.

So..I was hoping for some trick where I can use a blocking call in a
non blocking means....if that makes sense at all....

Thanks

On Fri, Jun 8, 2012 at 8:34 AM, Joseph Fouad <faw...@gmail.com> wrote:
> May be you can use brokva :) , it is a bit old, but it works fine for me

[Carl S. Yestrau Jr.]

If you are really concerned with blocking penalties maybe consider
webdis - https://github.com/nicolasff/webdis - as an async http bridge
for redis.

On Mon, Jun 4, 2012 at 7:49 PM, David Montgomery
<davidmo...@gmail.com> wrote: