new redis_tools, new redis-backed scheduler

[Niphlod]

Once PR1158 gets merged web2py will have a preferred way to use redis. It's through the new gluon.contrib.redis_utils "RConn".

Some redis libraries are coming along and redis-py, while maintaining a solid "#1 position" among them, could be one day in #2.

Moreover, connection parameters passed to redis have been added and it's been almost impossible to keep web2py's modules that use redis in sync. It'd have meant to lock a particular release of redis-py to a particular release of web2py's contrib redis_* modules.

Seen that and issue#958, a refactor was needed. Better late than ever, the refactor is here.

This bring BREAKING changes for the next version of web2py. Anyone using gluon.contrib.redis_cache and/or gluon.contrib.redis_session will need to alter app's code to match the new behaviour.

Basically now every module in web2py that uses redis needs (and should, for new modules you may want to contrib) to have a redis_conn parameter that takes a redis instance. That redis instance can be provided out of the box - tuned for the unique environment of web2py - with

from gluon.contrib.redis_utils import RConn

rconn = RConn()

now, RConn takes ALL parameters and passes them unchanged down the pipe directory to redis.StrictRedis .

This means that any parameter that was used to connect to redis for those modules isn't there anymore.

Specifically, if you had a redis instance available at 10.0.0.10 on port 5555, your instantiation of RedisCache was similar to

from gluon.contrib.redis_cache import RedisCache

cache.redis = RedisCache('10.0.0.10:5555',debug=True)

from here on, it'll be instead

from gluon.contrib.redis_utils import RConn

from gluon.contrib.redis_cache import RedisCache

rconn = RConn(host='10.0.0.10', port=5555)

cache.redis = RedisCache(redis_conn=rconn,debug=True)

want to connect to redis through ssl (previously unavailable)? pass the relevant ssl, ssl_keyfile, etc etc etc to RConn.

You can also reuse the rconn object as if it was a redis.StrictRedis instance in your app's code. Everything will work fine. Just use a sane key prefix (all modules use "w2p") in order to avoid conflicts. redis_cache, redis_session and redis_scheduler (and your app) can use a single RConn instance without issues.

Was this only a refactor ? Nope!

There is also a new (not so shiny, 'cause - guilty as charged - I'm using it for 10 months in production now) redis-backed scheduler. 

Of course it's experimental (as everything in contrib) but it's not untested (all w2p_scheduler_tests passed). 

This is one of the first modules completely coded on Windows (sigh). Linux has been tested too through w2p_scheduler_tests (still waiting for someone to come up with unittests that will run on our CI) but the production environment kicking for 10 months is WS2012R2 with redis 3.0 .

RScheduler is a slip-in of the factory scheduler, with the - usual - redis_conn parameter.

from gluon.contrib.redis_utils import RConn

from gluon.contrib.redis_scheduler import RScheduler

rconn = RConn()

mysched = RScheduler(db, ..., redis_conn=rconn)

This first release basically moves everything happening in the scheduler_worker table to redis, to alleviate locking and contention where it happens the most. 

It'll probably enable usage patterns for who had problems in production with a high number of workers. It's also a lot snappier because with redis tasks gets from QUEUED to RUNNING without passing from ASSIGNED. Use it, test it, break it, fix it (or report bugs). Read the source code, improve it, move ideas around.... just crunch tasks...do whatever you want with it. Hope you'll find it as useful as I am. 

[José Luis Redrejo]

This is really interesting, specially the scheduler and the possibility of sharing the redis connection.

Is there any ETA to have it in stable?

Thanks Niphlod

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups "web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web2py+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Niphlod]

was just merged in trunk.

[Lisandro]

Hi there.

Is there any working example using ssl to connect to redis?

My web2py app is served through HTTPS since some time ago (using an old web2py version). 

Today I've tried to update web2py to the last stable version, and I've noticed all this awesome improvements to redis cache. 

However, the way to use it has changed, so I need to modify my code. I've been able to get it to work **without** https. But when I use https, my app just can't connect to redis.

I've seen that RConn class receives several arguments regarding ssl. But I'm not sure how to use them.

If my app is served through HTTPS, do I need to pass those parameters to RConn? Do I need to generate those certificates locally? 

If that's the case, is there any example that can help me?

Thanks in advance!

Regards, 

Lisandro

[Niphlod]

first and foremost, ssl connection with redis and https have absolutely no relation.

you can get every combination of the matrix:

- http with std redis

- http with ssl redis

- https with std redis

- https with ssl redis

Redis server doesn't offer anything as ssl is concerned. Redis service providers though use something like stunnel to create an encrypted channel to connect to redis, and redis-py (the official, underlying library) supports it.

What you can pass to redis-py you can now pass to RConn (so, if you want to connect to redis through an ssl channel because your service provider require you to do so, you can pass ssl_keyfile,ssl_certfile,ssl_cert_reqs,ssl_ca_certs, etc etc etc)