How to combine Redis and and RDBMS in a website

[SN]

Hello all

I am currently designing a new website and are considering how to best
use redis. I have read that in theory you could completly replace an
RDBMS with Redis but that you probably should not.

Consider the following very rough specs of a website.

User registration + login/logout
users join discussion topics
users can make twits on discussion topics
users can look at activity reports of the discussions

If I want to make sure that a site like this will be massively
scalable, what are the best practices for combining Redis with an
RDBMS ?

1) Should some parts of data go in RDBMS and some in Redis - or
should Redis be used purely as a cache ? Are there any use cases
available to get inspiration from and/or do people have some
experiences about how they combine RDBMS and Redis ?

2) Another scenario could be to store everything in Redis and also
store all data in a RDBMS. Would it be good approach be to use some
message queue when making the store operations in RDBMS so that the
slow writing to RDBMS can be done in a background process? Are there
any examples of this anywhere ?

best regards SN.

[Demis Bellot]

At mflow.com, we stored all 'live' or transient data in Redis. i.e. things like User Sessions, latest flows/posts/purchases, most popular flows/posts/purchases etc. 

Everything ultimately was persisted on sharded PostgreSQL dbs however we were using Redis as a smart cache/data-view on top, where we were caching by validity (not time) so all reads would be serviced by Redis unless a user made a change to their profile, etc in which case we would clear the caches in Redis then.

2) Another scenario could be to store everything in Redis and also

store all data in a RDBMS. Would it be good approach be to use some 

message queue when making the store operations in RDBMS so that the

slow writing to RDBMS can be done in a background process? Are there 

any examples of this anywhere ?

Yeah we also had a Redis MQ implementations that would take 'async one way' web service requests and drop them in a Redis Message Queue which would be later processed by a background process in its own time (i.e. outside the original request boundary). I have an implementation of the C# Redis Message Queue used here:

https://github.com/mythz/ServiceStack.Redis/tree/master/src/ServiceStack.Redis/Messaging with some tests here: http://bit.ly/dJT5pt and http://bit.ly/gsYBDQ 

You should also have a look at Resque which is a robust message queue redis implementation specifically designed for this task: https://github.com/blog/542-introducing-resque (intro)  https://github.com/defunkt/resque#readme (project site)

Now I also think the use-cases of primarily storing data in Redis and then have a back-end process creating a mirror of that data in a full-text index or RDBMS is a very important use-case which is why I've proposed an additional operation that would facilitate this functionality in a non-invasive robust way outlined here:

http://groups.google.com/group/redis-db/browse_thread/thread/74996322e09d21cd/8dba43d9326adabb

At the moment there haven't been much discussion on this topic, but I'm still hoping for this or a better solution to be added in Redis to better support this use-case.

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

--
- Demis

http://twitter.com/demisbellot

http://www.servicestack.net/mythz_blog

[Aaron Boxer]

You might be interested in Alchemy, a lightweight RDBMS built on top of redis:

http://code.google.com/p/alchemydatabase/

[ben wiseley]

At http://deucescracked.com (Rails, MySql, Redis, Memcache) we've been moving things that are slow in MySql to Redis and keeping the rest in MySQL since it's easy to deal with in Rails (although this will change when we move to Rails 3 and write an adapter for Redis)

For example:  We store what users watch what videos and use this for display, sorting, display counts, recommendations, and other things on the site.  It is used on almost every page.  Given that the table was users * videos in size and had to be indexed, inserts were getting very slow and reads weren't doing that great either (also - locking issues).  We moved that entire table to Redis and it's lightening fast.  Instead of doing things like:

select videos.name from videos, video_views where videos.id <> video_views.video_id and video_views.user_id = current_user.id

We now do things like

select videos.name from videos where videos.id not in (REDIS.smembers "video-views:{current_user.id}")

And - since we're just doing a SADD when a user watches a video we're not being effected by the very slow inserts anymore.

We do similar things on latest-post-read - which, again is user * posts in size and was very slow.

Slowly we've been moving more and more over to Redis.

Our biggest challenge with Redis so far is that, for reporting, we still feel like we need MySql since responding to the boss' "Give me this and that, when this or this or this and that happened" is more or less impossible to do from Redis without first creating lots of indexes, sets, etc.

-ben

[SN]

Thanks a lot, looks very interesting. Will look into that ASAP.

/SN

On 19 Dec., 13:32, Demis Bellot <demis.bel...@gmail.com> wrote:
> At mflow.com, we stored all 'live' or transient data in Redis. i.e. things
> like User Sessions, latest flows/posts/purchases, most
> popular flows/posts/purchases etc.
>
> Everything ultimately was persisted on sharded PostgreSQL dbs however we
> were using Redis as a smart cache/data-view on top, where we were caching by
> validity (not time) so all reads would be serviced by Redis unless a user
> made a change to their profile, etc in which case we would clear the caches
> in Redis then.
>
> 2) Another scenario could be to store everything in Redis and also
>
> store all data in a RDBMS. Would it be good approach be to use some
>
> message queue when making the store operations in RDBMS so that the
>
> slow writing to RDBMS can be done in a background process? Are there
>
> any examples of this anywhere ?
>
> Yeah we also had a Redis MQ implementations that would take 'async one way'
> web service requests and drop them in a Redis Message Queue which would be
> later processed by a background process in its own time (i.e. outside the
> original request boundary). I have an implementation of the C# Redis Message

> Queue used here:https://github.com/mythz/ServiceStack.Redis/tree/master/src/ServiceSt...
> with
> some tests here:http://bit.ly/dJT5ptandhttp://bit.ly/gsYBDQ

>
> You should also have a look at Resque which is a robust message queue redis

> implementation specifically designed for this task:https://github.com/blog/542-introducing-resque(intro)https://github.com/defunkt/resque#readme(project site)

>
> Now I also think the use-cases of primarily storing data in Redis and then
> have a back-end process creating a mirror of that data in a full-text index
> or RDBMS is a very important use-case which is why I've proposed an
> additional operation that would facilitate this functionality in a

> non-invasive robust way outlined here:http://groups.google.com/group/redis-db/browse_thread/thread/74996322...

> > redis-db+u...@googlegroups.com<redis-db%2Bunsubscribe@googlegroups.c om>
> > .