Possibilities for Redis data aggregation from an RDBMS?

[camcaine]

Hello. This is my first post, and quite new to Redis so please be
gentle.

I am developing a rails app that gets data sync'd to it from an
iPhone. Most of the web app structure is standard relational
(account, users, teams etc.), which currently will go into a sql db.

Therefore I don't really need any of Redis' excellent incrementing/
inserting features. What I would really like to do is use the power
of Redis to do aggregations on the data using mainly lists, sets and
sorted sets, as doing this in SQL looks like a nightmare. The amount
of data is not huge and would fit into memory comfortably I think.

Is it kosher to use Redis in such a way – essentially dumping from SQL
into Redis without persisting Redis?
or
would it make more sense to store the data I want to aggregate in
Redis at the lowest level and skip the SQL?

Has anyone tried something similar?

One more question – is there a reason why there is only ZUNIONSTORE
and no ZUNION (like SUNION)?


Many thanks. I am looking forward to using Redis. It really is a
great tool.

Cameron

[Nick Quaranto]

This is certainly a valid use case. You can certainly use redis as a dependable data store if you keep appendonly mode on and remember to cron up BGREWRITEAOF once daily or so.

Or don't, treat it as a transient store only for temporary calculations and disable fsync all together (or much less frequently). You can really tune it to how you need to use it in your app.

As for ZUNION I thought about that too, recently. I'm not sure of why it doesn't exist, but you could always set an EXPIRE time on it so you recover that memory after you've pulled the data you need out of it.

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>

[Josiah Carlson]

ZUNION doesn't exist because you can implement ZUNION via a pipeline
that includes fetching all of your results and deleting the results
afterwards. It is just as fast and results in fewer commands to
implement (on the Redis server side).

- Josiah

[camcaine]

Could you give an example?

On Mar 8, 6:06 pm, Josiah Carlson <josiah.carl...@gmail.com> wrote:
> ZUNION doesn't exist because you can implement ZUNION via a pipeline
> that includes fetching all of your results and deleting the results
> afterwards. It is just as fast and results in fewer commands to
> implement (on the Redis server side).
>
>  - Josiah
>
>
>
>
>
>
>
> On Tue, Mar 8, 2011 at 6:45 AM, Nick Quaranto <n...@quaran.to> wrote:
> > This is certainly a valid use case. You can certainly use redis as a
> > dependable data store if you keep appendonly mode on and remember to cron up
> > BGREWRITEAOF once daily or so.
>
> > Or don't, treat it as a transient store only for temporary calculations and
> > disable fsync all together (or much less frequently). You can really tune it
> > to how you need to use it in your app.
>
> > As for ZUNION I thought about that too, recently. I'm not sure of why it
> > doesn't exist, but you could always set an EXPIRE time on it so you recover
> > that memory after you've pulled the data you need out of it.
>

[Josiah Carlson]

The commands to emulate a ZUNION is as follows:

ZUNIONSTORE <tempkey> <count> source1 source2 ...
ZRANGE <tempkey> 0 -1 WITHSCORES
DELETE <tempkey>

Wrap those in whatever client-side pipelining you want, or throw a
multi/exec around it, and you are done.

- Josiah

[camcaine]

Cool I see. I guess could use expire instead to just delete the key
after your done getting the data.