redis and map reduce

[S Ahmed]

Can you run map reduce on redis somehow?

Say I am storing user analytics in redis, could I somehow summary that information using map reduce?

[Geoffrey Hoffman]

On Thu, Dec 22, 2011 at 7:51 AM, S Ahmed <sahme...@gmail.com> wrote:

Can you run map reduce on redis somehow?

Say I am storing user analytics in redis, could I somehow summary that information using map reduce?

As a data store, sure... something like this could work:

Store analytics in hashes[1], store the keys of the hashes to process in a list[2], which is your unprocessed raw data. 

Then BRPOPLPUSH [3] -> in progress list -> and then BRPOPLPUSH again onto a finished queue. You could have weighted sets to calculate rollup info based on score (timestamps) of when the analytics came in, eg per hour gets tablulated with INCR [4] counts for things like the URL, browser types, etc.

[1] http://redis.io/commands/hmset

[2] http://redis.io/commands/rpush

[3] http://redis.io/commands/brpoplpush

[4] http://redis.io/commands/incr 

However, Redis doesn't do any of the actual mapping or reducing. Your code would act as the workers to move data from raw -> processing -> tabulate -> done. Someone may have already put together Hadoop & Redis or similar.