Redis working as a capped collection for keys?

[Guillermo Ortiz]

  Is it possible to work with Redis like a capped collection of MongoDB?.

  I would like to store just the last 100000 key/values pairs. If a new key/value comes, I would like to delete the oldest and set the new one. I don't know if it's possible to do something like that with Redis. 

[Itamar Haber]

Redis does not provide this behavior out-of-the-box at the moment. However, you can easily set this up via the use of a Sorted Set to keep track of the key names and their creation time and periodically/with every update use that information to delete older keys.

On Wed, Aug 12, 2015 at 3:47 PM, Guillermo Ortiz <konst...@gmail.com> wrote:

  Is it possible to work with Redis like a capped collection of MongoDB?.

  I would like to store just the last 100000 key/values pairs. If a new key/value comes, I would like to delete the oldest and set the new one. I don't know if it's possible to do something like that with Redis. 

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

--

Itamar Haber | Chief Developers Advocate
Redis Watch Newsletter - Curator and Janitor
Redis Labs - Enterprise-Class Redis for Developers

Mobile: +1 (415) 688 2443
Mobile (IL): +972 (54) 567 9692
Email: ita...@redislabs.com
Twitter: @itamarhaber
Skype: itamar.haber

Blog  |  Twitter  |  LinkedIn

[Mark Paluch]

Mongo capped collections work primary by document size (and optional limited by the number of documents). 

IMHO, limiting maxmemory, setting a TTL (a really large one to prevent too early expiration) and setting maxmemory-policy volatile-ttl should do the job on a global level. The Redis instance behaves then like one capped collection and you cannot use the Redis instance for longer lived data.

[The Baldguy]

On Wednesday, August 12, 2015 at 7:56:37 AM UTC-5, Guillermo Ortiz wrote:

  Is it possible to work with Redis like a capped collection of MongoDB?.

  I would like to store just the last 100000 key/values pairs. If a new key/value comes, I would like to delete the oldest and set the new one. I don't know if it's possible to do something like that with Redis. 

There is a relatively easy way to do this using a list and plain keys, wired together with some lua. Here is a lua script you can load, then call:

=====

local k = KEYS[1]

local collection = KEYS[2]

local v = ARGV[1]

local limit = tonumber(ARGV[2])

redis.call("LPUSH",collection,k)

redis.call("SET",k,v)

local size = redis.call("LLEN",collection)

if size > limit then

    local val = redis.call('RPOP',collection)

    if val then

        redis.call("DEL",val)

    end

end

return  redis.call("LLEN",collection)

=====

The way to use this would be to use http://redis.io/commands/script-load to load it, recording the SHA ID it returns.

Then when adding an item you would call (via http://redis.io/commands/evalsha) it with the following arguments:

1. the key you want to add/set

2. The name of the collection

3. the value you want to set the key to

4. the maximum size of the collection 

So for example, Assuming the SHA you get back from loading the script is theSHAidhere:

EVALSHA theSHAidhere newkey collection1 breakfast 4

This would create/set the key "newkey" with the value "breakfast". If the number of items added via this script is > 4 it would remove the oldest. Thus:

EVALSHA theSHAidhere newkey1 collection1 breakfast 4

EVALSHA theSHAidhere newkey2 collection1 secondbreakfast 4

EVALSHA theSHAidhere newkey3 collection1 thirdbreakfast 4

EVALSHA theSHAidhere newkey4 collection1 dinner 4

Results in precisely 4 items, and if only this script is used, 5 total keys in the database. Now if you run

EVALSHA theSHAidhere newkey5 collection1 supper 4

you still have 4 items in the collection, 5 keys total, and "newkey1" has been popped and deleted.

For memory efficiency you could use a hash and of HSET/HDEL instead of set/del. Indeed going this route would allow you to store the size of the collection in the hash itself. Using a hash is an exercise left for the reader. ;)

The downside is you have to 1) load the script (or use plain eval and "load" it every time), and 2) any modifications outside of the script may not play well. Specifically, if you modify the collection list directly such as by trim, or removing it, it doesn't purge the keys you removed from the list - you would have to do that manually. 

Cheers,

Bill

[The Baldguy]

On Wednesday, August 12, 2015 at 4:00:52 PM UTC-5, Mark Paluch wrote:

Mongo capped collections work primary by document size (and optional limited by the number of documents). 

IMHO, limiting maxmemory, setting a TTL (a really large one to prevent too early expiration) and setting maxmemory-policy volatile-ttl should do the job on a global level. The Redis instance behaves then like one capped collection and you cannot use the Redis instance for longer lived data.

You most certainly could still use Redis for longer lived data as you could still add keys w/o expiration since Redis doesn't have a global expiration setting. All you'd need to do is "SET sabot <big-value-here>" and the "capped"  collection is now capped by items outside of it and said items don't expire so would not be pruned when maxmemory is hit. Further, as maxmemory also includes some of the buffers Redis allocates you wouldn't have a direct correlation, even if you knew exactly how much memory the 100,000 keys would consume. In order to go this route you would need to:

  1. Know exactly how much memory the 100K would consume

  2. Know exactly how much memory you will need to allocate from the maxmemory "pool" for buffers

  3. Not use the instance for *anything* else

The hardest is likely item three as Redis gets it's foot in the door and soon it is being used for all sorts of things. ;)

Cheers,

Bill