Set TTL to members of a list, set, sorted set

[jason.桂林]

It is  very useful to set TTL to members of a list, set, sorted set.

Redis is not only a key value store, so why only we can set ttl of a key, we also need set TTL to members.

If redis can implement this feature it will be very cool.

--
Best regards,

桂林 (Gui Lin)

guileen@twitter

桂糊涂@weibo

guileen@github

[Josiah Carlson]

It won't be implemented because the memory overhead of offering the feature would further bloat Redis' already not-slim memory usage. This is easily the 10th time I've heard this feature requested, and every time it has been refused.

If you can describe what you are actually trying to do, there may be an alternate way to store your data which would allow for automatic data expiration.

 - Josiah

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/groups/opt_out.

[Guy Lubovitch]

Sorry for jumping on this thread but i have similar user case and now that i know TTL for list is not optional i would be happy to understand alternative patter to do this.

Use Case, On a web site i have users and account with many to one, i store user session object in redis and from time to time i need to get all users session that below to a specific account, but here comes the trick. the session expire after 10 min ( Config )
so this is easy i write user session with ttl of 10 min, now i need to get all active user session of a specific account. using a list would be easy as i can add a item to a list when i add a new user session, but user session expire and the list doesn't. 
Ltrim wont help me here since i need the actual live sessions.

any thought on how i solve this problem in redis ?

[Josiah Carlson]

When adding a user session, perform the following 3 operations in a MULTI/EXEC pipeline and/or Lua script:

SETEX <key> 600 <session information>

SADD <user>:sessions <key>

EXPIRE <user>:sessions 600

Periodically (every time a user manually logs out, logs in, and/or every 100th request from that user), run the following Lua script, passing a single "KEYS" argument of "<user>:sessions" . This also doubles as the "get me the sessions for user X" method:

-- warning, this is not Redis Cluster safe

local known = redis.call('SMEMBERS', KEYS[1])

for i, key in ipairs(known) do

    if not redis.call('EXISTS', key) then

        redis.call('SREM', KEYS[1], key)

    end

end

return redis.call('SMEMBERS', KEYS[1])

If the user does nothing for 10 minutes, all of their data is automatically cleaned up. If the user has multiple logins that do different things at different times, then you'll also clean-up after login, after logout, and periodically when the user is doing something.

There is an alternative that uses ZSETs with expiration times as scores instead of a SET, which makes cleanup a bit cleaner/faster, but it requires properly synced clocks on clients making requests, and it requires sane datetime handling on clients (which is not always a safe assumption to make).

 - Josiah

For more options, visit https://groups.google.com/d/optout.

[Guy Lubovitch]

Hi, first many thanks for this answer but i think something is missing here.

this is a Saas solution so i have no create session or remove session even otherwise it will be easier :). tell me what you think about this patter ?

when i add session i also add session id to account list

Setex <key> 600 <session info>
Rpush <account> <session-key>

i can also check for eviction for this account ( every set or every X sets ).

So create a loop

and take the list from the left side until the item exists -- something like this

while (true )
{
   key   = redis.blpop(<account>)
   if ( redis.exist( key ) )
   break;
}

the reason to use a list and not a set even though i need uniqueness, is the speed and being able to add to the left and take from the right. this list will have more then 1000 and probably closer to 10k

will appreciate your feedback.

[Josiah Carlson]

I've got a solution to your problem, which I've written in Python to be as explicit as possible (code is better than description). Note that the Lua code contained in this is *not* cluster safe.

Structures used:

STRING for holding session data:

    <session key> -> <session data>

ZSET for holding known sessions for the account: 

    <account> -> {<session key>: <expiration time>, ...}

ZSET for global session registry to clean up entries without requiring full 10 minute timeout for all of an account's sessions:

    all: -> {'<account>:<session key>': <expiration time>, ...}

Whenever you have new information about an account session, you would call the update_session() function defined as follows:

def update_session(conn, account, s_key, s_data, all_key='all:', expire=600):

    expires = time.time() + expire

    # Use a MULTI/EXEC transactional pipeline

    conn.pipeline(True) \

        .setex(s_key, expire, s_data) \

        .zadd(account, **{s_key: expires}) \

        .expire(account, expire) \

        .zadd(all_key, **{account + ":" + s_key: expires}) \

        .expire(all_key, expire) \

        .execute()

To clean up old sessions, you should periodically run the following Lua script directly, or the Python function. The function could be run as part of a daemon every few seconds or minutes, depending on how many sessions it needs to clean out each step, and how aggressive you want to be in your cleanup efforts. Note that calling the Python function will attempt to clean out *every* expired session it can find until it is done, so it has an unknown total execution time. Note that it makes multiple Lua script calls during this process, so Redis *can* continue to handle other requests as the cleanup is occurring. You can give it an upper limit by passing max_cleaned. You can also call the Lua script directly, which allows for a tunable number of sessions to clean out with each call (also exposed in the Python function), defaulting to 20. Up to around 100 per call should be reasonably safe, but the higher the number of items to be cleaned per Lua call, the higher the command latency, which could affect other commands running in Redis.

lua_script = '''

local to_clean = redis.call(

    'ZRANGEBYSCORE', KEYS[1], '-inf', ARGV[1], 'LIMIT', ARGV[2] or 20)

for i, val in irange(to_clean) do

    local account = string.match(val, '([^:]+):')

    local session = string.match(val, ':(.+)')

    -- removes the session from the account listing

    conn.zrem(account, session)

    -- removes the global entry that we just used

    conn.zrem(KEYS[1], val)

end

return #to_clean

'''

def cleanup(conn, all_key='all:', count=20, max_cleaned=2**64):

    cleaned = count

    total_cleaned = 0

    while cleaned == count and total_cleaned < max_cleaned:

        cleaned = conn.execute_command('EVAL',

            lua_script, 1, all_key, time.time(), count)

        total_cleaned += cleaned

    return total_cleaned

To get the list of valid sessions for a given account, you only need to perform:

conn.zrangebyscore(account, time.time(), "inf")

That will be correct, regardless of whether the cleanup() function or lua script has been called recently.

As a bonus feature, if every client stopped making calls and Redis was left alone, Redis should clean itself up entirely after 10 minutes. :)

Regards,

 - Josiah

[Guy Lubovitch]

Hi,

Many thanks for the detailed answer, it looks very good. what i am not clear about is the global range set. is it possible for a sorted set to hold 100k items ?

i guess another option is to have another list that holds all the accounts ( there should be less then 1000) and the use this set to go over the account range sets.

[Josiah Carlson]

I have held 155 million items in a ZSET before, 100k isn't a problem.

 - Josiah

[Guy Lubovitch]

Looks impressive just one last question. why not make it all in perl ( or in my case java ) and be able to support multiple redis?

all the lua script does is delete from zset.

[Josiah Carlson]

I don't understand your question. Are you asking about Redis Cluster support, or something else?

Yes, all it does is delete from the ZSET, but it does it in a single round-trip without any race conditions.

 - Josiah