TTL expiry: inconsistent behavior?

[Xiangrong Fang]

Hi Salvatore,

I noticed that TTL of a key is removed when you use SET, but why?   I thought that behavior is SAME when I use HSET, HDEL etc, but it seems that only SET will remove the TTL?

Another thought is about protocol compatibility.   If it is desirable to change behavior of a command, you may face the problem of incompatibility and may feel reluctant to change, even such change is reasonable.   In such case, is it possible to solve incompatibility problem by using CONFIG SET (redis.ini)?

Thanks,
Shannon


--

[Salvatore Sanfilippo]

On Sat, Oct 29, 2011 at 7:27 AM, Xiangrong Fang <xrf...@gmail.com> wrote:

Hi Salvatore,

I noticed that TTL of a key is removed when you use SET, but why?   I thought that behavior is SAME when I use HSET, HDEL etc, but it seems that only SET will remove the TTL?

I'll reply you with an example.

In Ruby and other dynamic programming languages you can call methods.

For instance HSET is like the [] method of an hash:

key[:foo] = 'bar'

When you call methods you are just modifying the object. Sometimes however you can just rebind a variable of some kind with a new object. You do that with the "=" operator:

key = "Hello World"

When you do this the old object is gone, the symbol "key" is associated to a new object.

In Redis it is exactly the same: SET assigns a key to a new string object. This string object is new so it does not have any of the previous properties of the old one, including the expire.

Another way to see it is that in Redis's semantics the TTL is bound to the value and not to the key itself.

Another thought is about protocol compatibility.   If it is desirable to change behavior of a command, you may face the problem of incompatibility and may feel reluctant to change, even such change is reasonable.   In such case, is it possible to solve incompatibility problem by using CONFIG SET (redis.ini)?

Only way to introduce an incompatibility in the API is to change the version so that the major number (the x of x.y.z) is incremented (http://semver.org). It is not impossible to do so, but we need very good reasons for something like that.

For instance Redis 3.0 may unify HMSET and HSET into a single command, but probably it is not a very good idea.

Similarly because of how the previous version of the protocol worked we have a few commands with a bad argument position. It is possible to improve that, but I guess this is already not a good idea.

We need to have good reasons and some real important improvement of Redis API or semantics for breaking the API to be legit.

Cheers,

Salvatore

 

Thanks,
Shannon


--

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

--
Salvatore 'antirez' Sanfilippo
open source developer - VMware

http://invece.org
"We are what we repeatedly do. Excellence, therefore, is not an act, but a habit." -- Aristotele

[Xiangrong Fang]

Hi Salvatore,

I don't agree with your reasoning :-)  But I don't have a good example to explain my point yet.  I just happened to encounter this inconsistency and feel it is not "logically" right.

I would like to ask, is this a design choice to let SET revoke the TTL, or it is a technical choice (i.e. you have to do it like this, given the current infrastructure?   If it is only a design choice I might raise this issue again if there is indeed a good reason.

Thanks,
Shannon

2011/10/29 Salvatore Sanfilippo <ant...@gmail.com>

--

[Josiah Carlson]

The only sane design choice available. The alternative (keeping the old TTL) would make using TTLs to expire caches not correct in some cases.

Let us say, for example, that you were to cache a user session and set a 5 day EXPIRE (the total time doesn't matter, but 5 days is reasonable for this example). Let us say that just under 5 days later, the session was re-set. If the TTL wasn't cleared, then there is a chance that before the TTL was updated with a subsequent EXPIRE (because it takes a nonzero amount of time to perform commands), the session could be cleared due to an expired TTL. This is a race condition. Rare in the case of a 5 day TTL, but not so rare and not so impossible on a heavily-loaded system with much shorter TTLs.

The re-setting of TTLs during SET is the only correct way of doing things. If you don't agree, read my previous paragraph a few times until you do. ;) That Redis doesn't reset TTLs on list, set, zset, or hash operations is understandable, though sometimes confusing. I don't know which I personally prefer (resetting or keeping TTL on list, set, zset, or hash modification), but I don't see anything necessarily wrong with keeping it as it is now.

Regards,

 - Josiah

[Joseph Jang]

Hi,

I don't think your example forces to choose a design choice.

An situation in your example happens because SET and EXPIRE are
separate operation, not because of semantics of expiration.
SETEX atomically store string and set expiry time and solves your
problem gently.

If you're on 'a heavilly-loaded system with much shorter TTLS',
everything can become problematic with non-atomic operations.
I can say we better not depend on TTL in the situation that requires
<= 1000 milliseconds-precison without SETEX

Thanks,
Joseph

> > 2011/10/29 Salvatore Sanfilippo <anti...@gmail.com>

[Josiah Carlson]

Hello Joseph,

On Thu, Nov 3, 2011 at 3:47 PM, Joseph Jang <josep...@gmail.com> wrote:
> I don't think your example forces to choose a design choice.

Force, perhaps not. But I think it informs and pushes the decision in
a particular way.

> An situation in your example happens because SET and EXPIRE are
> separate operation, not because of semantics of expiration.
> SETEX atomically store string and set expiry time and solves your
> problem gently.

Sure, and that's under the assumption that people using this software
know all of the commands. Easily 10-20% of all posts in this mailing
list are people asking "how do I do ...?" Not uncommonly, the answer
is a "use command ... [with command ...]" AKA reading the
documentation.

> If you're on 'a heavilly-loaded system with much shorter TTLS',
> everything can become problematic with non-atomic operations.
> I can say we better not depend on TTL in the situation that requires
> <= 1000 milliseconds-precison without SETEX

I agree 100%, which is why I write against Redis to never use
expiration and TTLs. Either it's around forever, or I manually remove
it.

I agree that the scenario I posed was on the edge of being reasonable,
being that there is a command that so quickly and easily solves the
problem (under the alternate semantic "don't clear TTLs when setting a
key"). But the thing is, Redis can and will be used by people who
don't know all the commands, or for one reason or another choose not
to use that command. While I'm sure that Salvatore didn't think of the
race condition I mentioned while choosing the semantic, that there
exists a race condition in the semantic that he didn't choose (which
is being advocated for by at least one person), where no such race
condition exists in the current semantic, suggests that the current
semantic should be kept.

Or you can go with Salvatore's explanation that "SET X Y" is like
"DELETE X" followed by "SET X Y", which is simpler and easier :)

Regards,
- Josiah