How changing expire time is supported in > 2.2
Kirzilla
In previous versions of redis < 2.2, it was impossible to change key
expire time because of issues with replication. This fact is
absolutely clear for me.
But how this issue is resolved in > 2.2? For example, we have two
redis instances: master and slave.
1 [00:00] MASTER SETEX my_key value 9
2 [00:05] SLAVE dumping from MASTER
3 [00:08] MASTER EXPIRE my_key 4
4 [00:09] SLAVE my_key expires at SLAVE
4 [00:10] SLAVE dumping from MASTER
As you can see on step #3-4 we have 1 second when my_key is already
expired at SLAVE, but not expired at MASTER because of being
prolongated on step #3.
How this issue is resolved in > 2.2? Are any restrictions added for
using EXPIRE and replication together?
Thank you.
Pieter Noordhuis
Not sure what you mean by "dumping from master", but the general issue
seems to be how expiry + replication is done in >=2.2. It is really
simple: the slave receives the EXPIRE family of commands as expected,
but doesn't actively expire keys. The master is responsible for key
expiry and sends a DEL over the wire (also written to the AOF when
used) when a key expires. This way is it not possible for the slave to
expire keys before the master does. In short: there are no
restrictions.
Cheers,
Pieter
> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>
>
Salvatore Sanfilippo
Pieter perhaps before releasing 2.2 stable is time to collectively
talk about how this is done for key eviction due to maxmemory?
Basically currently we do not synthesize DELs on eviction.
This is good and bad at the same time. It is good since it is possible
to have, in the case of a cache, a slave configured with for instance
half the max memory, and it will evict more than the master, and will
not deceive the master DELs. But it is bad as the slave will not be an
exact copy.
I think that Jeremy Zawodny is using exactly this setup, of
replication + maxmemory eviction. Jeremy: any comment about this is
really appreciated.
There is the possibility of adding this as an option but it is
probably a complex one to understand for many users, maybe it's better
if we pick what's the right thing to do, but honestly I'm not so sure
at the moment.
Cheers,
Salvatore
--
Salvatore 'antirez' Sanfilippo
open source developer - VMware
http://invece.org
"We are what we repeatedly do. Excellence, therefore, is not an act,
but a habit." -- Aristotele
Konstantin Merenkov
<pcnoo...@gmail.com> wrote:
> ..... The master is responsible for key
> expiry and sends a DEL over the wire (also written to the AOF when
> used) when a key expires. This way is it not possible for the slave to
> expire keys before the master does. In short: there are no
> restrictions.
How does it work in this scenario?
1) master has a lot of keys that should expire
2) slave gets these keys from master
3) master goes down
4) keys won't expire on the slave now? because master cannot issue DELs
--
Best Regards,
Konstantin Merenkov
Salvatore Sanfilippo
<kmer...@gmail.com> wrote:
> How does it work in this scenario?
> 1) master has a lot of keys that should expire
> 2) slave gets these keys from master
> 3) master goes down
> 4) keys won't expire on the slave now? because master cannot issue DELs
OMG do we look *so* naive? ;)
Just kidding. We transmit TTL information to the slave, just the slave
does not use this information when there is a connected master, but
once you do: SLAVEOF NO ONE against the slave turning it into a master
it will start to expire keys independently.
Cheers,
Salvatore
Konstantin Merenkov
> On Thu, Jan 27, 2011 at 11:47 AM, Konstantin Merenkov
> <kmer...@gmail.com> wrote:
>> How does it work in this scenario?
>> 1) master has a lot of keys that should expire
>> 2) slave gets these keys from master
>> 3) master goes down
>> 4) keys won't expire on the slave now? because master cannot issue DELs
>
> OMG do we look *so* naive? ;)
> Just kidding. We transmit TTL information to the slave, just the slave
> does not use this information when there is a connected master, but
> once you do: SLAVEOF NO ONE against the slave turning it into a master
> it will start to expire keys independently.
You are not answering to the question :-)
SLAVEOF NO ONE is issued by client to turn slave into a master. Master
can expire keys, right.
What I am talking about is: slave can live by itself when master is
disconnected, without getting any SLAVEOF NO ONE (i.e. slave is still
a slave). Does keys expiration works in the same way as if slave is a
master?
Salvatore Sanfilippo
<kmer...@gmail.com> wrote:
> On Thu, Jan 27, 2011 at 1:49 PM, Salvatore Sanfilippo <ant...@gmail.com> wrote:
>> On Thu, Jan 27, 2011 at 11:47 AM, Konstantin Merenkov
>> <kmer...@gmail.com> wrote:
>>> How does it work in this scenario?
>>> 1) master has a lot of keys that should expire
>>> 2) slave gets these keys from master
>>> 3) master goes down
>>> 4) keys won't expire on the slave now? because master cannot issue DELs
>>
>> OMG do we look *so* naive? ;)
> I didn't mean to offend you, I just want to know for sure :-)
I was just kidding ;) And indeed I think you spot a bug.
> You are not answering to the question :-)
> SLAVEOF NO ONE is issued by client to turn slave into a master. Master
> can expire keys, right.
> What I am talking about is: slave can live by itself when master is
> disconnected, without getting any SLAVEOF NO ONE (i.e. slave is still
> a slave). Does keys expiration works in the same way as if slave is a
> master?
Currently the only condition is: if it is a master. But this is probably wrong:
/* Expire a few keys per cycle, only if this is a master.
* On slaves we wait for DEL operations synthesized by the master
* in order to guarantee a strict consistency. */
if (server.masterhost == NULL) activeExpireCycle();
this should probably be instead:
if (server.masterhost == NULL || server.replstate !=
REDIS_REPL_CONNECTED) ...
Because after all whatever we have in the slave at this point is going
to be removed on new master->slave sync.
But on the other side, the change is not so useful, since after the
slave is disconnected it is going to face one of this two scenarios:
1) the master <-> slave link will be established again, and the data
will be removed anyway.
2) the slave will be elected as new master.
In both cases keys will expire again. But still I think you have a
point here. I need to evaluate if there are problems changing this
code now that the replication is non blocking and we re-enter the
event loop, but should be safe actually as we don't call timers while
re-entering the event loop in the .rdb load process.
Cheers,
Salvatore
>
>
> --
> Best Regards,
> Konstantin Merenkov
>
> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>
>
--
Konstantin Merenkov
> On Thu, Jan 27, 2011 at 12:44 PM, Konstantin Merenkov
> <kmer...@gmail.com> wrote:
>> On Thu, Jan 27, 2011 at 1:49 PM, Salvatore Sanfilippo <ant...@gmail.com> wrote:
>>> On Thu, Jan 27, 2011 at 11:47 AM, Konstantin Merenkov
>>> <kmer...@gmail.com> wrote:
>>>> How does it work in this scenario?
>>>> 1) master has a lot of keys that should expire
>>>> 2) slave gets these keys from master
>>>> 3) master goes down
>>>> 4) keys won't expire on the slave now? because master cannot issue DELs
>>>
>>> OMG do we look *so* naive? ;)
>
>> I didn't mean to offend you, I just want to know for sure :-)
>
> I was just kidding ;) And indeed I think you spot a bug.
> Because after all whatever we have in the slave at this point is going
> to be removed on new master->slave sync.
> But on the other side, the change is not so useful, since after the
It is useful for me.
I keep access tokens in redis that should expire on time.
It is also useful for caching when you read from slave and absence of
a key means you should get new data from database.
I bet real life can give more use cases :-)
Jeremy Zawodny
Salvatore Sanfilippo
> Hi Salvatore,
> We are using 2.2 with replication and maxmemory (with lru eviction). In
> fact, we just recently (in the last week) hit the point where we're actually
> at the memory limit, so evictions are probably happening. Currently we have
> the same amount of memory on the masters and slaves, but we've just started
> talking about ways to grow the cluster and it's possible (but not likely)
> that the master and slaves could end up with different amounts of RAM. For
> the sake of simplicity, we'd probably use the same maxmemory values anyway
> and possibly run additional redis-server instances to take advantage of more
> RAM.
Yes this is definitely a good point: when there are boxes with big
asymmetries in memory amount there is the option to run more instances
instead of having masters and slaves with different amounts of memory
that is an odd configuration after all.
> I'm glad you wrote this email, since I had assumed all along that
> maxmemory-based deletes are transmitted to the slave the same way that
> expire/ttl deletes are. At the moment our slaves are purely for fail-over
This suggests that DEL transmitted by default is the least surprise pick.
After all a replica should be a replica.
> purposes but this would be an important detail if we decided to send more
> complex queries to our slaves as a way of distributing load. We have
> thought about doing that a few times and the inconsistent results would have
> been surprising.
Indeed, good point. Even if it's a cache one would expert the replicas
to have the same set of keys, especially if they are used for reads.
> On the one hand, I'm inclined to say "you should make that behavior an
> option that the user can configure in redis.conf." But then I think about it
> a bit more and wonder why anyone would not want it turned on by default.
> Said another way, if the default had always been that the master sends DEL
> to the slaves on each maxmemory expire, and then someone suggesting
> disabling that (so that we could have the current behavior and run slaves
> with a different maxmemory value) wouldn't it seem like an edge case? (Or at
> least a less common use.) I could see people arguing against that because
> it adds more code for a feature that most users will never want or need.
> Thoughts?
Yes... an option in the "tweaks" part of redis.conf could do it, but I
think that since we will always be in time to add this, maybe it's
better to just enable DELs to fix that behavior, and then check if in
the future there will be users that need to take a different pat. Very
unlikely actually...
Btw it seems clear that as it is it looks a lot like a bug that's
better to fix :)
Fixing it and reporting back to the group.
Thanks!
Salvatore
Salvatore Sanfilippo
https://github.com/antirez/redis/commit/452229b6fb0251f82d9caa5388db1160f10bc2d5
Just tried a randomized test, and at the end DEBUG DIGEST showed the
same SHA1 for both
the instances. In general I think this is pretty safe being one line
and already tested code for expire.
I guess we'll have an RC5 :)
Cheers,
Salvatore
Jeremy Zawodny
Salvatore Sanfilippo
> Excellent, thanks!
> We probably need to schedule a time to finally upgrade. Our uptime 2+
> months has been great, but we're on an older RC now. I need to bump up
> maxmemory anyway. :-)
For bumping maxmemory you can just do:
CONFIG SET maxmemory <newvalue> :)
But I guess you need to actually add the RAM before to proceed, most
of the time ;)
Jeremy Zawodny
Al Moser
Josiah Carlson
--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/groups/opt_out.