Corner case with timestamp TTLs

[Slawomir Pryczek]

Hi Everyone, i have just came upon very strange corner case when doing SET using UNIX_TIMESTAMP plus TTL in memcached running in vmware container. It seems that on barebone servers there is no such issue.

So if the container is running for some time, eg. a week or more, then it seems that timestamps between PHP and memcached are de-synchronizing, so the code like this will no longer work as expected.

$mc = __getMemcache();
$mc->set("key", "value", 0, time()+10);
var_dump($mc->get("key"));
Will return false.

The workaround is to restart memcached from time to time, then it again starts working correctly and will return "value".
Doing $mc->set("key", "value", 0, 10) will not result in same issue, so problem might be that:
1. vmware keeps timestamps counted per process and the errors are aggregating over time
2. there's some cache for timestamp build into memcached so it's using some internal code for getting timestamp and the errors are aggregating there

Any comments on that?
Basically it'd be good to put into docs if using TIMESTAMP+time is safe to do on barebone servers and on VMs. Maybe some info that it shouldn't be used for small TTLs because for higher ones 1-2 minute differences are probably irrelevant anyway...

Thanks,
Slawomir.

[dormando]

Yo,

In general we don't recommend using timestamps for short TTL's. They were
really meant for dates far off in the future (months, where the accuracy
wouldn't matter too much). There're zero use cases for TIMESTAMP + TTL
where ttl is short. simply stop adding time() to it, or run it through a
function to convert if the requested TTL is huge.

That said, your problem isn't specifically "VM's" or not VM's, it's that
your VM sucks at keeping time!

memcached uses a monotonic clock; which will only ever go forward. This
ensures that if the system clock bounces backwards that an item doesn't
end up becoming immortal due to underflow.

Even on a good system the monotonic clock will drift over a long period of
uptime, usually after a few months. Just by a couple seconds. CPU's with
buggier clocks could shift by minutes I guess. Or in this case VM's.

There isn't too much that could be done about this simply, which is why we
highly recommend people stick to relative TTL's. If I could remove the
date feature I would, but we can't for backwards compatibility reasons.

> --
>
> ---
> You received this message because you are subscribed to the Google Groups "memcached" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to memcached+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/memcached/72771e82-79aa-4b23-9563-1b4099930e68%40googlegroups.com.
>
>

[Slawomir Pryczek]

Hi Dormando, thanks for explaining, very useful post. I read a lot about monotonic clocks and "leap second" problem recently, that's nice memcached is using monotonic. Will switch code to use relative TTLs

> To unsubscribe from this group and stop receiving emails from it, send an email to memc...@googlegroups.com.