3 second delay on get

[Antonio]

Hello!

I'm seeing seeing a very small (but significant) percentage of our memcache get() requests taking just over 3 seconds to execute (about 3.001-3.015 seconds). 

It seems to happen pretty randomly--and all the other requests take under a millisecond. Whether or not the get request hits or misses seems to make no difference.

Some googling suggests that this delay might correspond to TCP SYN packets being dropped and then resent. If that's the case, is there a way to time out more quickly and either retry or fail (and hit the database)?

Here's what we're running:

CentOS 5.6

Memcached server: 1.4.5  (five 4GB instances on 5 web servers all on a private network on the same rack)

Client: PECL Memcached PHP extension 1.0.2 (libmemcached 0.31)

Web server: Nginx 1.0.2 with php-fpm

We're setting the following options:

$mc->setOption(Memcached::OPT_NO_BLOCK, true);

$mc->setOption(Memcached::OPT_TCP_NODELAY, true);

$mc->setOption(Memcached::OPT_POLL_TIMEOUT, 100);

$mc->setOption(Memcached::OPT_RETRY_TIMEOUT, 1);

$mc->setOption(Memcached::OPT_CONNECT_TIMEOUT, 100);

$mc->setOption(Memcached::OPT_LIBKETAMA_COMPATIBLE, true);

$mc->setOption(Memcached::OPT_DISTRIBUTION, Memcached::DISTRIBUTION_CONSISTENT);

Any ideas or insight would be much appreciated.

Thanks,

Antonio

[Brian Aker]

Do you have a test case with data?

BTW I would upgrade to a much more recent version then 0.31.

[Adam Thomason]

On Mon, May 23, 2011 at 1:30 PM, Antonio <anto...@gmail.com> wrote:

Some googling suggests that this delay might correspond to TCP SYN packets being dropped and then resent.

This was exactly what was happening to us with the same symptoms. Finding and replacing the faulty switch cured it.

[FarYang H]

how you fix this problem then? 

在 2011年5月24日星期二UTC+8上午4时30分47秒，Antonio写道：