I get 'Could NOT connect to memcache server' sometimes even when server is up

[snehal]

team,

i am having this problem for quite sometime now. i have setup a
memcached server on app1. and this is accessed from app1, app3 and
app4 servers. I have seen that the server is always up and it has data
cached in it. but still quite a few times (atleast 20-30 times) every
day I keep getting the message

"Could NOT connect to memcache server"

i dont really understand why this is the case. When I get this
message, I go and check "ps -e | grep memcache" and see that the
server is up and it also has the data cached.

I also telnel localhost 12345 and run "stats" and see that its NOT
running out of connections. Here are the STATS details:

stats
STAT pid 1683
STAT uptime 688406
STAT time 1278866365
STAT version 1.4.5
STAT pointer_size 64
STAT rusage_user 2085.774352
STAT rusage_system 8767.835955
STAT curr_connections 11
STAT total_connections 8968555
STAT connection_structures 87
STAT cmd_get 551823008
STAT cmd_set 112357
STAT cmd_flush 0
STAT get_hits 551806056
STAT get_misses 16952
STAT delete_misses 0
STAT delete_hits 0
STAT incr_misses 0
STAT incr_hits 0
STAT decr_misses 0
STAT decr_hits 0
STAT cas_misses 0
STAT cas_hits 0
STAT cas_badval 0
STAT auth_cmds 0
STAT auth_errors 0
STAT bytes_read 6392394673
STAT bytes_written 761330644533
STAT limit_maxbytes 134217728
STAT accepting_conns 1
STAT listen_disabled_num 0
STAT threads 4
STAT conn_yields 0
STAT bytes 90809899
STAT curr_items 48165
STAT total_items 112357
STAT evictions 0
STAT reclaimed 0
END

Can someone please tell me why do i STILL keep getting the COULD NOT
CONNECT to the server ERROR? Would really really appreciate a quick
response.

Thanks,
Snehal

[Dustin]

On Jul 11, 9:40 am, snehal <sne...@gmail.com> wrote:

> Can someone please tell me why do i STILL keep getting the COULD NOT
> CONNECT to the server ERROR? Would really really appreciate a quick
> response.

What is producing that error?

[Snehal Shinde]

These are my PHP Memcache settings

[root@app1 ~]# php -i | grep memcache

PHP Warning:  Unknown: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'UTC' for 'UTC/0.0/no DST' instead in Unknown on line 0

memcache

memcache support => enabled

memcache.allow_failover => 1 => 1

memcache.chunk_size => 8192 => 8192

memcache.default_port => 11211 => 11211

memcache.default_timeout_ms => 1000 => 1000

memcache.hash_function => crc32 => crc32

memcache.hash_strategy => standard => standard

memcache.max_failover_attempts => 20 => 20

Snehal

On Sun, Jul 11, 2010 at 12:31 PM, Snehal Shinde <sne...@gmail.com> wrote:

When users access the site, to get data it tries to connect to memcache. I check if I am getting an error value back while connecting. That is when I get this error message. However, as soon as I get this error, I immediately go and check if the process is up and everytime its up and working fine.

Is this a problem with memcache.

default_timeout_ms ? its set to 1000 by default. do I need to bump it up? if so, how to do that? I have iptables turned off on the server that is running memcached server. Also its a persistent memcache by default. I think what might be happening is that though the memcached server is up and running at times the servers are not able to connect, maybe cause it timesout before it gets data back?

Any help or pointers would be highly appreciated

Snehal

[Snehal Shinde]

When users access the site, to get data it tries to connect to memcache. I check if I am getting an error value back while connecting. That is when I get this error message. However, as soon as I get this error, I immediately go and check if the process is up and everytime its up and working fine.

Is this a problem with memcache.

default_timeout_ms ? its set to 1000 by default. do I need to bump it up? if so, how to do that? I have iptables turned off on the server that is running memcached server. Also its a persistent memcache by default. I think what might be happening is that though the memcached server is up and running at times the servers are not able to connect, maybe cause it timesout before it gets data back?

Any help or pointers would be highly appreciated

Snehal

On Sun, Jul 11, 2010 at 12:17 PM, Dustin <dsal...@gmail.com> wrote:

[a.]

If you telnet to port 12345 (as you mentioned in your first mail), then maybe you should change the 11211 (which is the default port for memcached) in your php.ini?

a.

[Snehal Shinde]

In my PHP code I am using the correct port number as shown below:

$mResult = $this->cache->connect($my_ip,"11234");

if($mResult===false) {

$this->error = "Could NOT connect to memcache server";

error_log($this->error);

[Trond Norbye]

On 11. juli 2010, at 22.49, Snehal Shinde <sne...@gmail.com> wrote:

In my PHP code I am using the correct port number as shown below:

$mResult = $this->cache->connect($my_ip,"11234");

That's different from 12345.....

Trond

[James Phillips]

Port is 11211 not 11234

[Snehal Shinde]

Yes the default port is 11211 but i override it in my php code to 11234 and i do habe the memcached server running at port 11234. So that is not the problem atall

[Dustin]

On Jul 11, 4:44 pm, Snehal Shinde <sne...@gmail.com> wrote:
> Yes the default port is 11211 but i override it in my php code to 11234 and i do habe the memcached server running at port 11234. So that is not the problem atall

You said you successfully telnetted to port 12345, not 11234.

[Jay Paroline]

Clearly that is not what he meant. You can see from his initial post
that the memcached server is getting plenty of hits, so unless it's
shared by other non-php clients, his configurations on the server and
client match. Otherwise the server would be empty and the issue would
not be intermittent.

We actually run into this issue as well, with no known or demonstrable
network problems between the boxes, but we're also running an old
version of memcached and the failure rate is something like .001% of
daily traffic so getting it resolved hasn't been a high priority for
us.

[Snehal Shinde]

Yes Jay is right. My server and client config are in sync. The problem
is only intermittent and so i felt maybe increasing the default
timeout might help. Any idea how i can increase the default memcache
timeout for the php client Memcache?

[dormando]

On Sun, 11 Jul 2010, Snehal Shinde wrote:

> Yes Jay is right. My server and client config are in sync. The problem
> is only intermittent and so i felt maybe increasing the default
> timeout might help. Any idea how i can increase the default memcache
> timeout for the php client Memcache?

I forget offhand :/ if it's not listed in pecl.php.net/memcache then
you're a little SOL.

Are you on version 3.x or version 2.x? (I might've missed this from
earlier in the thread). I've heard rumors of random connect failures in
3.x.

Also, your network might be experiencing a really low rate of packet loss.
If you can change the timeout, set it to 2 seconds for a while.. if you
still get random timeouts, set it to 4 seconds (higher than the SYN
retry). If it never fails or almost never fails at 4 seconds then you have
some packet loss that might be fixable.

It should never take memcached longer than a few ms to set up a new
connection, so anything worse than that usually means the box is hosed
(swapping) or the network is hosed (maxed, packet loss, etc).

[Snehal Shinde]

I am using the stable 2.x one. I have set the timeout to 3 secs now. Lets see how that goes

Snehal

[dormando]

On Mon, 12 Jul 2010, Snehal Shinde wrote:

> I am using the stable 2.x one. I have set the timeout to 3 secs now. Lets see how that goes
> Snehal

You might want to stick with 2 or 4 seconds to test :) putting it right on
the line with the SYN timeout will still give you inconsistent results...

[Snehal Shinde]

switched to 4 secs now. also if its a packet loss issue will it make sense to reduce the $retry_interval from the default 15 secs to say 2 secs? http://us2.php.net/manual/en/memcache.setserverparams.php

also the error that i get intermittenly is "COULD NOT CONNECT TO SERVE". Is there a way to find out more? why it was not able to connect? If I restart the memcached with -vvv option, will it give a lot more details in the error message? just trying to find a way to get more info

snehal

[Gavin M. Roy]

What is your pageview velocity?  There's really no good reason you should be seeing intermittent failure and having timeouts that high will impact your pageload performance.

As was mentioned, if you're seeing behavior like this you ever have serious network or server hardware/configuration issues independent of memcached.

[dormando]

> switched to 4 secs now. also if its a packet loss issue will it make sense to reduce the $retry_interval from the default 15 secs to say 2 secs?�http://us2.php.net/manual/en/memcache.setserverparams.php
> also the error that i get intermittenly is "COULD NOT CONNECT TO SERVE". Is there a way to find out more? why it was not able to connect? If I restart the memcached with -vvv option, will it give a lot
> more details in the error message? just trying to find a way to get more info
>
> snehal

top posting.

If you can, I'd recommend writing a commandline php nugget and run it in a
loop reconnecting over and over. Run that under strace and log to file, so
you can see what syscall was returning (RST, timeout, etc).

running memcahced in -vv or -vvv mode will create an intense amount of
logging.