memcached failing

[kevin wincott]

hi

we are using memcached 1.4.5_2 on 3 servers, each server has the same
version of all software installed. on two servers memcached is working
perfectly, however on the 3rd if doesnt work, it appears to not return
any data, the sql connection is working fine and the code we are using
is OK as it is shared via NFS to all 3 servers, here is a sample of
when we try to use memcache:

host# /usr/local/bin/memcached -vv -d -u nobody
host# slab class 1: chunk size 80 perslab 13107
slab class 2: chunk size 104 perslab 10082
slab class 3: chunk size 136 perslab 7710
slab class 4: chunk size 176 perslab 5957
slab class 5: chunk size 224 perslab 4681
slab class 6: chunk size 280 perslab 3744
slab class 7: chunk size 352 perslab 2978
slab class 8: chunk size 440 perslab 2383
slab class 9: chunk size 552 perslab 1899
slab class 10: chunk size 696 perslab 1506
slab class 11: chunk size 872 perslab 1202
slab class 12: chunk size 1096 perslab 956
slab class 13: chunk size 1376 perslab 762
slab class 14: chunk size 1720 perslab 609
slab class 15: chunk size 2152 perslab 487
slab class 16: chunk size 2696 perslab 388
slab class 17: chunk size 3376 perslab 310
slab class 18: chunk size 4224 perslab 248
slab class 19: chunk size 5280 perslab 198
slab class 20: chunk size 6600 perslab 158
slab class 21: chunk size 8256 perslab 127
slab class 22: chunk size 10320 perslab 101
slab class 23: chunk size 12904 perslab 81
slab class 24: chunk size 16136 perslab 64
slab class 25: chunk size 20176 perslab 51
slab class 26: chunk size 25224 perslab 41
slab class 27: chunk size 31536 perslab 33
slab class 28: chunk size 39424 perslab 26
slab class 29: chunk size 49280 perslab 21
slab class 30: chunk size 61600 perslab 17
slab class 31: chunk size 77000 perslab 13
slab class 32: chunk size 96256 perslab 10
slab class 33: chunk size 120320 perslab 8
slab class 34: chunk size 150400 perslab 6
slab class 35: chunk size 188000 perslab 5
slab class 36: chunk size 235000 perslab 4
slab class 37: chunk size 293752 perslab 3
slab class 38: chunk size 367192 perslab 2
slab class 39: chunk size 458992 perslab 2
slab class 40: chunk size 573744 perslab 1
slab class 41: chunk size 717184 perslab 1
slab class 42: chunk size 1048576 perslab 1
<16 server listening (auto-negotiate)
<17 server listening (auto-negotiate)
<18 send buffer was 9216, now 233016
<19 send buffer was 9216, now 233016
<18 server listening (udp)
<19 server listening (udp)
<18 server listening (udp)
<19 server listening (udp)
<18 server listening (udp)
<19 server listening (udp)
<18 server listening (udp)
<19 server listening (udp)
<20 new auto-negotiating client connection
20: Client using the ascii protocol
<20 get sqlassoc:5770ee132150b6fca3b63c477c9fb12a
>20 END
<20 connection closed.

can anyone point me in a direction of where to go next to fix the
issue

[dormando]

> hi
>
> we are using memcached 1.4.5_2 on 3 servers, each server has the same
> version of all software installed. on two servers memcached is working
> perfectly, however on the 3rd if doesnt work, it appears to not return
> any data, the sql connection is working fine and the code we are using
> is OK as it is shared via NFS to all 3 servers, here is a sample of
> when we try to use memcache:
>
> host# /usr/local/bin/memcached -vv -d -u nobody
> host# slab class 1: chunk size 80 perslab 13107

[snip]

> <20 new auto-negotiating client connection
> 20: Client using the ascii protocol
> <20 get sqlassoc:5770ee132150b6fca3b63c477c9fb12a
> >20 END
> <20 connection closed.
>
> can anyone point me in a direction of where to go next to fix the
> issue

Was this your *entire* log after starting the server? It looks as though
you've just sent a 'get' without ever putting data into it via 'set's
first.

I'm suspicious of your configuration as well. Are all 3 of your servers
supposed to be using one local memcached instance, or are you configuring
it as a proper cluster, with all 3 memcached instances listed in the
client configurations?

[Yiftach Shoolman]

Do you have any info on your eviction rate and hit ratio on that server ?

--
Yiftach Shoolman
+972-54-7634621

[Kevin Wincott]

hi

thanks for the replies, I inherited this from a previous developer so
please accept my apologies for lack of details.

this was the entire log from when I started the server, the setup stores
session data between the 3 servers, producing a load balancing effect.
the code is stored on a central server and is mounted via NFS to the 3
web servers, the code works fine on the other 2 so it must be related to
memcached on this particular server, however, the same version is in use
as on all the other servers, and this was working and suddenly broke :(

[dormando]

I'm not sure why you don't think it's functioning. I see the server
starting, then a *get* but no *sets*. Your mystery is probably in your
application setup somehow. You're probably swallowing some errors that
could be useful, or things aren't quite as synced up as you believe they
are.

And I don't care if the code is synced up or not, I'm curious as to what
your client configuration is for memcached. What is the *exact* server
list you're passing to the clients? Is it just localhost, a cluster, or
...?