Memcached allocation errors

[Bram]

Hi,

We're using memcached to cache content generated by another
application on the same server. Items that are cached are set to never
expire, as the application currently self decides when it will refresh
items already in the cache.

We run memcache with verbose output, and I occasionally see these one
or more entries in stderr:

Couldn't realloc input buffer

Sometimes, the memcached process will disappear, though I am not
seeing anything about that in the stderr/stdout output, other than the
message above. Sometimes there can be multiple realloc error messages
in stderr with memcached still running.

It would appear that these issues only start creeping up once memcache
has reached its full capacity and starts evicting old entries. by
then, pyblibmc (which uses libmemcached) also occasionally produces
these errors:

ProtocolError: error 8 from memcached_set: PROTOCOL
ERROR
WriteError: error 5 from memcached_set: WRITE
FAILURE

Restarting memcached will stop the flow of errors until its full
capacity has been reached.

We run memcached as follows:

memcached -v -l 127.0.0.1:11211 -d -m 1024 -c 2000 -P /home/user/var/
run/memcached.pid

We're running version 1.4.10 on a debian lenny virtual server (OpenVZ)
on a RedHat 5.4 host (all 32-bits)

Memcached is run with ulimit -v 3145728 and ulimit -n 2048

On average, it has only about 10 active connections. Stats output (of
an instance that is currently showing these errors):

Server: 127.0.0.1 (11211)
pid: 27796
uptime: 106129
time: 1325774833
version: 1.4.10
libevent: 1.3e
pointer_size: 32
rusage_user: 66.145944
rusage_system: 380.068220
curr_connections: 9
total_connections: 4891692
connection_structures: 189
reserved_fds: 20
cmd_get: 3943901
cmd_set: 1128643
cmd_flush: 0
cmd_touch: 0
get_hits: 3819106
get_misses: 124795
delete_misses: 0
delete_hits: 0
incr_misses: 0
incr_hits: 0
decr_misses: 0
decr_hits: 0
cas_misses: 0
cas_hits: 0
cas_badval: 0
touch_hits: 0
touch_misses: 0
auth_cmds: 0
auth_errors: 0
bytes_read: 11143419126
bytes_written: 28858478649
limit_maxbytes: 1073741824

Any tips?

[dormando]

Can you try 1.4.9? You may also be low on system memory. If downgrading
doesn't work, reduce the memory limit from -m

[Bram]

I just upgraded to 1.4.10 from 1.4.5. In 1.4.5 we used a unix socket
instead of tcp, and memcached would crash as soon as a single
"Couldn't realloc input buffer" error ocurred.

The server has plenty of memory and is monitored by nagios which
doesn't show any oddities at the time of the crashes. The only thing
that worries me is that openvz might have something to do with it?
Although with these generous limits and few connections I can't think
of anything.

Taking a quick look at the source code at the realloc error message
shows that it's a buffer used to read from a connection to the
memcached. It doubles this buffer in size whenever not enough memory
is available to read data from the connection. It starts at 2K. I
cannot imagine that any connection could have so much data going over
it that this would ever be a problem?? Very strange.. maybe I should
add extra debugging info there displaying the amount of memory it
tried to allocate.

For now I have just made sure that all the objects do have an expiry
time set so that it should never hit full capacity. I'll update here
if this at least solves the crashing / protocol / memory errors.

[dormando]

Can you try the 1.4.11-beta1 cut:
http://code.google.com/p/memcached/wiki/ReleaseNotes1411beta1

I closed a few bugs related to 1.4.10. Would be very interested to see if
you still have issues with it, and what the exact errors are.

If you could get it to crash under gdb and get a backtrace as well, that's
major bonus points.

Thanks, and sorry,
-Dormando