How to determine if memcache is full

[Eduardo SIlvestre]

Hello everyone,

every weeks my memcache server stop accepting more connections. Today
before restart daemon, i've check stats.

stats
STAT pid 30026
STAT uptime 938964
STAT time 1306667508
STAT version 1.2.2
STAT pointer_size 64
STAT rusage_user 682.422648
STAT rusage_system 1636.058247
STAT curr_items 1630320
STAT total_items 44025061
STAT bytes 1395547058
STAT curr_connections 1
STAT total_connections 49126599
STAT connection_structures 348
STAT cmd_get 52910303
STAT cmd_set 44025061
STAT get_hits 42444282
STAT get_misses 10466021
STAT evictions 0
STAT bytes_read 45726499071
STAT bytes_written 41367374940
STAT limit_maxbytes 1610612736
STAT threads 1
END

My memcached.conf

# Run memcached as a daemon. This command is implied, and is not
needed for the
# daemon to run. See the README.Debian that comes with this package
for more
# information.
-d

# Log memcached's output to /var/log/memcached
logfile /var/log/memcached.log

# Be verbose
# -v

# Be even more verbose (print client commands as well)
# -vv

# Start with a cap of 64 megs of memory. It's reasonable, and the
daemon default
# Note that the daemon will grow to this size, but does not start out
holding this much
# memory
-m 2048

# Default connection port is 11211
-p 11211

# Run the daemon as root. The start-memcached will default to running
as root if no
# -u command is present in this config file
-u nobody

# Specify which IP address to listen on. The default is to listen on
all IP addresses
# This parameter is one of the only security measures that memcached
has, so make sure
# it's listening on a firewalled interface.
-l 192.168.0.14

# Limit the number of simultaneous incoming connections. The daemon
default is 1024
-c 64000

# Lock down all paged memory. Consult with the README and homepage
before you do this
# -k

# Return error when memory is exhausted (rather than removing items)
-M

# Maximize core file limit
# -r

Memcache information at php.ini:

; Handler used to store/retrieve data.
session.save_handler = memcache
session.save_path = "tcp://192.168.0.14:11211"

How can i determine if memcache is full or when it is close to 80%? My
idea is to try to clear all information from the memcache with more
than 24 hours or something like that.

Best Regards,

--eduardo

[dormando]

> Hello everyone,
>
> every weeks my memcache server stop accepting more connections. Today
> before restart daemon, i've check stats.
>
> stats
> STAT pid 30026
> STAT uptime 938964
> STAT time 1306667508
> STAT version 1.2.2

Please upgrade to a newer version :) That one has grown a lot of hair,
memcached does not stop accepting connections when it gets full. It's
supposed to do more useful things instead.

-Dormando

[Eduardo Silvestre]

Hello Dormando,

 thans for your feedback. In fact i'm using the last stable version at debian repositories (http://packages.debian.org/lenny/memcached). Why will no longer accept new connections?  Can i determine the cause based on stats?

I'm collecting data from memcache with cacti templates, and can't find any reason for this situation.

Best Regards,

[Artur Ejsmont]

I don't know if possible maybe you have Apache threads mounting up on web servers to the default limit of 1024 sockets on the memcached? Used to happen to us a few times on DB timeouts.

We also used to have crashes of memcached from time to time so maybe buggy version?

Art

[Marc Bollinger]

> In fact i'm using the last stable version at debian repositories (http://packages.debian.org/lenny/memcached).

Then also upgrade your OS to Squeeze (latest stable) :)
http://packages.debian.org/squeeze/memcached

- Marc

On Mon, May 30, 2011 at 6:12 AM, Eduardo Silvestre <jsilv...@gmail.com> wrote:
> Hello Dormando,
>

>  thans for your feedback.Why will

[dormando]

> Hello Dormando,
>
> �thans for your feedback. In fact i'm using the last stable version at debian repositories (http://packages.debian.org/lenny/memcached). Why will no
> longer accept new connections?� Can i determine the cause based on stats?
>
> I'm collecting data from memcache with cacti templates, and can't find any reason for this situation.

Lenny is too old to support; the cause is because you're 10+ revisions
behind and we fixed too many crashes to list. We're on 1.4.5 (soon to be
1.4.6 or 1.6.0). You can try to fetch it from backports or build yourself.

[PK Hunter]

Actually I joined this google group to ask the same thing, and found this thread. 

I start memcached on a CentOS 64 bit server with 8 GB of RAM, with the following settings: 

    memcached -d -m 96 -n 10 -c 4096 -f 1.05 -l 127.0.0.1 -p 11211 

Yet, the memcached on my server has just about 2,600 keys, and seeing the actual number of bytes stores including the characters needed to store keys + their values, it is 632,817. Which I understand is about 0.60 MB, right?

What am I missing?

If I manually try to add keys, and I wrote a PHP program using the memcache library to add 6,000 keys, the keys do NOT get added, and also the "$status ["evictions"]" remains at 0. So I'm not sure why the server is stopping at circa 2,600 mark.

Any ideas would be much appreciated!

These are the stats of my server:

Memcache Server version: 1.4.5 

Process id of this server process 21808

Number of seconds this server has been running 200229

Accumulated user time for this process 48.654603 seconds

Accumulated system time for this process 198.973751 seconds

Total number of items stored by this server ever since it started 637981

Number of open connections 6

Total number of connections opened since the server started running 1505905

Number of connection structures allocated by the server 70

Cumulative number of retrieval requests 2657551

Cumulative number of storage requests 637981

Number of keys that have been requested and found present 2507883 (94.4%)

Number of items that have been requested and not found 149668(5.6%)

Total number of bytes read by this server from network 97.1105 MB

Total number of bytes sent by this server to network 3839.72 MB

Number of bytes this server is allowed to use for storage. 96 MB

Number of valid items removed from cache to free memory for new items. 0

 

[Eduardo Silvestre]

Hello,

 every weeks we need restart memcached daemon... I try do flush_all with no lucky.  Do you know other command to flush ?

nobody   15555  0.1  6.8 586716 564768 ?       S    Jun06   5:45 /usr/bin/memcached -m 2048 -p 11211 -u nobody -l 192.168.52.52 -c 64000 -M

stats
STAT pid 15555
STAT uptime 219153
STAT time 1307613237

STAT version 1.2.2
STAT pointer_size 64

STAT rusage_user 151.757484
STAT rusage_system 338.165134
STAT curr_items 447844
STAT total_items 9176979
STAT bytes 498694125
STAT curr_connections 6
STAT total_connections 9658642
STAT connection_structures 190
STAT cmd_get 10255192
STAT cmd_set 9176979
STAT get_hits 8671302
STAT get_misses 1583890
STAT evictions 0
STAT bytes_read 11289578042
STAT bytes_written 10440713334
STAT limit_maxbytes 2147483648
STAT threads 1
END

(graph of my memcached attached)

In last version this problem is fixed?

Best Regards,

[Geoffrey Hoffman]

PK - 

Just a side point here, but:

> memcached -d -m 96 -n 10 -c 4096 -f 1.05 -l 127.0.0.1 -p 11211

-m <num>

Use <num> MB memory max to use for object storage; the default is 64 megabytes.

-c <num>

Use <num> max simultaneous connections; the default is 1024.

So, my question is, you're starting memcached on a server with 8GB of RAM to use only 96 mb of with 4096 connections? 

Your question is probably still valid and unanswered but for what it's worth if you do -m 1024 (1GB) it'll take over 10 times longer to fill up...

> the actual number of bytes stores including the characters needed to store keys + their values, it is 632,817.

Yeah, 63K - so 96mb ought to be plenty anyway - where do you see that?

I know that Redis is recommended 32-bit, so I checked if there are differences between 32-bit and 64-bit memcached.

Based on: http://serverfault.com/questions/172237/memcached-memory-usage-difference-on-32-and-64-bit

...it appears to be likely that memcached has to allocate more blocks to store the same data on a 64-bit machine than on a 32-bit machine, though if you really only have 64K of data it should all be a moot point.

[Adam Lee]

your email and your pasted stats seem to greatly disagree...  you're allocating 96MB for memcached and that's how much it thinks it's stored.  it also says it has stored 630k items- where are you getting 2,600?

also, as an aside, does your test program only store one size of item?

awl

[Adam Lee]

why do you need to restart it?  you're telling it that it's allowed to use up to 2G and it never breaks that...  flush_all only flushes the cache, it doesn't deallocate memory.

awl

[Eduardo Silvestre]

Hello Adam,

 it's possible deallocate memory on the fly ? Did i need change something ? Or upgrade memcache version ?

Regards,

[Dustin]

On Jun 9, 11:58 am, Eduardo Silvestre <jsilves...@gmail.com> wrote:
> Hello Adam,
>
>  it's possible deallocate memory on the fly ?

It is generally not possible to return memory to the system (from
malloc using sbrk anyway).

You don't need to, though. Give memcached as much memory as you
think it should have. Don't worry about what it does with it.