memcached and max connections (and may be some references to see where memory goes)

Den Samo

unread,

Dec 27, 2013, 8:04:00 PM12/27/13

to memc...@googlegroups.com

Hi everybody,

What would be best practice to set max connections? We have "memory leak" issue (memory usage grows, eats all available memory, process crashes). I use quotes as I am not sure that this is actually memory leak. I still thinking on the way to validate this. One potential suspect is memory used for network connections. We have -c 128000. And -m 24576 as memory limiter. the box itself has 48G of memory and dedicated to memcache.

Also, more abstract question: is there any way to see how memcache uses memory? like what amount was reserved for connections?

Thank you and Happy Holidays,

Den

dormando

unread,

Dec 27, 2013, 8:14:16 PM12/27/13

to memc...@googlegroups.com

> Hi everybody,What would be best practice to set max connections? We have "memory leak" issue (memory usage grows, eats all available memory,

Hi!

What version are you running? If it's not 1.4.17, can you try that version
and see if it still "leaks"? There was at least one bug fix in the last
two versions which could drop a connection structure.

The "stats" output can tell you how much memory is dictated for the slab
cache and for the hash table, but not all types of memory are accounted
presently.

That is a very large number of connections though. Can you provide some
'stats' output? How many connections do you usually have? Does your memory
usage continue to go up while the number of connections is stable, or do
the number of connections slowly increase with memory usage?

Den Samo

unread,

Dec 30, 2013, 1:31:16 PM12/30/13

to memc...@googlegroups.com

hi Dormando,

we use version 1.4.15. So it would be first step to update. Thank you!

>Does your memory usage continue to go up while the number of connections is stable, or do
>the number of connections slowly increase with memory usage?

I am doing history query now. But to be sure, do you mean "curr_connections" to compare or "connection_structures" or "total_connections" (this one has HUGE value, see below)

here is the stats from one of the servers:

stats

STAT pid 10409

STAT uptime 322887

STAT time 1388427859

STAT version 1.4.15

STAT libevent 1.4.13-stable

STAT pointer_size 64

STAT rusage_user 230345.511159

STAT rusage_system 354573.674608

STAT curr_connections 4151

STAT total_connections 1251550661

STAT connection_structures 6655

STAT reserved_fds 40

STAT cmd_get 26733462674

STAT cmd_set 291587557

STAT cmd_flush 0

STAT cmd_touch 0

STAT get_hits 26157644553

STAT get_misses 575818121

STAT delete_misses 6433865116

STAT delete_hits 91028651

STAT incr_misses 0

STAT incr_hits 0

STAT decr_misses 0

STAT decr_hits 0

STAT cas_misses 0

STAT cas_hits 28992

STAT cas_badval 22

STAT touch_hits 0

STAT touch_misses 0

STAT auth_cmds 0

STAT auth_errors 0

STAT bytes_read 7326099432348

STAT bytes_written 31415635467012

STAT limit_maxbytes 25769803776

STAT accepting_conns 1

STAT listen_disabled_num 0

STAT threads 8

STAT conn_yields 0

STAT hash_power_level 25

STAT hash_bytes 268435456

STAT hash_is_expanding 0

STAT slab_reassign_running 0

STAT slabs_moved 0

STAT bytes 22814358539

STAT curr_items 34689744

STAT total_items 291584654

STAT expired_unfetched 27821

STAT evicted_unfetched 91912345

STAT evictions 135352494

STAT reclaimed 34495

-Den

dormando

unread,

Dec 30, 2013, 2:32:10 PM12/30/13

to memc...@googlegroups.com

> hi Dormando,
> we use�version 1.4.15. So it would be first step to update. Thank you!
>
> >Does your memory�usage continue to go up while the number of connections is stable, or do�
> >the number of connections slowly increase with memory usage?�
>
> I am doing history query now. But to be sure, do you mean "curr_connections" to compare or "connection_structures" or "total_connections" (this one
> has HUGE value, see below)

curr_connections is the current number of connected clients.
connection_structure is something I forgot about: If that number goes
wacky that's a memory leak (but there's memory within the structures isn't
totalled up anywhere yet).
total_connections is ticked up every time a client connects. so a
connect/disconnect/connect ups it by 2.

I'm curious if your memory leaks as curr_connections grows, or if that
number is relatively static (going up and down with traffic levels) but
memory continually slowly increases.

Please let me know when you've tried 1.4.17 - and if it still leaks,
provide 'stats' output from that version. Thanks!

> --
> �
> ---
> You received this message because you are subscribed to the Google Groups "memcached" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to memcached+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

Den Samo

unread,

Dec 30, 2013, 3:22:11 PM12/30/13

to memc...@googlegroups.com

Dormando, thank you for so prompt reply.

So, my historical query does not show anything bad with curr_connections - typical saw without growth trend. So we update to 17 and do observations. Also, as for fixed bug: do you mean this one: https://github.com/memcached/memcached/commit/b2734f8321230bd52e36df7f82a6b1d71532e496 ?

On Monday, December 30, 2013 11:32:10 AM UTC-8, Dormando wrote:

> hi Dormando,
> we useï¿½version 1.4.15. So it would be first step to update. Thank you!
>
> >Does your memoryï¿½usage continue to go up while the number of connections is stable, or doï¿½
> >the number of connections slowly increase with memory usage?ï¿½

> ï¿½

dormando

unread,

Dec 30, 2013, 3:55:07 PM12/30/13

to memc...@googlegroups.com

On Mon, 30 Dec 2013, Den Samo wrote:

> Dormando, thank you for so prompt reply.So, my historical query does not show anything bad with�curr_connections - typical saw without growth

> trend. So we update to 17 and do observations. Also, as for fixed bug: do you mean this
> one:�https://github.com/memcached/memcached/commit/b2734f8321230bd52e36df7f82a6b1d71532e496 ?

That was one of them, yes. There's also some new counters and general
bugfixes; it's best to get a clean view of it.

> On Monday, December 30, 2013 11:32:10 AM UTC-8, Dormando wrote:
> > hi Dormando,

> > we use�version 1.4.15. So it would be first step to update. Thank you!

> >
> > >Does your memory�usage continue to go up while the number of connections is stable, or do�

> > >the number of connections slowly increase with memory usage?�

> > �

> > ---
> > You received this message because you are subscribed to the Google Groups "memcached" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to memcached+...@googlegroups.com.
> > For more options, visit https://groups.google.com/groups/opt_out.
> >
> >
>

> --
> �

Den Samo

unread,

Jan 24, 2014, 4:00:57 PM1/24/14

to memc...@googlegroups.com

Hi Dormando,

It took some time but we installed .17 and still observe the same issue with memory:

after 24 hours of run it consumed almost all memory from 48Gb available on dedicated box memcached process shows:

Rss 43499344

Shared 540

Private 43498804

Swap 0

Pss 43498845

=================

with limit set to 24Gb:

memcached -d -p 11211-m 24576 -c 128000 -P /var/run/memcached/memcached.pid -d -r -t 8 -o slab_reassign

stats are:

stats

STAT pid 9023

STAT uptime 133857

STAT time 1390591888

STAT version 1.4.17

STAT libevent 1.4.13-stable

STAT pointer_size 64

STAT rusage_user 153837.208189

STAT rusage_system 378991.192581

STAT curr_connections 6774

STAT total_connections 1155527288

STAT connection_structures 42638

STAT reserved_fds 40

STAT cmd_get 17749198141

STAT cmd_set 279314075

STAT cmd_flush 0

STAT cmd_touch 0

STAT get_hits 17065742324

STAT get_misses 683455817

STAT delete_misses 7573002573

STAT delete_hits 121708418

STAT incr_misses 0

STAT incr_hits 0

STAT decr_misses 0

STAT decr_hits 0

STAT cas_misses 1

STAT cas_hits 5887

STAT cas_badval 5

STAT touch_hits 0

STAT touch_misses 0

STAT auth_cmds 0

STAT auth_errors 0

STAT bytes_read 2295246013025

STAT bytes_written 27303765419361

STAT limit_maxbytes 25769803776

STAT accepting_conns 1

STAT listen_disabled_num 0

STAT threads 8

STAT conn_yields 0

STAT hash_power_level 25

STAT hash_bytes 268435456

STAT hash_is_expanding 0

STAT slab_reassign_running 0

STAT slabs_moved 0

STAT malloc_fails 0

STAT bytes 22652935663

STAT curr_items 31910169

STAT total_items 279313363

STAT expired_unfetched 35817

STAT evicted_unfetched 40144725

STAT evictions 81628302

STAT reclaimed 46197

Any idea where (and how) we can look where memory goes?

Thank you,

Den

On Monday, December 30, 2013 12:55:07 PM UTC-8, Dormando wrote:

On Mon, 30 Dec 2013, Den Samo wrote:

> Dormando, thank you for so prompt reply.So, my historical query does not show anything bad withï¿½curr_connections - typical saw without growth

> trend. So we update to 17 and do observations. Also, as for fixed bug: do you mean this

> one:ï¿½https://github.com/memcached/memcached/commit/b2734f8321230bd52e36df7f82a6b1d71532e496 ?

That was one of them, yes. There's also some new counters and general
bugfixes; it's best to get a clean view of it.

> On Monday, December 30, 2013 11:32:10 AM UTC-8, Dormando wrote:
> > hi Dormando,

> > we useï¿½version 1.4.15. So it would be first step to update. Thank you!
> >
> > >Does your memoryï¿½usage continue to go up while the number of connections is stable, or doï¿½
> > >the number of connections slowly increase with memory usage?ï¿½

> > ï¿½ ï¿½ ï¿½ > Hi everybody,What would be best practice to set max connections? We have "memory leak" issue (memory usage grows, eats all
> available
> > ï¿½ ï¿½ ï¿½ memory,
> > ï¿½ ï¿½ ï¿½ > process crashes). I use quotes as I am not sure that this is actually memory leak. I still thinking on the way to validate
> this. One
> > ï¿½ ï¿½ ï¿½ potential
> > ï¿½ ï¿½ ï¿½ > suspect is memory used for network connections. We have -c 128000. And -m 24576 as memory limiter. the box itself has 48G of
> memory
> > ï¿½ ï¿½ ï¿½ and dedicated
> > ï¿½ ï¿½ ï¿½ > to memcache.
> > ï¿½ ï¿½ ï¿½ >
> > ï¿½ ï¿½ ï¿½ > Also, more abstract question: is there any way to see how memcache uses memory? like what amount was reserved for
> connections?
> > ï¿½ ï¿½ ï¿½ >
> > ï¿½ ï¿½ ï¿½ > Thank you and Happy Holidays,
> > ï¿½ ï¿½ ï¿½ > Den
> >
> > ï¿½ ï¿½ ï¿½ Hi!
> >
> > ï¿½ ï¿½ ï¿½ What version are you running? If it's not 1.4.17, can you try that version
> > ï¿½ ï¿½ ï¿½ and see if it still "leaks"? There was at least one bug fix in the last
> > ï¿½ ï¿½ ï¿½ two versions which could drop a connection structure.
> >
> > ï¿½ ï¿½ ï¿½ The "stats" output can tell you how much memory is dictated for the slab
> > ï¿½ ï¿½ ï¿½ cache and for the hash table, but not all types of memory are accounted
> > ï¿½ ï¿½ ï¿½ presently.
> >
> > ï¿½ ï¿½ ï¿½ That is a very large number of connections though. Can you provide some
> > ï¿½ ï¿½ ï¿½ 'stats' output? How many connections do you usually have? Does your memory
> > ï¿½ ï¿½ ï¿½ usage continue to go up while the number of connections is stable, or do
> > ï¿½ ï¿½ ï¿½ the number of connections slowly increase with memory usage?
> >
> > --
> > ï¿½

> > ---
> > You received this message because you are subscribed to the Google Groups "memcached" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to memcached+...@googlegroups.com.
> > For more options, visit https://groups.google.com/groups/opt_out.
> >
> >
>
> --

> ï¿½

dormando

unread,

Jan 24, 2014, 4:16:14 PM1/24/14

to memc...@googlegroups.com

Hey,

I can add some more accounting this weekend and make a branch for you to
try. Do you have a better process for quickly iterating from now on? It's
going to suck if it's weeks between versions.

If the accounting doesn't catch it or make it obvious we might have to
take a core dump of a leaked process and examine it.

> --

Reply all

Reply to author

Forward