Why would I have a low percentage of used chunks for a slab class and still be getting evictions for it?
66 views
Skip to first unread message
red 888
Jun 21, 2017, 11:22:48 AM6/21/17
to memcached
Here is a snap shot of one of my slab class's stats (from PHPMemcachedAdmin):
So what I'm confused about is why, with my used_chunks so low, am I getting evictions for this slab class?
used_chunks are chunks with items that have not expired yet and total_chunks is all allocated chunks including chunks with items that _have_ expired correct?
If thats the case it means if 25% of the allocated chunks are "used" and 75% of the allocated chunks have expired. So it should have plenty of chunks to reclaim before having to evict I would think.
dormando
Jun 21, 2017, 2:27:48 PM6/21/17
to memcached
Is the evictions counter increasing all the time? It might only be
completely full sometimes, which would cause evictions (like during peak).
if used is less than total but evictions is going up, either you have an
old broken version of memcached or that phpmemcachedadmin thing is wrong,
and you should just look at the output of the "stats items" and "stats
slabs" commands.
On Wed, 21 Jun 2017, red 888 wrote:
> Here is a snap shot of one of my slab class's stats (from PHPMemcachedAdmin):
>
> [Capture.PNG]
>
> ---
> You received this message because you are subscribed to the Google Groups "memcached" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to memcached+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
completely full sometimes, which would cause evictions (like during peak).
if used is less than total but evictions is going up, either you have an
old broken version of memcached or that phpmemcachedadmin thing is wrong,
and you should just look at the output of the "stats items" and "stats
slabs" commands.
On Wed, 21 Jun 2017, red 888 wrote:
> Here is a snap shot of one of my slab class's stats (from PHPMemcachedAdmin):
>
>
>
> So what I'm confused about is why, with my used_chunks so low, am I getting evictions for this slab class?
>
>
> used_chunks are chunks with items that have not expired yet and total_chunks is all allocated chunks including chunks with items that _have_ expired
> correct?
>
>
> If thats the case it means if 25% of the allocated chunks are "used" and 75% of the allocated chunks have expired. So it should have plenty of chunks to
> reclaim before having to evict I would think.
>
>
>
> --
>
> So what I'm confused about is why, with my used_chunks so low, am I getting evictions for this slab class?
>
>
> used_chunks are chunks with items that have not expired yet and total_chunks is all allocated chunks including chunks with items that _have_ expired
> correct?
>
>
> If thats the case it means if 25% of the allocated chunks are "used" and 75% of the allocated chunks have expired. So it should have plenty of chunks to
> reclaim before having to evict I would think.
>
>
>
>
> ---
> You received this message because you are subscribed to the Google Groups "memcached" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to memcached+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
red 888
Jun 22, 2017, 10:27:12 AM6/22/17
to memc...@googlegroups.com
If I look at the slab stats over time I see the % of chunks_used (relative to total_chunks) spike up (some slab classes hit 45% some hit 75%) then spike down and at the same time they are spiking down evictions spike up. This happens about every 2-4 minutes.
I've attached a screen shot of this pattern in a time series (raw stats not phpmemchacheadmin). In the graphs I'm filtering for just one slab class but I see this for all the most active slabs ( roughly 5 active slab classes).
> To unsubscribe from this group and stop receiving emails from it, send an email to memcached+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
--
---
You received this message because you are subscribed to a topic in the Google Groups "memcached" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/memcached/VXm-hnGzqXY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to memcached+unsubscribe@googlegroups.com.
dormando
Jun 22, 2017, 12:44:28 PM6/22/17
to memc...@googlegroups.com
Looks like a weird access pattern. you're filling memory, evicting a bit,
then losing all of it at once. Is your "cmd_flush" counter increasing?
On Thu, 22 Jun 2017, red 888 wrote:
> If I look at the slab stats over time I see the % of chunks_used (relative to total_chunks) spike up (some slab classes hit 45% some hit 75%) then spike
> down and at the same time they are spiking down evictions spike up. This happens about every 2-4 minutes.
>
> I've attached a screen shot of this pattern in a time series (raw stats not phpmemchacheadmin). In the graphs I'm filtering for just one slab class but I
> see this for all the most active slabs ( roughly 5 active slab classes).
then losing all of it at once. Is your "cmd_flush" counter increasing?
On Thu, 22 Jun 2017, red 888 wrote:
> If I look at the slab stats over time I see the % of chunks_used (relative to total_chunks) spike up (some slab classes hit 45% some hit 75%) then spike
> down and at the same time they are spiking down evictions spike up. This happens about every 2-4 minutes.
>
> I've attached a screen shot of this pattern in a time series (raw stats not phpmemchacheadmin). In the graphs I'm filtering for just one slab class but I
> see this for all the most active slabs ( roughly 5 active slab classes).
> Inline image 2
>
> On Jun 21, 2017 2:27 PM, "dormando" <dorm...@rydia.net> wrote:
> Is the evictions counter increasing all the time? It might only be
> completely full sometimes, which would cause evictions (like during peak).
>
> if used is less than total but evictions is going up, either you have an
> old broken version of memcached or that phpmemcachedadmin thing is wrong,
> and you should just look at the output of the "stats items" and "stats
> slabs" commands.
>
> On Wed, 21 Jun 2017, red 888 wrote:
>
> > Here is a snap shot of one of my slab class's stats (from PHPMemcachedAdmin):
> >
> > [Capture.PNG]
> >
> >
> > So what I'm confused about is why, with my used_chunks so low, am I getting evictions for this slab class?
> >
> >
> > used_chunks are chunks with items that have not expired yet and total_chunks is all allocated chunks including chunks with items that
> _have_ expired
> > correct?
> >
> >
> > If thats the case it means if 25% of the allocated chunks are "used" and 75% of the allocated chunks have expired. So it should have plenty
> of chunks to
> > reclaim before having to evict I would think.
> >
> >
> >
> > --
> >
> > ---
> > You received this message because you are subscribed to the Google Groups "memcached" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to memcached+...@googlegroups.com.
>
> On Jun 21, 2017 2:27 PM, "dormando" <dorm...@rydia.net> wrote:
> Is the evictions counter increasing all the time? It might only be
> completely full sometimes, which would cause evictions (like during peak).
>
> if used is less than total but evictions is going up, either you have an
> old broken version of memcached or that phpmemcachedadmin thing is wrong,
> and you should just look at the output of the "stats items" and "stats
> slabs" commands.
>
> On Wed, 21 Jun 2017, red 888 wrote:
>
> > Here is a snap shot of one of my slab class's stats (from PHPMemcachedAdmin):
> >
> > [Capture.PNG]
> >
> >
> > So what I'm confused about is why, with my used_chunks so low, am I getting evictions for this slab class?
> >
> >
> > used_chunks are chunks with items that have not expired yet and total_chunks is all allocated chunks including chunks with items that
> _have_ expired
> > correct?
> >
> >
> > If thats the case it means if 25% of the allocated chunks are "used" and 75% of the allocated chunks have expired. So it should have plenty
> of chunks to
> > reclaim before having to evict I would think.
> >
> >
> >
> > --
> >
> > ---
> > You received this message because you are subscribed to the Google Groups "memcached" group.
> > For more options, visit https://groups.google.com/d/optout.
> >
> >
>
> --
>
> ---
> You received this message because you are subscribed to a topic in the Google Groups "memcached" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/memcached/VXm-hnGzqXY/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to memcached+...@googlegroups.com.
> >
> >
>
> --
>
> ---
> You received this message because you are subscribed to a topic in the Google Groups "memcached" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/memcached/VXm-hnGzqXY/unsubscribe.
> You received this message because you are subscribed to the Google Groups "memcached" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to memcached+...@googlegroups.com.
red 888
Jun 22, 2017, 2:24:04 PM6/22/17
to memcached
So when I look at my stats I see no cmd_flush commands are being run. cmd_flush is 0 and stays at 0.
dormando
Jun 22, 2017, 4:52:18 PM6/22/17
to memcached
Think I've asked this a few times; what version are you running?
red 888
Jun 22, 2017, 6:16:15 PM6/22/17
to memcached
Oh sorry! Version 1.4.34.
dormando
Jun 22, 2017, 7:53:56 PM6/22/17
to memcached
Can you paste the full output from the 'stats' command against the server?
Also: "stats settings" would be great.
Also; do you know anything about your data? what the TTL's are, for
instance? It's possible a bunch of stuff expires, which can get reclaimed
quickly by some processes depending on your start arguments.
Also: "stats settings" would be great.
Also; do you know anything about your data? what the TTL's are, for
instance? It's possible a bunch of stuff expires, which can get reclaimed
quickly by some processes depending on your start arguments.
red 888
Jun 22, 2017, 8:33:06 PM6/22/17
to memcached
The expiration for all items is 30 mins
=================== stats ==================================
STAT pid 1
STAT uptime 874624
STAT time 1498177737
STAT version 1.4.34
STAT libevent 2.0.21-stable
STAT pointer_size 64
STAT rusage_user 149928.880000
STAT rusage_system 20235.240000
STAT curr_connections 342
STAT total_connections 14816
STAT rejected_connections 0
STAT connection_structures 344
STAT reserved_fds 10
STAT cmd_get 2431931655
STAT cmd_set 1674656279
STAT cmd_flush 0
STAT cmd_touch 0
STAT cmd_config_get 109145
STAT cmd_config_set 2
STAT get_hits 1737530329
STAT get_misses 694401326
STAT get_expired 49107
STAT get_flushed 0
STAT delete_misses 0
STAT delete_hits 0
STAT incr_misses 0
STAT incr_hits 0
STAT decr_misses 0
STAT decr_hits 0
STAT cas_misses 4
STAT cas_hits 451106648
STAT cas_badval 530100
STAT touch_hits 0
STAT touch_misses 0
STAT auth_cmds 0
STAT auth_errors 0
STAT bytes_read 1319246358299
STAT bytes_written 1238308983641
STAT limit_maxbytes 6787432448
STAT launch_time_maxbytes 6787432448
STAT accepting_conns 1
STAT listen_disabled_num 0
STAT time_in_listen_disabled_us 0
STAT threads 2
STAT conn_yields 0
STAT hash_power_level 22
STAT hash_bytes 33554432
STAT hash_is_expanding 0
STAT curr_config 1
STAT slab_reassign_rescues 0
STAT slab_reassign_chunk_rescues 0
STAT slab_reassign_evictions_nomem 0
STAT slab_reassign_inline_reclaim 0
STAT slab_reassign_busy_items 0
STAT slab_reassign_running 0
STAT slabs_moved 0
STAT lru_crawler_running 0
STAT lru_crawler_starts 629012
STAT lru_maintainer_juggles 513869396
STAT malloc_fails 0
STAT log_worker_dropped 0
STAT log_worker_written 0
STAT log_watcher_skipped 0
STAT log_watcher_sent 0
STAT bytes 2520842198
STAT curr_items 2234247
STAT total_items 1674126175
STAT expired_unfetched 48225226
STAT evicted_unfetched 0
STAT evicted_active 0
STAT evictions 0
STAT reclaimed 447052919
STAT crawler_reclaimed 437349953
STAT crawler_items_checked 217893516538
STAT lrutail_reflocked 12575
STAT moves_to_cold 916002463
STAT moves_to_warm 143197938
STAT moves_within_lru 655572218
STAT direct_reclaims 0
==================================== stats settings =================================================
stats settings
STAT maxbytes 6787432448
STAT launch_time_maxbytes 6787432448
STAT maxconns 65000
STAT tcpport 11211
STAT udpport 11211
STAT inter NULL
STAT verbosity 1
STAT oldest 0
STAT evictions on
STAT domain_socket NULL
STAT umask 700
STAT growth_factor 1.15
STAT chunk_size 5
STAT num_threads 2
STAT num_threads_per_udp 2
STAT stat_key_prefix :
STAT detail_enabled no
STAT reqs_per_event 20
STAT cas_enabled yes
STAT tcp_backlog 1024
STAT binding_protocol auto-negotiate
STAT auth_enabled_sasl no
STAT item_size_max 1048576
STAT config_max 16
STAT config_size_max 65536
STAT maxconns_fast yes
STAT hashpower_init 16
STAT slab_reassign yes
STAT slab_automove 0
STAT slab_chunk_max 524288
STAT lru_crawler yes
STAT lru_crawler_sleep 100
STAT lru_crawler_tocrawl 0
STAT tail_repair_time 0
STAT flush_enabled yes
STAT dump_enabled yes
STAT hash_algorithm murmur3
STAT lru_maintainer_thread yes
STAT hot_lru_pct 32
STAT warm_lru_pct 32
STAT expirezero_does_not_evict no
STAT idle_timeout 0
STAT watcher_logbuf_size 262144
STAT worker_logbuf_size 65536
STAT track_sizes no
dormando
Jun 22, 2017, 9:01:50 PM6/22/17
to memcached
Nice. Looks like you actually have new features enabled.
> STAT crawler_reclaimed 437349953
> STAT crawler_items_checked 217893516538
These stats tell me what I was saying before; you probably have a bunch of
expired stuff in memory, when the crawler wakes up it reclaims a bunch of
it. The spikes you're seeing may be because the crawler is sleeping too
long, or your traffic comes in bursts as well.
Some scheduling fixes for the LRU crawler went into the latest version, I
think. Should smooth it out a bit.
you can also do a magic trick:
telnet to your server, and type "watch evictions" - which will show you
exactly what the objects getting evicted look like.
> STAT crawler_reclaimed 437349953
> STAT crawler_items_checked 217893516538
expired stuff in memory, when the crawler wakes up it reclaims a bunch of
it. The spikes you're seeing may be because the crawler is sleeping too
long, or your traffic comes in bursts as well.
Some scheduling fixes for the LRU crawler went into the latest version, I
think. Should smooth it out a bit.
you can also do a magic trick:
telnet to your server, and type "watch evictions" - which will show you
exactly what the objects getting evicted look like.
red 888
Jun 26, 2017, 8:12:36 AM6/26/17
to memcached
Thanks for your help on this! So I'm currently bound to this version of memcached (I'm using AWS Elasticache and this is the latest version they support) so will have to make do for now.
It seems before I try decreasing the item expiration time I should experiment with tweaking the LRU sleep time? Would cutting the sleep time say in half be safe or is there a danger there?
Also, I found a new memcached term I can't find a lot of info on if you'll indulge me- what exactly are "reflocks"?
dormando
Jun 26, 2017, 6:31:16 PM6/26/17
to memcached
Also before you decrease the item expiration time; are the evictions that
bad? You can look at the "eviction age" for the different classes, which
says how long ago the last evicted item was accessed before being ejected.
Typically evictions are on the least used items, so they've been in the
cache the longest anyway. If you get benefit out of items typically being
able to store for 30 minutes, keep at it.
You can cut the LRU sleep to 0 and it's fine, but that's just the wait
between checks, not the scheduling before the next run.
reflocks you can ignore. it's mostly useful for me when looking at bug
reports. It just means when something looked at the tail, it was
refcounted. In old code that used to stop progress, but for the last few
years it's fine.
did you confirm if your traffic is spikey or not?
bad? You can look at the "eviction age" for the different classes, which
says how long ago the last evicted item was accessed before being ejected.
Typically evictions are on the least used items, so they've been in the
cache the longest anyway. If you get benefit out of items typically being
able to store for 30 minutes, keep at it.
You can cut the LRU sleep to 0 and it's fine, but that's just the wait
between checks, not the scheduling before the next run.
reflocks you can ignore. it's mostly useful for me when looking at bug
reports. It just means when something looked at the tail, it was
refcounted. In old code that used to stop progress, but for the last few
years it's fine.
did you confirm if your traffic is spikey or not?
Reply all
Reply to author
Forward
0 new messages