Question Regarding Slab Imbalance
52 views
Skip to first unread message
Tony Wu
Nov 12, 2020, 5:09:46 PM11/12/20
to memcached
We are currently using memcached provided by AWS ElastiCache service in modern mode to store web session keys among other things. We are observing session keys being evicted before TTL even though the cluster seems to have ample free memory, which hints at slab imbalance. I've included a print out of slab stats below in terms of sizing and total / free pages.
What would be the best way to smooth out this imbalance? We already enabled slab reassign / auto-move under modern mode.
Thanks.
STAT 1:chunk_size 96
STAT 1:total_chunks 131064
STAT 1:free_chunks 1664
STAT 1:free_chunks_end 0
STAT 2:chunk_size 104
STAT 2:total_chunks 60492
STAT 2:free_chunks 5327
STAT 2:free_chunks_end 0
STAT 3:chunk_size 112
STAT 3:total_chunks 46810
STAT 3:free_chunks 79
STAT 3:free_chunks_end 0
STAT 4:chunk_size 120
STAT 4:total_chunks 26214
STAT 4:free_chunks 48
STAT 4:free_chunks_end 0
STAT 5:chunk_size 128
STAT 5:total_chunks 32768
STAT 5:free_chunks 6193
STAT 5:free_chunks_end 0
STAT 6:chunk_size 136
STAT 6:total_chunks 23130
STAT 6:free_chunks 4594
STAT 6:free_chunks_end 0
STAT 7:chunk_size 144
STAT 7:total_chunks 7281
STAT 7:free_chunks 5894
STAT 7:free_chunks_end 0
STAT 8:chunk_size 152
STAT 8:total_chunks 13796
STAT 8:free_chunks 12008
STAT 8:free_chunks_end 0
STAT 9:chunk_size 160
STAT 9:total_chunks 6553
STAT 9:free_chunks 6316
STAT 9:free_chunks_end 0
STAT 10:chunk_size 168
STAT 10:total_chunks 6241
STAT 10:free_chunks 6186
STAT 10:free_chunks_end 0
STAT 11:chunk_size 176
STAT 11:total_chunks 5957
STAT 11:free_chunks 5946
STAT 11:free_chunks_end 0
STAT 12:chunk_size 184
STAT 12:total_chunks 5698
STAT 12:free_chunks 5664
STAT 12:free_chunks_end 0
STAT 13:chunk_size 192
STAT 13:total_chunks 5461
STAT 13:free_chunks 5380
STAT 13:free_chunks_end 0
STAT 14:chunk_size 200
STAT 14:total_chunks 5242
STAT 14:free_chunks 5135
STAT 14:free_chunks_end 0
STAT 15:chunk_size 208
STAT 15:total_chunks 5041
STAT 15:free_chunks 4892
STAT 15:free_chunks_end 0
STAT 16:chunk_size 216
STAT 16:total_chunks 4854
STAT 16:free_chunks 4770
STAT 16:free_chunks_end 0
STAT 17:chunk_size 224
STAT 17:total_chunks 4681
STAT 17:free_chunks 4655
STAT 17:free_chunks_end 0
STAT 18:chunk_size 232
STAT 18:total_chunks 4519
STAT 18:free_chunks 4489
STAT 18:free_chunks_end 0
STAT 19:chunk_size 240
STAT 19:total_chunks 4369
STAT 19:free_chunks 4345
STAT 19:free_chunks_end 0
STAT 20:chunk_size 248
STAT 20:total_chunks 4228
STAT 20:free_chunks 4217
STAT 20:free_chunks_end 0
STAT 21:chunk_size 256
STAT 21:total_chunks 4096
STAT 21:free_chunks 4091
STAT 21:free_chunks_end 0
STAT 22:chunk_size 264
STAT 22:total_chunks 3971
STAT 22:free_chunks 3966
STAT 22:free_chunks_end 0
STAT 23:chunk_size 272
STAT 23:total_chunks 3855
STAT 23:free_chunks 3851
STAT 23:free_chunks_end 0
STAT 24:chunk_size 280
STAT 24:total_chunks 3744
STAT 24:free_chunks 3738
STAT 24:free_chunks_end 0
STAT 25:chunk_size 288
STAT 25:total_chunks 3640
STAT 25:free_chunks 3638
STAT 25:free_chunks_end 0
STAT 26:chunk_size 296
STAT 26:total_chunks 3542
STAT 26:free_chunks 3533
STAT 26:free_chunks_end 0
STAT 27:chunk_size 304
STAT 27:total_chunks 3449
STAT 27:free_chunks 3448
STAT 27:free_chunks_end 0
STAT 28:chunk_size 312
STAT 28:total_chunks 3360
STAT 28:free_chunks 3350
STAT 28:free_chunks_end 0
STAT 29:chunk_size 320
STAT 29:total_chunks 3276
STAT 29:free_chunks 3273
STAT 29:free_chunks_end 0
STAT 30:chunk_size 328
STAT 30:total_chunks 3196
STAT 30:free_chunks 3174
STAT 30:free_chunks_end 0
STAT 31:chunk_size 336
STAT 31:total_chunks 3120
STAT 31:free_chunks 3118
STAT 31:free_chunks_end 0
STAT 32:chunk_size 344
STAT 32:total_chunks 3048
STAT 32:free_chunks 3043
STAT 32:free_chunks_end 0
STAT 33:chunk_size 352
STAT 33:total_chunks 2978
STAT 33:free_chunks 2954
STAT 33:free_chunks_end 0
STAT 34:chunk_size 360
STAT 34:total_chunks 2912
STAT 34:free_chunks 2903
STAT 34:free_chunks_end 0
STAT 35:chunk_size 368
STAT 35:total_chunks 2849
STAT 35:free_chunks 2828
STAT 35:free_chunks_end 0
STAT 36:chunk_size 376
STAT 36:total_chunks 2788
STAT 36:free_chunks 2766
STAT 36:free_chunks_end 0
STAT 37:chunk_size 384
STAT 37:total_chunks 2730
STAT 37:free_chunks 2684
STAT 37:free_chunks_end 0
STAT 38:chunk_size 392
STAT 38:total_chunks 2674
STAT 38:free_chunks 2621
STAT 38:free_chunks_end 0
STAT 39:chunk_size 400
STAT 39:total_chunks 2621
STAT 39:free_chunks 2564
STAT 39:free_chunks_end 0
STAT 40:chunk_size 408
STAT 40:total_chunks 2570
STAT 40:free_chunks 2527
STAT 40:free_chunks_end 0
STAT 41:chunk_size 416
STAT 41:total_chunks 2520
STAT 41:free_chunks 2216
STAT 41:free_chunks_end 0
STAT 42:chunk_size 424
STAT 42:total_chunks 2473
STAT 42:free_chunks 2256
STAT 42:free_chunks_end 0
STAT 43:chunk_size 432
STAT 43:total_chunks 2427
STAT 43:free_chunks 2299
STAT 43:free_chunks_end 0
STAT 44:chunk_size 440
STAT 44:total_chunks 2383
STAT 44:free_chunks 1898
STAT 44:free_chunks_end 0
STAT 45:chunk_size 448
STAT 45:total_chunks 2340
STAT 45:free_chunks 1962
STAT 45:free_chunks_end 0
STAT 46:chunk_size 456
STAT 46:total_chunks 2299
STAT 46:free_chunks 2192
STAT 46:free_chunks_end 0
STAT 47:chunk_size 472
STAT 47:total_chunks 2221
STAT 47:free_chunks 2061
STAT 47:free_chunks_end 0
STAT 48:chunk_size 488
STAT 48:total_chunks 2148
STAT 48:free_chunks 2047
STAT 48:free_chunks_end 0
STAT 49:chunk_size 504
STAT 49:total_chunks 2080
STAT 49:free_chunks 1955
STAT 49:free_chunks_end 0
STAT 50:chunk_size 520
STAT 50:total_chunks 2016
STAT 50:free_chunks 1788
STAT 50:free_chunks_end 0
STAT 51:chunk_size 536
STAT 51:total_chunks 1956
STAT 51:free_chunks 1870
STAT 51:free_chunks_end 0
STAT 52:chunk_size 552
STAT 52:total_chunks 1899
STAT 52:free_chunks 1829
STAT 52:free_chunks_end 0
STAT 53:chunk_size 568
STAT 53:total_chunks 1846
STAT 53:free_chunks 1762
STAT 53:free_chunks_end 0
STAT 54:chunk_size 584
STAT 54:total_chunks 1795
STAT 54:free_chunks 1613
STAT 54:free_chunks_end 0
STAT 55:chunk_size 600
STAT 55:total_chunks 1747
STAT 55:free_chunks 1446
STAT 55:free_chunks_end 0
STAT 56:chunk_size 616
STAT 56:total_chunks 1702
STAT 56:free_chunks 1358
STAT 56:free_chunks_end 0
STAT 57:chunk_size 632
STAT 57:total_chunks 1659
STAT 57:free_chunks 1497
STAT 57:free_chunks_end 0
STAT 58:chunk_size 648
STAT 58:total_chunks 1618
STAT 58:free_chunks 1398
STAT 58:free_chunks_end 0
STAT 59:chunk_size 664
STAT 59:total_chunks 3158
STAT 59:free_chunks 1784
STAT 59:free_chunks_end 0
STAT 60:chunk_size 680
STAT 60:total_chunks 1542
STAT 60:free_chunks 1301
STAT 60:free_chunks_end 0
STAT 61:chunk_size 696
STAT 61:total_chunks 3012
STAT 61:free_chunks 302
STAT 61:free_chunks_end 0
STAT 62:chunk_size 712
STAT 62:total_chunks 4416
STAT 62:free_chunks 631
STAT 62:free_chunks_end 0
STAT 63:chunk_size 524288
STAT 63:total_chunks 26108
STAT 63:free_chunks 0
STAT 63:free_chunks_end 0
dormando
Nov 12, 2020, 5:27:46 PM11/12/20
to memcached
What version are they running now? That stat output looks pretty sparse.
Unfortunately when it comes to elasticache the answer is probably to just
beg them to upgrade to a newer version. they tend to run really old and
newer verisons do a lot better at balancing memory.
This is also suspect:
of memory. If you're setting the slab multiplier (-f) to something really
aggressive, you should stop that :)
> --
>
> ---
> You received this message because you are subscribed to the Google Groups "memcached" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> memcached+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/memcached/927615f7-187d-4c8f-9e08-f1f341b9a04an%40googlegroups.com.
>
>
Unfortunately when it comes to elasticache the answer is probably to just
beg them to upgrade to a newer version. they tend to run really old and
newer verisons do a lot better at balancing memory.
This is also suspect:
> STAT 62:chunk_size 712
> STAT 62:total_chunks 4416
> STAT 62:free_chunks 631
> STAT 62:free_chunks_end 0
> STAT 63:chunk_size 524288
anything you store that's larger than 712 bytes will use half a megabyte
> STAT 62:total_chunks 4416
> STAT 62:free_chunks 631
> STAT 62:free_chunks_end 0
> STAT 63:chunk_size 524288
of memory. If you're setting the slab multiplier (-f) to something really
aggressive, you should stop that :)
>
> ---
> You received this message because you are subscribed to the Google Groups "memcached" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> memcached+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/memcached/927615f7-187d-4c8f-9e08-f1f341b9a04an%40googlegroups.com.
>
>
Tony Wu
Nov 12, 2020, 6:35:40 PM11/12/20
to memc...@googlegroups.com
Hi Dormando,
Thanks for the reply, I believe the engine version is 1.5. By multiplier I assumed you meant “chunk_size_growth_factor”, which is currently set at 1.02.
Best,
Tony Wu
> To view this discussion on the web visit https://groups.google.com/d/msgid/memcached/alpine.DEB.2.22.394.2011121424300.2800104%40dskull.
Thanks for the reply, I believe the engine version is 1.5. By multiplier I assumed you meant “chunk_size_growth_factor”, which is currently set at 1.02.
Best,
Tony Wu
dormando
Nov 12, 2020, 6:50:47 PM11/12/20
to memc...@googlegroups.com
1.5.what?
also yes, chunk_size_growth_factor. don't set it to 1.02. set it to
something so your slab classes are more evenly distributed. the max is 63.
> To view this discussion on the web visit https://groups.google.com/d/msgid/memcached/76D01DF5-D89D-4CF1-B786-3F1B8FA9B2A0%40gmail.com.
>
also yes, chunk_size_growth_factor. don't set it to 1.02. set it to
something so your slab classes are more evenly distributed. the max is 63.
>
Tony Wu
Nov 12, 2020, 6:57:52 PM11/12/20
to memc...@googlegroups.com
Hi,
The engine version we have running right now is 1.5.16. I just saw an announcement that 1.6.6 is available, so we’ll probably plan to migrate.
Adjusting growth factor makes sense. I did a quick google and don’t see any definitive formula to calculate an appropriate number. I’ll probably start with 2 or 3 and adjust it from there.
Thanks,
Tony Wu
> To view this discussion on the web visit https://groups.google.com/d/msgid/memcached/alpine.DEB.2.22.394.2011121549540.2800104%40dskull.
The engine version we have running right now is 1.5.16. I just saw an announcement that 1.6.6 is available, so we’ll probably plan to migrate.
Adjusting growth factor makes sense. I did a quick google and don’t see any definitive formula to calculate an appropriate number. I’ll probably start with 2 or 3 and adjust it from there.
Thanks,
Tony Wu
dormando
Nov 12, 2020, 7:09:53 PM11/12/20
to memc...@googlegroups.com
what's the default? you probably don't need to change it from the default
at all, which should have a good spread. default is 1.25 I think.
> To view this discussion on the web visit https://groups.google.com/d/msgid/memcached/0A14BD5F-5023-4FDB-A725-B3F6E6E2A37F%40gmail.com.
>
at all, which should have a good spread. default is 1.25 I think.
>
Tony Wu
Nov 12, 2020, 7:12:34 PM11/12/20
to memc...@googlegroups.com
Yeah, default is 1.25. We can certainly start with 1.25.
Thanks again,
Tony Wu
> To view this discussion on the web visit https://groups.google.com/d/msgid/memcached/alpine.DEB.2.22.394.2011121608080.2800104%40dskull.
Thanks again,
Tony Wu
Reply all
Reply to author
Forward
0 new messages