Memcache : Slab Memory Allocation and Dynamic Expansion

[shubham srivastava]

I have a use case where I need couple of millions keys and the size of the values would be around 1mb. There would be another million keys who will be around 2.5mb in size. All my keys would have infinite expiration time.

Does the slab memory allocation implementation keep on adding memory to adhere the key/value pair. Intentionally would there be a limiting factor where I need lot of key/value pairs to be of similar size. I have all the memory to my disposal so creating memcache with 1mb or 2.5mb or 1G slab size won't be an issue anyways.

How does chunk size inside slab gets decided etc. If I have all the slabs with free_chunks as 0 then will I have another slab created etc.

I am coming from a problem faced by a colleague where his all the key/value sizes were around the same and he started to find lot of evictions happening.

Regards,

Shubham

[Yiftach Shoolman]

You can find a nice description on how slab allocator works here.

Also, remember that Memcached value size is limited to 1MB.

--
Yiftach Shoolman
+972-54-7634621

[shubham srivastava]

This actually asserts that if I have all my data of similar size and if it has no expiry then I will have lot of evictions. 

This seems to be a serious shortcoming :-> any alternate to support the same(all key/value of similar size) in Memcache.

Regards,

Shubham

[dormando]

What?

You set the max memory limit when you start the daemon. What specifically
is your problem? If you've assigned all of your slab pages into one slab
class, then start putting new items into another class, you'll need to
enable slab rebalancing. See the 1.4.11 release notes:

http://code.google.com/p/memcached/wiki/ReleaseNotes1411

1.4.14 (coming in the next day or two) has improved on this as well.