Why there is no memory pool in Redis?

[slade]

As we know, the use of general memory allocator such as ptmalloc will result in memory fragmentation problem. Memcached uses slab allocator to deal with this. My question is why there is no memory pool in Redis?  Is using jemalloc or tcmalloc instead of ptmalloc enough to cope with memory fragmentation problem?

Thank You very much!

[Josiah Carlson]

By default, Redis uses the jemalloc memory allocator, though you can force it to use tcmalloc with a compiler flag. If you look at the design of jemalloc, it uses miniature pools to satisfy memory allocations. Before worrying too much about which allocator Redis uses, you should read the papers and docs about jemalloc.

For Redis specifically, few people have issues with jemalloc and fragmentation, and those that do would have been in a worse situation with *any* of the other allocators currently available.

 - Josiah

On Mon, Aug 25, 2014 at 3:53 AM, slade <zju...@gmail.com> wrote:

As we know, the use of general memory allocator such as ptmalloc will result in memory fragmentation problem. Memcached uses slab allocator to deal with this. My question is why there is no memory pool in Redis?  Is using jemalloc or tcmalloc instead of ptmalloc enough to cope with memory fragmentation problem?

Thank You very much!

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[slade]

Josiah, thanks for your reply, I will read some docs about jemalloc. BTW, how do you think that memcached uses its own memory pool rather than jemalloc. Could the particular pool bring some distinct improvement on memory allocation?
 
在 2014年8月25日星期一UTC+8下午11时10分05秒，Josiah Carlson写道：

[Josiah Carlson]

There are several major differences between the way jemalloc handles its miniature pools compared to how Memcached handles its slabs, the biggest of which being that Memcached never releases memory back to the OS. Of course that's because Memcached pre-allocates all memory in advance, and assumes that all memory will be wanted for the duration of of the process. On the other hand, jemalloc was designed as a generic memory allocator that happens to be well-behaved when it comes to dealing with memory fragmentation over time, assuming reasonable allocation behavior.

Knowing what I do about jemalloc and Memcached's memory allocator, I don't believe that there is any advantage to using any part of the Memcached slab allocator in Redis at all, and I know there are several drawbacks. The primary drawback being that most Redis users wan't Redis' memory usage to be a function of actually used memory, but the Memcached slab allocator doesn't actually return memory to the OS, so prevents this kind of (desired) behavior. This is literally a deal killer for many Redis users.

Don't get me wrong, I proposed using parts of what Memcached was using on an individual slab basis, as Python uses a similar method of pre-allocated arenas for some of its smaller objects to great success (and that's where I'm coming from). But in practice, that additional overhead of design and implementation outside of the allocator, which already does 99% of the same stuff already, is quite literally a waste of time. The jemalloc allocator is pretty awesome, and you don't need to trust me: read about it. And if after reading you think that any part of the Memcached system offers any advantages to Redis, state your reasons with references to externally verifiable information. Until then, please read up on jemalloc :)

 - Josiah

[slade]

OK, I'll read sth about the awesome jemalloc :)

Thank you so much.

在 2014年8月27日星期三UTC+8上午2时18分04秒，Josiah Carlson写道：