Do we have any materials to discuss the implementations about the pmemobj memory management?

[李鹏]

To everyone who may help,

Question 1:

I have spent several days working on the code of libpmemobj (c version), and knew lots about types and basic usages.

However, when I want to dig deeper, such as the memory managerment, I failed. It seemed too complex. So if someone can share some materials about the internal implementation details? Thanks a lot. 

I am reading the libpmemobj/heap.c, palloc.c, memblock.c. 

Question 2:

Another question is that I heard many researchers say that libpmemobj works not so efficient. We have the requirements to allocate objects on PM, deallocate some of them, read objects and traverse through all the objects. The objects should be persistent.

What can we do to achieve the best performance? The object size is generally 16B ~ 1KB. 

It acts as the persist object pool, which is responsible for the persist memory fragments (owing to deallocation of objects). And we do not want to use transaction for persistence. We want to persist the object as we want.

We do not know whether palloc works as well as jemalloc.  

Wish someone could help me.

 

[Piotr Balcer]

Hi,

I'm the primary author of libpmemobj's allocator.

Question 1:

I have spent several days working on the code of libpmemobj (c version), and knew lots about types and basic usages.

However, when I want to dig deeper, such as the memory managerment, I failed. It seemed too complex. So if someone can share some materials about the internal implementation details? Thanks a lot. 

I am reading the libpmemobj/heap.c, palloc.c, memblock.c. 

libpmemobj's design is a moving target, and the documentation that we've initially wrote for it is mostly outdated. We are currently working on a brand new design doc that we will maintain alongside the code.

I do believe however that our memory allocator source code is fairly readable and well documented on the frontend/layout side (palloc, memblock) but I'm willing to admit that the recycler module (responsible for scalability) is

difficult to understand properly. Also keep in mind that palloc, by necessity, combines the properties of memory allocators and file systems, and as a result is fairly more complex.

 

Question 2:

Another question is that I heard many researchers say that libpmemobj works not so efficient. We have the requirements to allocate objects on PM, deallocate some of them, read objects and traverse through all the objects. The objects should be persistent.

What can we do to achieve the best performance? The object size is generally 16B ~ 1KB. 

It acts as the persist object pool, which is responsible for the persist memory fragments (owing to deallocation of objects). And we do not want to use transaction for persistence. We want to persist the object as we want.

We do not know whether palloc works as well as jemalloc.  

Memory allocation is always a balance of time and space constraints. In libpmemobj, we've decided to favor space (by reducing fragmentation) instead of time (performance), because we expect it to be the bigger problem in long running applications. That doesn't mean that we don't care about performance, we very much do, but not at the cost of worse fragmentation.  And above all, we made sure that our allocator is correct and power-fail atomic, which is more difficult than it sounds.

I've yet to see an open-source implementation (not just a research paper) of a memory allocator for persistent memory that is both correct and superior to libpmemobj.

For your use case, you don't need to do anything special, just use either the atomic API or the transactional one, write and read your data, and you can traverse it using libpmemobj's object iterator (POBJ_FIRST/POBJ_NEXT). You might also want to consider using libpmemobj++ pmem::vector class if you expect to allocate a lot of objects of the same size.

To optimize space efficiency, I strongly suggest using custom allocation classes. This feature is explained in detail in the CTL namespace (config) man page (http://pmem.io/pmdk/manpages/linux/master/libpmemobj/pmemobj_ctl_get.3).

libpmemobj's persistent allocator is not going to perform as fast as any volatile one, simply because it is transactional and has to ensure failure atomicity. But, based on our own measurements on select workloads, palloc is more space efficient than jemalloc, especially in multi-thread applications - which is what we really care about.

Piotr

[Steve Scargall]

There is a recorded webinar with an introduction to the libpmemobj allocator internals here - https://software.intel.com/en-us/videos/introduction-to-persistent-memory-allocator-and-transactions.