Libpmemobj Allocation Performance Issue

[Karthik Velayutham]

Hi all,

I'm working on a project that involves the use of a hashtable that is allocated in persistent memory. During the load phase of inserting thousands of keys, the hashtable has to be resized and buckets have to be allocated as well. I noticed pmemobj_alloc() is surprisingly slow at the rate of just 3103 operations/second, which was surprising to me. I suspect that my usage of libpmemobj may be incorrect, and I was wondering if anyone had any insights to what could be causing the slow performance. 

For some additional details, I have followed the proper initialization semantics as much as I can. I have laid out the memory pool properly, initialized the root, etc. I am emulating persistent memory by using a ext4-DAX mount, so there is no actual persistent memory being used as of now. I'm just unsure if PMDK is perhaps not scaling with larger allocations or my use case of the function is simply incorrect e.g.:

if (pmemobj_alloc(pop, &table_oid, num_buckets * sizeof(bucket_t), TOID_TYPE_NUM(bucket_t), 0, 0)) {

fprintf(stderr, "pmemobj_alloc failed for table_oid in clht_hashtable_create\n");

assert(0);

}

Is it possible I should use another PMDK allocator? Any insights would be much appreciated. I can provide further details if needed.

Thanks!

[Piotr Balcer]

Hi,

There are some scenarios in which the memory allocator will show increased average latency for operations. The one easiest to encounter is near OOM operation, where the allocator has to scour the entire heap looking for free memory that might simply not be available. Since you mention large allocations, the other possibility is high thread contention on the global heap - the larger the object, the more frequently the global heap lock has to be acquired, and after a threshold (~2 megabytes), all allocations will be performed under a lock.

I recommend running your workload under perf and sharing your findings with us - you could also just send us a flamegraph (https://github.com/brendangregg/FlameGraph), which would make it easy to pinpoint the problem.

Piotr

[Karthik Velayutham]

Thanks for getting back to me, Piotr!

I ran the workload under perf and this is the results I am getting, which surprised me:

As you can see, the majority of the time is spent on the method clht_put, which just acquires a lock to the bucket being modified and then proceeds to insert values . In fact, the performance doesn't seem to be bottlenecked by the resizing allocations at all, at least according to this profiler. Based on the information you have given, I now suspect that the issue comes up when each thread attempts to allocate a new object in persistent memory (which would probably be serialized.) Do you know how I could use "perf" to profile a specific method that I call in clht_put? It doesn't seem to be showing up in the results and I suspect that either the overhead was insignificant or it just was missed by "perf" possibly.

Thanks!

[Marcin Ślusarz]

Compile your example with -fno-omit-frame-pointer and -g and if possible disable optimizations (-O0).

I really recommend using FlameGraph [*] for visualization of perf traces.

You may need to use the debug version of libpmemobj (the one in pmdk_debug dir) to see reliable stack traces.

[*] https://github.com/brendangregg/FlameGraph 

Marcin

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/6a4f8797-b56a-42be-81a7-81b659f68257%40googlegroups.com.

[Abdullah Al Raqibul Islam]

Hi Piotr,

Is there any publicly available benchmark suite to check the performance of PMEM memory allocation?

You mentioned “the larger the object, the more frequently the global heap lock has to be acquired”, I didn’t get that. Can you please explain why it will occur? My understanding is, memory allocation method is lock protected and the lock is only taken while allocating the memory. So if we call the memory allocation method (for objects with size greater than 2 MB) from a single thread, it will perform better than if we call it from several threads concurrently. Is this what you tried to say?

Thank you

-- 
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/22bcdd3d-65c8-4da7-833d-ec6587d56f99%40googlegroups.com.

[Piotr Balcer]

Hi,

We have benchmarks directory on our repo, it has several different types of benchmarks, including some focused on memory allocation performance.

Each heap has only one avl tree (which is protected under a lock) that contains free memory 'chunks'. Each time a thread wants to allocate an object, it first needs to acquire a 'chunk', possibly from the avl tree, subdivide it into smaller 'units', and then return one of more 'unit' to the caller - the rest of the units are placed in the thread's arena for later reuse.

This scales well for small sizes, because each time the avl tree is accessed the threads acquires many allocatable units. But for larger allocations the number of those units is smaller, and huge allocations are handled directly from the avl tree.

This is done to maximize the amount of memory available in the global heap so that it's accessible for all threads. It's a trade-off between scalability and fragmentation/usability.

You can find more information about libpmemobj's allocator in our book (it's free).

Piotr

[Abdullah Al Raqibul Islam]

Thanks Piotr. I will check the book for details. Thanks for your brief but clear explanation.

On Apr 10, 2020, at 3:42 PM, Piotr Balcer <pi...@balcer.eu> wrote:

Hi,

We have benchmarks directory on our repo, it has several different types of benchmarks, including some focused on memory allocation performance.

Each heap has only one avl tree (which is protected under a lock) that contains free memory 'chunks'. Each time a thread wants to allocate an object, it first needs to acquire a 'chunk', possibly from the avl tree, subdivide it into smaller 'units', and then return one of more 'unit' to the caller - the rest of the units are placed in the thread's arena for later reuse.

This scales well for small sizes, because each time the avl tree is accessed the threads acquires many allocatable units. But for larger allocations the number of those units is smaller, and huge allocations are handled directly from the avl tree.

This is done to maximize the amount of memory available in the global heap so that it's accessible for all threads. It's a trade-off between scalability and fragmentation/usability.

You can find more information about libpmemobj's allocator in our book (it's free).

Piotr

pt., 10 kwi 2020, 21:26 użytkownik Abdullah Al Raqibul Islam <ais...@uncc.edu> napisał:

[Karthik Velayutham]

Thanks for the suggestions Marcin. I'll give it a shot.

On Friday, 10 April 2020 14:24:51 UTC-5, Marcin Ślusarz wrote:

Compile your example with -fno-omit-frame-pointer and -g and if possible disable optimizations (-O0).

I really recommend using FlameGraph [*] for visualization of perf traces.

You may need to use the debug version of libpmemobj (the one in pmdk_debug dir) to see reliable stack traces.

[*] https://github.com/brendangregg/FlameGraph 

Marcin

pt., 10 kwi 2020 o 21:09 Karthik Velayutham <karthik....@gmail.com> napisał(a):

Thanks for getting back to me, Piotr!

I ran the workload under perf and this is the results I am getting, which surprised me:

As you can see, the majority of the time is spent on the method clht_put, which just acquires a lock to the bucket being modified and then proceeds to insert values . In fact, the performance doesn't seem to be bottlenecked by the resizing allocations at all, at least according to this profiler. Based on the information you have given, I now suspect that the issue comes up when each thread attempts to allocate a new object in persistent memory (which would probably be serialized.) Do you know how I could use "perf" to profile a specific method that I call in clht_put? It doesn't seem to be showing up in the results and I suspect that either the overhead was insignificant or it just was missed by "perf" possibly.

Thanks!

On Friday, 10 April 2020 01:17:56 UTC-5, Piotr Balcer wrote:

Hi,

There are some scenarios in which the memory allocator will show increased average latency for operations. The one easiest to encounter is near OOM operation, where the allocator has to scour the entire heap looking for free memory that might simply not be available. Since you mention large allocations, the other possibility is high thread contention on the global heap - the larger the object, the more frequently the global heap lock has to be acquired, and after a threshold (~2 megabytes), all allocations will be performed under a lock.

I recommend running your workload under perf and sharing your findings with us - you could also just send us a flamegraph (https://github.com/brendangregg/FlameGraph), which would make it easy to pinpoint the problem.

Piotr

W dniu piątek, 10 kwietnia 2020 04:20:02 UTC+2 użytkownik Karthik Velayutham napisał:

Hi all,

I'm working on a project that involves the use of a hashtable that is allocated in persistent memory. During the load phase of inserting thousands of keys, the hashtable has to be resized and buckets have to be allocated as well. I noticed pmemobj_alloc() is surprisingly slow at the rate of just 3103 operations/second, which was surprising to me. I suspect that my usage of libpmemobj may be incorrect, and I was wondering if anyone had any insights to what could be causing the slow performance. 

For some additional details, I have followed the proper initialization semantics as much as I can. I have laid out the memory pool properly, initialized the root, etc. I am emulating persistent memory by using a ext4-DAX mount, so there is no actual persistent memory being used as of now. I'm just unsure if PMDK is perhaps not scaling with larger allocations or my use case of the function is simply incorrect e.g.:

if (pmemobj_alloc(pop, &table_oid, num_buckets * sizeof(bucket_t), TOID_TYPE_NUM(bucket_t), 0, 0)) {

fprintf(stderr, "pmemobj_alloc failed for table_oid in clht_hashtable_create\n");

assert(0);

}

Is it possible I should use another PMDK allocator? Any insights would be much appreciated. I can provide further details if needed.

Thanks!

--
You received this message because you are subscribed to the Google Groups "pmem" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pm...@googlegroups.com.

[Karthik Velayutham]

So I tried just running the code with some of the compiler flags that you had mentioned (I will try using FlameGraph soon for a better visualization), but I noticed these results:

Is it normal to have such a high writeback overhead? My implementation does create a bunch of buckets for the key-value pairs, but I was surprised to see this much overhead. Thanks!

[Piotr Balcer]

No, not if you are using DAX, so make sure you are mounting with -odax.
Run your code with PMEMOBJ_CONF="prefault.at_open=1;prefault.at_create=1" env variable to avoid page faults during your benchmark run.

Piotr

[Karthik Velayutham]

Hey Piotr,

I am running with the "-o dax" mount, specifically this is what I used: sudo mount -o dax /dev/pmem0 /mnt/pmem

I tried exporting the configurations that you mentioned, but I am still getting 72% overhead from cache flushes. My code actually does call cache flushes independent of the PMDK libraries, so I'm not sure if this is what is actually contributing to the overhead.

Thanks,

Karthik

[Karthik Velayutham]

To provide some more context, we are manually making some cache line flushes with memory fences to ensure some the ordering of instructions and persistence. I'm not sure if pmemobj_persist provides better overhead in this respect. 

[Piotr Balcer]

With that variable, you should not be seeing a significant kernel involvement after initial page faults. What's your kernel version?

Try also with PMEM_IS_PMEM_FORCE=1.

If that doesn't help, I'll need flamegraphs to know more.

Piotr

[Karthik Velayutham]

My machine's kernel version is 4.13.1. I'll try with that as well. I will send some flamegraphs later today. Thanks so much.

[Marcin Ślusarz]

Kernels before 4.15 do not support MAP_SYNC flag, which pmdk uses to detect whether it's safe to flush from userspace.

If it's just for testing purposes, you can use PMEM_IS_PMEM_FORCE=1 environment variable to override this detection,

but if you want your data to be safe you need a newer kernel.

Marcin

To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/5b75b25f-394b-4106-a165-6245a4528719%40googlegroups.com.

[Karthik Velayutham]

Hi all,

So it turns out the Linux kernel version was the problem. We upgraded the kernel version and now the performance is much, much more reasonable. However, the one thing I don't understand is what caused the significant discrepancy in terms of performance between the two kernel versions. Was there a lack of kernel support for NVDIMMs? If so, what was it specifically that caused the issues that I faced?

Thanks!

On Saturday, 11 April 2020 13:55:48 UTC-5, Marcin Ślusarz wrote:

Kernels before 4.15 do not support MAP_SYNC flag, which pmdk uses to detect whether it's safe to flush from userspace.

If it's just for testing purposes, you can use PMEM_IS_PMEM_FORCE=1 environment variable to override this detection,

but if you want your data to be safe you need a newer kernel.

Marcin

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/5b75b25f-394b-4106-a165-6245a4528719%40googlegroups.com.

[Piotr Balcer]

Like Marcin said, kernels prior to 4.15 did not support MAP_SYNC, which is necessary for safe user space flushing. Without this feature, libpmemobj falls back to calling msync on every flush - which is why you saw degraded performance.

To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/a79e22ff-40eb-4d38-8cb3-b12e0a13e8d8%40googlegroups.com.

[Karthik Velayutham]

Got it. Thank you so much for the help!

On Sunday, 12 April 2020 14:33:28 UTC-5, Piotr Balcer wrote:

Like Marcin said, kernels prior to 4.15 did not support MAP_SYNC, which is necessary for safe user space flushing. Without this feature, libpmemobj falls back to calling msync on every flush - which is why you saw degraded performance.

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/a79e22ff-40eb-4d38-8cb3-b12e0a13e8d8%40googlegroups.com.