Running pmemkv_bench with multiple threads

48 views
Skip to first unread message

msl

unread,
Dec 1, 2021, 11:35:37 PM12/1/21
to pmem
Hi, 

I tried following but it looks like spawning multiple threads is more of an overhead than getting better performance. Is this expected? Does this argument work? 

Single thread: 1.1 microsec/op  

./pmemkv_bench --db=/mnt/pmem/kvstore --db_size_in_gb=1 --num=100000 --reads=100000 --key_size=5 --value_size=5 --benchmarks=readseq --threads=1

Multiple threads: 1.4 microsec/op

./pmemkv_bench --db=/mnt/pmem/kvstore --db_size_in_gb=1 --num=100000 --reads=100000 --key_size=5 --value_size=5 --benchmarks=readseq --threads=5

Thanks in advance!

Igor

unread,
Dec 2, 2021, 6:24:45 AM12/2/21
to msl, pmem
Hi,

if you run pmemkv-bench with multiple threads the total number of operations is num_threads * num (each thread reads all the elements).
A higher value of microsec/op is expected in that case because scaling is not perfectly linear. You should, however, see a higher total throughput.

You might find this performance report useful: https://pmem.io/pmemkv/reports.html

Best regards,
Igor

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/974a2aaa-b3c7-4aad-b081-ba0d0ee41aafn%40googlegroups.com.

its msl

unread,
Dec 7, 2021, 8:18:56 PM12/7/21
to Igor, pmem
Hi Igor,
Thanks for your response. Yes, I did see a higher throughput. 
I also noticed that it takes a really long time (few hours for 1M entries) to populate the data in pmem kvstore before it starts running the benchmark. Is that expected?

Thanks!

Igor

unread,
Dec 9, 2021, 4:26:23 AM12/9/21
to its msl, pmem
Yes, that's correct. Also, if you just call `mount` you can see if and how /mnt/pmem is mounted currently.

Best regards,
Igor

On Wed, Dec 8, 2021 at 8:05 PM its msl <its.m...@gmail.com> wrote:
Okay, that makes sense. 
When I created the fsdax namespace (/dev/pmem0), I saw that /mnt/pmem was already mounted so I did not have to mount it explicitly. 

Is this the right way to mount it using the -o dax flag?
  mount /dev/pmem0 /mnt/pmem -o dax

Thanks!

On Wed, Dec 8, 2021 at 1:49 AM Igor <chori...@gmail.com> wrote:
Hi,

no, it should not be so slow - at least on pmem. Can you verify if /mnt/pmem is mount with -o dax flag?
If it's not, then pmemkv will use msync to ensure data consistency which is very costly.

Best regards,
Igor

its msl

unread,
Dec 12, 2021, 10:06:53 PM12/12/21
to Igor, pmem
Thanks Igor!
One last question: I have written a script in cpp which simply issues sequential reads on already created kvstore. When I did the same using pmemkv_bench, it took much longer per operation. Does pmemkv_bench verify status and data for every read? Is there any other overhead?

Script: 1.3 microsec/op

./pmemkv_basic_script --file /mnt/pmem/kvstore --threads 10

Pmemkv_bench: 2.2 microsec/op

./pmemkv_bench --db=/mnt/pmem/kvstore --db_size_in_gb=10 --num=10000000 --key_size=8 --value_size=64 --engine=cmap --threads=1 --benchmarks=readseq

Note that both are using the same underlying kvstore and issuing 10M read operations.

Thanks!

Igor

unread,
Dec 14, 2021, 8:24:13 AM12/14/21
to its msl, pmem
Pmemkv_bench does not verify the data but it keeps track of how many elements were found and also saves the current timestamp after each operation (to be able to build histogram).
For small key/values the overhead of this can be significant. If you are not doing anything with the value in you program then those times looks correct.

Don't hesitate to ask if you have more questions.

Best regards,
Igor

its msl

unread,
Dec 16, 2021, 9:39:55 PM12/16/21
to Igor, pmem
Okay, thanks for clarifying it.

I ran the following script to find out how many entries with key_size=16B and value_size=100B can be held in a kvstore of capacity 25GB. It can store only ~66M entries which is just 30% of total capacity.  So, it looks like there is almost an overhead of 60%!! Is that expected?


./pmemkv_fill_cpp /mnt/pmem/kvstore 25 cmap 16 100
Number of elements: 66956026
Closing database

(I remember reading in one of the old threads that the overhead for recent versions of PMDK is 16B by default per value entry. But the above number is still way off.)

Thanks!

Igor

unread,
Dec 17, 2021, 1:06:28 PM12/17/21
to its msl, pmem
Right, the overhead of cmap is actually quite big and the result you got looks correct. You can take a look at the actual implementation to see why: https://github.com/pmem/libpmemobj-cpp/blob/master/include/libpmemobj%2B%2B/container/concurrent_hash_map.hpp#L277 and https://github.com/pmem/pmemkv/blob/master/src/engines/cmap.h

The 16B overhead you mentioned is per each memory allocation. Internally, in pmemkv, both key and value are (usually) dynamically allocated. This means at least 32B overhead. But that's not all.
Were are wrapping both key and value in pmem::obj::string (which adds at least another 16B overhead each - for `size` and `capacity` fields). There are also per-element and per-bucket locks (in the case of pmemkv a single lock is 64B). And lastly, each node in a hashmap contains a pointer (8B) to the next node in a bucket.

Cmap is probably the least memory-efficient data structure we have right now. You might want to try radix engine which should have a smaller memory overhead: https://github.com/pmem/pmemkv/blob/master/doc/ENGINES-experimental.md#radix

Best regards,
Igor

msl

unread,
Dec 17, 2021, 1:37:55 PM12/17/21
to pmem
But radix engine does not support concurrent access, right?

Igor

unread,
Dec 17, 2021, 1:54:41 PM12/17/21
to msl, pmem
That's correct, it does not support concurrent accesses.

However, in the current implementation this true only in presence of writes (meaning you cannot perform multiple concurrent writes or concurrent write and read).
As of now, you can read radix from multiple threads (e.g. use readseq or readrandom in pmemkv-bench). However, we do not guarantee this will always be true.

Best regards,
Igor

Reply all
Reply to author
Forward
0 new messages