Potential performance issue with libpmemobj

[Amir Saman Memaripour]

Hi,

I have implemented a persistent B-Tree on top of the libpmemobj library and have been trying to measure the performance of the implementation by inserting 200K objects of size 1024 bytes. Although I am using DRAM to store the persistent pool, I noticed that the B-Tree can only serve 3K ops/sec. I did some profiling and found out that the program is spending 99% of its time in ctree_remove() function in src/libpmemobj/ctree.c. Here is the first few lines of the gprof output for the program:

  %   cumulative   self              self     total

 time   seconds   seconds    calls  us/call  us/call  name

 98.22    107.26   107.26                             ctree_remove

  0.32    107.61     0.35    99999     3.45     3.59  btree_ReadNode

  0.16    107.78     0.17                             flush_clflush

  0.11    107.90     0.13                             out_log

  0.08    107.99     0.09                             ctree_insert

I am not quite sure, but I think this function is called as part of the allocation process. Considering that the program does not issue more than three allocations per insert, it is not clear that why the program should spend 99% of its time executing this function? Is there anything that I can do (either with my code or the library) to resolve the issue?

Thanks

[Piotr Balcer]

Hi,
We are currently at a stage in development in which we focus more on making things run correctly rather than fast. Not to say we don't care about performance ;) But we really didn't fine-tune everything yet. The ctree_remove function is going to take the majority of the allocation time no matter what, but the numbers you are getting are probably a result of a bug. Can you please try your code with the library from this branch (no API changes, just internals):
https://github.com/pbalcer/nvml/tree/obj_pmalloc_grow
You should see a noticable improvement, here are the gprof results of a pmalloc(4k) in a loop(320k) benchmark:

Each sample counts as 0.01 seconds.

  %   cumulative   self              self     total          
 time   seconds   seconds    calls  us/call  us/call  name   

 57.15      0.04     0.04                             pmalloc
 28.57      0.06     0.02   320001     0.06     0.06  heap_get_block_data
 14.29      0.07     0.01   320729     0.03     0.03  ctree_insert
  0.00      0.07     0.00   320729     0.00     0.00  bucket_insert_block
  0.00      0.07     0.00   320628     0.00     0.00  bucket_get_rm_block_bestfit
  0.00      0.07     0.00   320628     0.00     0.00  bucket_is_empty
  0.00      0.07     0.00   320628     0.00     0.00  bucket_lock

Each sample counts as 0.01 seconds.

  %   cumulative   self              self     total          

 time   seconds   seconds    calls  ns/call  ns/call  name   
 33.34      0.03     0.03                             pmalloc
 22.22      0.05     0.02   320001    62.50    62.50  heap_get_block_data
 11.11      0.06     0.01   320628    31.19    31.19  ctree_remove
 11.11      0.07     0.01   320002    31.25    31.25  lane_hold
 11.11      0.08     0.01   320001    31.25    31.25  heap_unlock_if_run
 11.11      0.09     0.01                             pmem_flush

I ran it several times, getting different results, but looks better than what I'm getting on current master.
If you need more help, write me an email (piotr.balcer at intel.com), I'm also available on IRC. I'll be happy to inspect your code for any API misuses.

Piotrek

[Amir Saman Memaripour]

Hello Piotrek,

Thanks for your email. I replaced the version that I was using (v0.2) with your branch, but didn't see any improvement. I am using libpmemobj and the transnational interface, so each allocation triggers a set of list copy operations which I think are causing the extra long delay. What I am trying to understand is the way allocation is done and the reason we need to have separate lists for each object type. Is there any documentation that I can use to understand what's happening under the hood?

Thanks

[Piotr Balcer]

I cannot reproduce performance issues you are seeing. I wrote a simple benchmark (https://gist.github.com/pbalcer/506677b5d67b88311fd2) that uses transactions to create a singly-linked list of objects. Inserting 200k objects with size 1024 bytes took 2 seconds on a non-optimized debug build. I've attached profiling results and it seems that ctree_remove takes only about 4% of the application time. This was done on a ramdisk with tmpfs (to minimize the time spent on pagefaulting/flushing).
Since my last email I've made some further optimizations, could you please pull the branch again and try again? If this doesn't help there's not much else I can do without looking at your application.

All of the internal lists operations take a tiny amount of time in comparison to the allocation process so I doubt the problem lays there. They are used as a transaction redo log - when you allocate inside a transaction that new object is first linked with the internal transaction list - depending on whether the transaction commits or aborts, the object is then either moved to the destination list (all objects are always stored on a list - that's a design choice we made) or removed and freed. I'm not sure if I can share the design document we have (it still bears the "confidential" thingy on every page, don't know if there's a good reason anymore), I'll talk with the team about it when I have the chance.

Piotrek

[Amir Saman Memaripour]

All the performance issues seems to be resolved with the recent changes you pushed to the branch. Now I am getting reasonable results from my test applications and they can achieve 160 ops/sec with a single thread. Thanks for sending the pointer to your development branch, it was really helpful. Do you have any estimation on when this branch is going to be merged with the master? Are these optimizations going to be available with the next release?

Also, it would be great if you can make the documentation publicly available anytime soon.

Thanks

[Piotr Balcer]

Yey! :)
Today, I hope. And yes, they will be available with the next release. Glad I was able to help.
I'll try to get the docs public.

Piotrek