Performance regression from 1.2 to 1.3.1 and 1.4.1

[Krishna Harathi]

We measured a significant ~34X performance regression (POBJ_LIST_REMOVE_FREE() in particular)

in pmemobj pmdk 1.3.1 and 1.4.1 compared to 1.2.

For instance, 180K POBJ_LIST_REMOVE_FREE() calls took 500ms in 1.2 vs 17000ms in 1.3.1 and 1.4.1.

We are currently using pmemobj pmdk 1.3.1 but considering moving back to 1.2.0 for this performance issue.

1. Is there a known reason or issue for this performance regression?

2. Is there a compelling reason for us not to revert to 1.2.0?

Any help or comment in this regard is appreciated.

Thanks.

Krishna Harathi

[Piotr Balcer]

Hi,

On which operating system (and which exact kernel version) are you running your measurements? Also, which 1.2 release of the libpmemobj are you using? 1.2.0?

Looking at our own benchmarking data, the performance of the library improved since 1.2.0. What I suspect is happening is that you are running on

an older kernel (<4.15) version without support for MAP_SYNC, and libpmem resorts to using msync() for data flushing. For a quick test, rerun your code with

PMEM_IS_PMEM_FORCE=1 environment variable.

In the 1.2.0 release of the library assumed that any dax-enabled file system allowed for user space flushing. But that behavior has changed starting with 1.2.1 patch

release where only device-dax namespaces were allowed to use optimized flushing:

https://github.com/pmem/pmdk/releases/tag/1.2.1

Since then, the linux kernel DAX/NVDIMM support has matured significantly, and it's now possible to use a file system /w a MAP_SYNC which makes user space

flushing possible again - this is automatically detected by libpmem and it will do the right thing to ensure data consistency.

Hope this answers your questions. If the results are still this much worse after kernel upgrade or forcing pmem, please let us know and we will try to reproduce the

regression.

Piotr

[Krishna Harathi]

Hi Piotr,

We are using Linux Kernel 4.9.30 in one of our releases, and you are spot on - I retested with PMEM_IS_PMEM_FORCE=1 environment variable, and the performance of pmdk 1.3.1 is now on-par with 1.2.0

Is setting the environment variable PMEM_IS_PMEM_FORCE=1 for pmdk 1.3.1 a viable solution as a fix (instead of upgrading the kernel)? What is the downside to it?

We are also using Linux kernel 4.16.17 in our current release, I will be testing with this kernel and pmdk 1.3.1 and will report the results asap.

Thanks, your help is very much appreciated!

Krishna Harathi

[Piotr Balcer]

Running /w PMEM_IS_PMEM_FORCE=1, despite kernel indicating otherwise, might lead to data consistency issues.

The recommended solution is to upgrade your kernel to at least 4.15. On older kernels, the only other option is to

use device dax - you can find more information about it in ndctl's man pages (http://pmem.io/ndctl/ndctl-create-namespace.html).

Piotr