Do I need to use the pmem library?

[Hayden Livingston]

The promise of Intel Optane and other NVDIMM was that they can be byte addressable using regular OS APIs once memory mapped.

This addition of the pmem library complicates things for my product.

Is it strictly required or can I just memory map the device and read/write to it?

[Hayden Livingston]

I'd like to clarify that my program will be the sole user of the device arbitrating access to the device via my own scheme, figuring out where the data of interest is located and how large it is so that multiple threads/programs will behave and therefore libpmem is like adding a filesystem level overhead for my use case.

[Piotr Balcer]

PMDK is a suite of libraries, ranging from libpmem which provides low-level OS abstraction on useful primitives (map, flush, optimized memcpy etc), through

things like libpmemlog which add a very simple functionality on top of libpmem, to libpmemobj, which provides memory allocation and fail-safe atomic transactions.

We hope that at least the low-level interfaces we've prepared are generally useful, and that most people would want to take advantage of them. But you most certainly

don't have to use any of them, and rely on the operating system interfaces directly.

Keep in mind that if you want direct byte addressability (without page cache), what is strictly required is DAX. For your particular use case, it seems what you want

is device DAX, see ndctl man page for more information: http://pmem.io/ndctl/ndctl-create-namespace.html

We usually recommend that people should simply use fsdax, which requires a file system on top, but you get to keep all the useful features that a file system can

take advantage of, like transparent huge pages.

Piotr

[Daniel Waddington]

My understanding is that you still get transparent huge pages using device dax and mmap MAP_SYNC providing the alignment is right. Am I wrong?

[Daniel Waddington]

I think the answer to your question is that you can use as much as pmdk as you wish, nothing is essential but you have to rewrite some of what is provided in pmdk for practical use. In my opinion, the hardest thing about PM programming is crash consistency and restart, and relocatable data structures. PMDK helps with these but you may want to roll your own parts. PMDK isn't perfect (e.g. numa support) but it helps to get going quickly. It deals with many nuances that may not be obvious as first glance.

Daniel

[Andy Rudoff]

Yes, well, kind of.  My understanding of the "transparent huge pages" feature is that it opportunistically combines pages into huge pages when possible.  With Device DAX, if things are aligned you get huge pages from the start and since there's no file system to move allocations around, they will always be huge pages.  So I guess that's a yes (and MAP_SYNC isn't necessary with Device DAX).

-andy

[Andy Rudoff]

Good summary.  PMDK is a convenience, not a requirement.  Use it, steal from it, hack it up, or ignore it.  Your choice.

But if you decide not to use PMDK, please remember to follow some of the things that libpmem does, like checking if instructions like CLWB exist before using them, and checking the ACPI property to see if CPU caches even need flushing instead of unconditionally flushing.  This will help your code work optimally in future generations.  The nice thing about libpmem is we figured out how to do all these steps on Windows and Linux and put it into a fairly small & simple library, so even if you decide you don't want transactions or allocators, the stuff in libpmem is probably worth checking out.

-andy

[Piotr Balcer]

Yea, I guess it depends on how one defines transparent huge pages. If I'm reading the docs (https://www.kernel.org/doc/Documentation/vm/transhuge.txt) correctly, it's more about "automatic promotion and demotion of page sizes" ;)

And devdax precludes that because "device-dax ... guarantees fault granularity with respect to a given page size (4K, 2M, or 1G on x86) set at configuration time" . But at that point it's semantics discussion ;)

[Hayden Livingston]

I didn't even known about CLWB so thanks for that.

Let me ask another question, what if I guarantee no power failures then does it become easier to reason about persistent memory?

Do I still need to think about CLWB, PCOMMIT, etc?

My program will be ready in 2020. I'm hoping by then we can have all such pmem devices have power backup. I still think there is the case that your programs commits to pmem some garbage due to a bug, but oh well you can't save from that without some transactional system.

[Andy Rudoff]

Great question!  Here's my take:

Historically, programmers are used to memory being volatile.  If you're building a data structure, like a hash table or a tree, you use locks to protect critical sections and make the code MT-safe, but if the power fails or the kernel crashes or the program crashes, the data structure is gone so the programmer doesn't usually worry about recovering from that.

For storage, of course, the situation is quite different.  Programmers use techniques like logging, checksums, atomic pointer updates and so on to make sure they can maintain a consistent data structure in the face of failure.  Database and file system programmers are particularly well-versed in this.

For persistent memory data structures, you have to decide the requirements of your application.  Programs and systems can crash for many reasons, not just power failure.  So how will you maintain consistency?  Will you throw away the data structure like volatile memory?  Will you have a way to detect inconsistencies?  How do you decide if a particular byte of persistent memory is allocated or free and is your heap going to be consistent in the face of failure?

The PMDK libraries offer several different ways to solve these problems, each one targeted towards various use cases.  And we're not done -- we're still improving performance, coming up with new APIs for new use cases, etc.  So like I said before, use them or don't use them, but it is really up to your application requirements to decide what crash/powerfail cases you need to handle.  Just remember that a power failure is not the only reason you might need to maintain consistency.

Hope that helps,

-andy

[Hayden Livingston]

Thank you for the insights.

I will certainly look at libpmem (I was bundling everything else in PMDK as libpmem).

I agree with you that the value proposition for persistent memory is data structures that persist after program restarts. And I will also admit that if you end up wanting resiliency you're starting to build a filesytem :)

Although I want to say thank you to the persistent memory pioneers, because making a storage layer byte addressable helps average developers who are interested in either making a custom storage solution or filesystem easy. Maybe the traditional storage layers also supported that but for some reason I had never thought of so many storage ideas prior to persistent memory. Maybe it is marketing, but I think not.

[Niall Douglas]

Given your low level needs which seem more around data persistence than persistent memory, you may wish to give the reference implementation library for the P1031 proposed standard C++ support for this sort of stuff a whirl: https://ned14.github.io/llfio/

It is aware of persistent memory, and it will take advantage of it if asked. But it is designed for good performance across all storage types, rather than excellent performance on any one storage type. If you specifically need excellent support for persistent memory and don't care about anything else, Intel's toolkit is a better choice.

Niall

[technovelist]

Oh, I don't know about that. I think your library is excellent!