pmem use cases and challenges

[Vijay Chidambaram]

Hi everyone,

I'm Vijay Chidambaram, a professor at the University of Texas at Austin. My group does research on storage systems and distributed systems. Some of our recent work include the PebblesDB key-value store and the CrashMonkey tool for finding bugs in file systems. We have some research ongoing about finding crash-consistency bugs in persistent-memory storage systems. 

Normally, our research is guided by the problems developers and users face in the field, but we are having trouble gauging that for persistent memory. 

So I wanted to ask: what challenges are folks facing when using persistent memory? I see that persistent memory is being used (or planned to be used) in key-value stores and databases. Are there other interesting use cases?

Thanks,

Vijay Chidambaram

http://www.cs.utexas.edu/~vijay/

[Steve]

The biggest challenge so far is to get access to some actual pm devices!

Steve Heller

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To post to this group, send email to pm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/cd782f9a-2a9e-4e9a-a113-b70e2adb894a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Adrian Jackson]

Hi,

In the NEXTGenIO project (http://www.nextgenio.eu) we are looking at using devdax style access directly in applications, as a checkpointing target, or as an endpoint for distributed parallel filesystems.

cheers

adrianj

[Steve Scargall]

Hi Vijay

On Monday, 7 January 2019 10:08:54 UTC-7, Vijay Chidambaram wrote:

what challenges are folks facing when using persistent memory?

The Persistent Memory Developer Kit (PMDK) available from https://pmem.io/pmdk solves a lot of the [hard] challenges that developers face, such as:

• Cross CPU/Platform support
• If a developer writes an app using specific CPU instructions, that code may not be transportable to another CPU generation.  Successive CPU generations may introduce new instructions that optimize certain operations.  These are obviously not backward compatible.  PDMK detects CPU and device features and does the most optimal code path.
• Data Persistence
• Stores are not guaranteed to be persistent until flushed. More optimal flushing where the application flushes stores from the CPU caches directly instead of calling into the kernel is also possible.  When and how to flush are very important.  Flush too frequently and you lose out on performance.  Flush too infrequently and you introduce data consistency issues.  Using the optimal flush instruction/method is also very important (see 1st point).
  
• CPUs have out-of-order CPU execution and cache access/flushing.
  
• Data Consistency
• Anything larger than 8-bytes on x86 is not powerfail atomic, so it is up to software to implement whatever transactions, logging, or recovery is required for consistency.  Several PMDK libraries have transactions to resolve this.
  
• Memory Leaks
• Memory Leaks on NVDIMMs are persistent.  Rebooting the host won't help.
• Error Handling
• Applications may need to detect and handle hardware errors directly. Since applications have direct access to the persistent memory media, any errors will be returned back to the application as memory errors.  This will result in BUS errors that applications have to handle or dump a core as they traditionally do.

I see that persistent memory is being used (or planned to be used) in key-value stores and databases. Are there other interesting use cases?

This discussion has documents links to current open-source code using PMDK which you can use for use cases.

In general, we're only scratching the surface of what is going to be possible.  These are just a few:

• In-Memory, Relational, and Key-Value Databases are a great starting place as they typically have pluggable storage engines which can be easily modified to use PMDK and Persistent Memory.  
  
• IOT - With more memory available, we can ingest more data at a higher rate since we not waiting on slower non-volatile storage.
• 5G/Comms - 5G data limits are huge compared with <=4G.  
  
• Faster application restarts - Since data can now sit close to the CPU, applications don't need to page in vast amounts of data to start up.  Restart times decrease from minutes to seconds.
• Filesystems - NOVA is written from the ground up using PMDK.  ZFS could benefit from NVM for it's ARC.
• Storage Appliances could use NVM for faster client IO, caching, etc.
• Cloud/Virtual Machines can achieve higher density per physical server given much higher amounts of available memory
• etc, etc, etc

Regards,

    Steve

[Vijay Chidambaram]

Thanks Steve and Adrian! Your response was really useful.

PMDK is awesome, and solves many important problems. But I would like
to understand better what the users of PMDK feel is missing today. Are
there abstractions that users want that PMDK does not provide (or
would need to be radically redesigned to provide)?

Are folks running into problems designing applications with PMDK? For
example, is debugging an issue? Are there tools that users (or perhaps
PMDK developers themselves) feel are missing?

Handling memory leaks and hardware errors also seems like a promising
area to explore.

I am grateful that this google group is so responsive and allows me to
interact with PMDK developers and users like this :)

Thanks,
Vijay

> --
> You received this message because you are subscribed to the Google Groups "pmem" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
> To post to this group, send email to pm...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/a5d98abd-6730-4120-841f-e6d9c95a56c3%40googlegroups.com.

[Steve Scargall]

On Tuesday, 8 January 2019 17:45:13 UTC-7, Vijay Chidambaram wrote:

PMDK is awesome, and solves many important problems. But I would like
to understand better what the users of PMDK feel is missing today. Are
there abstractions that users want that PMDK does not provide (or
would need to be radically redesigned to provide)?  

Are folks running into problems designing applications with PMDK? For
example, is debugging an issue? Are there tools that users (or perhaps
PMDK developers themselves) feel are missing?

If you are able to attend the Persistent Memory Hackathon being held on Jan 23rd, this would be a great event to get answer those questions by interacting with developers and ISV's that have or are going through that process.  See https://www.snia.org/pm-summit/hackathon.  It precedes this year's SNIA Persistent Memory Summit (https://www.snia.org/pm-summit).  The summit lectures should be recorded for replay.  The hackathon will not be recorded (that I'm aware of).

- Steve

[Vijay Chidambaram]

Thanks for the pointer Steve! I will see if my students or I could attend.

Thanks,
Vijay

> --
> You received this message because you are subscribed to the Google Groups "pmem" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
> To post to this group, send email to pm...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/cebefba5-71bf-440d-b36d-2b06bbd795cb%40googlegroups.com.