Crash consistency testing for PMDK

[Jayashree Mohan]

Hi,

We are researchers at the Systems and Storage Lab at the University of Texas at Austin. We recently built a tool called CrashMonkey that tests whether POSIX file systems recover correctly from a crash (after performing a sequence of file-system operations). CrashMonkey found bugs in btrfs and f2fs, and even in a verified file system from MIT. The tool is open-source (https://github.com/utsaslab/crashmonkey). More details about CrashMonkey can be found here ( http://www.cs.utexas.edu/~vijay/papers/osdi18-crashmonkey.pdf)

We are thinking of extending CrashMonkey to work on persistent memory file systems and lib-pmem. We plan to record all cachelines written to pmem along with flushes and fences, and then reason about what could go wrong if we crash.

We had two questions:

1) Would users be interested in a crash-consistency testing framework for lib-pmem? Is someone already working on this?

2) What would be the best way to go about testing whether an application using lib-pmem recovers correctly from a crash? The guarantees are clear for POSIX file systems, but not so much for lib-pmem. Our current idea is to provide hooks in the transactional interface that PMDK offers for dynamic object allocation on NVM (using TX_BEGIN, TX_COMMIT, etc). We think this interface is a natural fit for crash-consistency testing as we can check for violations of ACID properties if a crash occurs anywhere during/after the transaction. We are open to feedback and suggestions on how to go about doing this for lib-pmem, and any specific guarantees that needs to be tested using a framework like CrashMonkey.

Thanks,
Jayashree Mohan

[Andy Rudoff]

Hi Jayashree,

CrashMonkey sounds great.  For PMDK, there are three libraries that maintain a consistent on-media layout: libpmemblk, libpmemlog, and libpmemobj.  The last one, libpmemobj, is the general-purpose library, supporting allocations, transactions, and a huge API for persistent memory.  We have some rudimentary consistency checking for pmemblk and pmemlog (via the "pmempool check" command) but I think full consistency checking for pmemobj would definitely be interesting.

To answer your second question, I think looking for missing flushes or stores is interesting, but something that we have fair coverage on using tools like our valgrind plugin, the new "persistence inspector" tool done at Intel, and some other PMDK tools we have.  I personally would be more interested in a full fsck-like examination of the libpmemobj layout, looking for inconsistencies not just due to library bugs, but due to application bugs scribbling where they shouldn't.

And checking for libpmemobj consistency is great, but of course it only checks the internal data structures of libpmemobj.  In the general case, the point of the persistent memory programming model is to give applications direct access.  So it would be interesting if there were a way for an application writer to describe their own data structures to your program, and have that program use the description to perform consistency checks.  Sort of a data-driven fsck tool...  Just an idea.

-andy

[Ma, Zhiqiang]

Hi Jayashree,

 

The Persistence Inspector Andy mentioned is publically available from the bottom of  https://software.intel.com/en-us/persistent-memory?elqTrackId=66bf31a82cb84422988ad2cde14bb118&elqaid=18599&elqat=2

 

Zhiqiang

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To post to this group, send email to pm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/f87e9ce8-d281-450f-a556-6df39e4988cc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jayashree Mohan]

Hi Andy,

Thanks for the quick response and your thoughts on testing PMDK.

CrashMonkey can be extended to provide application specific checks. One way of doing this would be to let each libpmemobj expose a method that would describe the consistency checks to be performed after recovering from a crash. This allows an application developer to specify the crash-consistency semantics of the new data structures in this method, which CrashMonkey would use to check the consistency of on-media layout.

With CrashMonkey, the application developer need not worry about generating workloads and writing the checker to represent the expected on-media state. Given a new data structure, the list of api for this data structure and the expected guarantees, CrashMonkey can be extended to automatically generate workloads (sequences involving the api) and simulate crashes during the execution of this workload. Additionally, it enables automatic testing by generating oracles representing the true state of the system after correctly recovering from the crash. CrashMonkey can thus help build crash-consistent pmem data structures.

Thanks,
Jayashree Mohan