Bugs reported by pmemcheck

[Jayashree Mohan]

Hi all,

We are researchers at the Systems and Storage Lab at UT Austin. We are working on building a crash-consistency testing framework for Persistent Memory applications, something along the lines of our prior work, CrashMonkey (https://github.com/utsaslab/crashmonkey).

As a first step to evaluating the framework, we are looking for any known/reported crash-consistency bugs in PMDK (something like this one, https://github.com/pmem/pmdk/commit/04ec84e23ed40be92bd89b9d34c39fbf28cafe0b#diff-f2692f0bb21a212d07a5d1bc2115c071). Is there a list of bugs reported by pmemcheck or the Persistence Inspector tool that we could access?

Any pointers to such bugs will be greatly appreciated!

Thanks,
Jayashree Mohan

[Piotr Balcer]

Hi,

We have had pmemcheck integrated into our CI process for years now, which means that it's very unlikely that a pmemcheck-detectable bug will slip through all the way to the master branch.

The bugs that did slip through were mostly located in our examples. Here are the ones I could find:

https://github.com/pmem/pmdk/pull/3134

https://github.com/pmem/pmdk/pull/3126

We've also built a comprehensive consistency checking tool that might be of interest to you:

http://pmem.io/pmdk/manpages/linux/v1.5/pmreorder/pmreorder.1.html

https://github.com/pmem/pmdk/blob/master/src/tools/pmreorder/pmreorder.py

Piotr

[Jayashree Mohan]

Thanks for the pointers Piotr! 

So if I understand this right, pmemcheck ensures that the internal mechanisms of undo log, recovery, etc are bug free, while the data structures built using the pmemobj library like the examples you pointed out, require additional testing. 

And, how were these bugs in btree/rteee discovered ? Was that a result of pmemcheck testing on the examples? 

Thanks,

Jayashree

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To post to this group, send email to pm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/5c5d3413-709f-4290-82dd-1d95df95757b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Jayashree Mohan

[Piotr Balcer]

The reason why the bugs in the examples I linked weren't found earlier was that we simply didn't yet have automatic pmemcheck tests for those specific examples.

When we finally added those tests, they revealed the new issues. You can see this in https://github.com/pmem/pmdk/pull/3134/files, where

@marcinslusarz wrote new tests for examples, and then he fixed the issues those tests found.

And just to be extra clear, pmemcheck has a very limited ability to find any bugs with application logic. It simply verifies if all persistent memory
modifications are followed by appropriate flushing/fencing instructions, and it also detects missing snapshots in undo log transactions. That's it.

By no means does that guarantee that pmemobj's internal mechanisms are bug free ;) That's why we've built the pmemreorder tool which I linked in
my previous email.

[Jayashree Mohan]

Hi Piotr,

The reason why the bugs in the examples I linked weren't found earlier was that we simply didn't yet have automatic pmemcheck tests for those specific examples.

When we finally added those tests, they revealed the new issues. You can see this in https://github.com/pmem/pmdk/pull/3134/files, where

@marcinslusarz wrote new tests for examples, and then he fixed the issues those tests found.

Got it. But, did you stumble upon these test cases randomly? The ability of pmemcheck to find bugs seems to rely on the test cases written. Do you employ any fuzzing techniques to generate random test cases and test them with pmemcheck?

 

And just to be extra clear, pmemcheck has a very limited ability to find any bugs with application logic. It simply verifies if all persistent memory
modifications are followed by appropriate flushing/fencing instructions, and it also detects missing snapshots in undo log transactions. That's it.

By no means does that guarantee that pmemobj's internal mechanisms are bug free ;) That's why we've built the pmemreorder tool which I linked in
my previous email.

We are interested in finding bugs that pmemcheck or pmemreorder would not find (in a reasonable amount of time). I see that the pmreorder tool is pretty new; Is it integrated into your testing, just like pmemcheck? If not, were there any interesting bugs found using pmreorder?

Thanks,

Jayashree

[Marcin Ślusarz]

śr., 23 sty 2019 o 19:47 Jayashree Mohan <jayash...@gmail.com> napisał(a):

Hi Piotr,

The reason why the bugs in the examples I linked weren't found earlier was that we simply didn't yet have automatic pmemcheck tests for those specific examples.

When we finally added those tests, they revealed the new issues. You can see this in https://github.com/pmem/pmdk/pull/3134/files, where

@marcinslusarz wrote new tests for examples, and then he fixed the issues those tests found.

Got it. But, did you stumble upon these test cases randomly? The ability of pmemcheck to find bugs seems to rely on the test cases written. Do you employ any fuzzing techniques to generate random test cases and test them with pmemcheck?

If you look closely at those fixes you will notice "Found by PMTest" in commit descriptions :). I got it from a draft of a paper about PMTest.

Btree example is quite simple, so reproducing bugs reported by this tool was a matter of writing simple test that exercises buggy code paths and running it under pmemcheck.

 

 

And just to be extra clear, pmemcheck has a very limited ability to find any bugs with application logic. It simply verifies if all persistent memory
modifications are followed by appropriate flushing/fencing instructions, and it also detects missing snapshots in undo log transactions. That's it.

By no means does that guarantee that pmemobj's internal mechanisms are bug free ;) That's why we've built the pmemreorder tool which I linked in
my previous email.

We are interested in finding bugs that pmemcheck or pmemreorder would not find (in a reasonable amount of time). I see that the pmreorder tool is pretty new; Is it integrated into your testing, just like pmemcheck? If not, were there any interesting bugs found using pmreorder?

pmreorder is not integrated yet. It depends on the consistency check phase to not modify the pool and pmemobj can't guarantee it yet (it must perform recovery process on "open"). We want to address it in pmdk 1.6. 

 

Marcin