Persistent memory issue on Windows

[steve]

Hi all,

I've run into what looks like a Windows bug relating to power failure scenarios. I have attached source code for a minimal repro that works as
expected on Linux but not on Windows. You can do whatever you want with the code, as it has no trade secrets of mine.

I. Here's the test scenario on Linux, assuming that /mnt/PM1 is a DAX pmem volume.

A. Run the program PowerFailRepro as follows to initialize the pmem file:
PowerFailRepro initialize /mnt/PM1 test.pfr 1000000

B. Run the program again with the following arguments to start processing 1 million records:
PowerFailRepro process /mnt/PM1 test.pfr 1000000

C. While the program is outputting its "..." progress indicator, turn off power.

D. Restart the machine.

E. Run the program again with the following arguments:
PowerFailRepro readonlycheck /mnt/PM1 test.pfr 1000000

The desired result is that it tells you the next record number it is going to process, without changing the file.
This works as expected on Linux.

F. Run the program again with the following arguments:
PowerFailRepro process /mnt/PM1 test.pfr 1000000

This should pick up where the previous run left off, and at the end announce that the test has passed.

This works as expected on Linux.

-----------------------------------------
II. The test scenario on Windows is this, assuming that you have a DAX volume on your n: drive.

A. Run the program PowerFailRepro as follows to initialize the pmem file:

PowerFailRepro.exe initialize n:\test.pfr 1000000

B. Run the program again with the following arguments to start processing 1 million records:
PowerFailRepro.exe process n:\test.pfr 1000000 //starts processing 1 million records

C. While the program is outputting its "..." progress indicator, turn off power.

D. Restart the machine.

E. Run the program again with the following arguments:
PowerFailRepro.exe readonlycheck n:\test.pfr 1000000

The desired result is that it tells you the next record number it is going to process, without changing the file.

However, what happens in my testing is that it says:

"No transactions completed or canceled!"

which means that all the transactions were lost.

--------------------

Both the successes on Linux and the failures on Windows have been consistent in my dozen or so tests, which should help in diagnosing the issue.

Would anyone here like to take a look at this? Of course if any further details are needed, I'll be happy to help.
------------
Steve Heller

[Igor]

Hi,

First, I would suggest running your code under valgrind pmemcheck (https://pmem.io/2015/07/17/pmemcheck-basic.html) and pmreorder: (https://pmem.io/2019/02/04/pmreorder-basics.html).
Those tools work under linux only but you can use them to verify if your program is indeed correct.

If you'll have any problems with running pmreorder (it requires several steps) you can take a look how we use it in libpmemobj-cpp or pmdk tests.
For example here: https://github.com/pmem/libpmemobj-cpp/tree/master/tests/array_slice_pmreorder

Igor

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/pus7lf9ust2hm0a5kr6evtk60j567usjie%404ax.com.

[steve]

Hi Igor,

Are there any detailed instructions on how to build valgrind pmemcheck?

I did this:
-------------
git clone https://github.com/pmem/valgrind.git
cd valgrind
./autogen.sh
-------------
with the result:
-------------
running: aclocal
./autogen.sh: 1: eval: aclocal: not found
error: while running 'aclocal'
-------------
I tried this:
-------------
apt-get install aclocal
-------------
but apt-get couldn't locate that package.

Any assistance would be appreciated.

------------
Steve Heller

[Igor]

I'm not sure if there is a list of required dependencies, but you need at least: automake, autoconf and build-essential (at least on ubuntu).
Also, on ubuntu you can just use `dpkg -S "command name"` to find out what package is required.

Igor

[steve]

Hi Igor,

Looks like it was automake that was missing. It's building now.
Thanks!

------------
Steve Heller

[steve]

On Mon, 7 Sep 2020 13:12:48 +0200, Igor <chori...@gmail.com> wrote:

>Hi,
>

>First, I would suggest running your code under valgrind pmemcheck (
>https://pmem.io/2015/07/17/pmemcheck-basic.html) and pmreorder: (

>https://pmem.io/2019/02/04/pmreorder-basics.html).
>Those tools work under linux only but you can use them to verify if your
>program is indeed correct.

When I run my program under valgrind pmemcheck with no options, I get:
Number of stores not made persistent: 0.
ERROR SUMMARY: 0 errors

Adding options doesn't make any difference:
--flushcheck=yes shows no errors
--mult-stores=yes shows no errors

Isn't that enough to indicate that losing ALL my transactions under Windows is probably not a bug in my code?
If not, I'll continue with pmreorder.

[ppbb...@gmail.com]

Hi Steve,

Can you tell us the exact OS type (client/server/SAC/LTSC) and version on which you are encountering this problem?

Also, can you check if a program that simply increments a counter in a tight loop, just like you are doing it in your example, exhibits the same behavior of losing content on power failure?

Piotr

[steve]

On Tue, 15 Sep 2020 07:09:17 -0700 (PDT), "ppbb...@gmail.com" <ppbb...@gmail.com> wrote:

>Hi Steve,
>Can you tell us the exact OS type (client/server/SAC/LTSC) and version on
>which you are encountering this problem?

Windows 10 Enterprise LTSC (x64, obviously).
Version 1809, OS build 17763.1457.

>Also, can you check if a program that simply increments a counter in a
>tight loop, just like you are doing it in your example, exhibits the same
>behavior of losing content on power failure?

I'm sorry, I don't understand what you are asking me to do.
Could you explain further? I could do a Zoom call or the like if that would help.

>Piotr
>
>poniedzia?ek, 7 wrze?nia 2020 o 17:23:03 UTC+2 st...@steveheller.org
>napisa?(a):

------------
Steve Heller

[ppbb...@gmail.com]

I briefly looked at your code and concluded that the problem boils down to the `TotalTransactionProcessingCount` counter value being zeroed out after a power failure. And, if the reading of my code is correct, this counter is never decremented. This makes me think that the same problem could be reproduced by effectively boiling down this code to:

struct pmem {

  uint64_t counter;

};

struct pmem *pmem = mmap(...);

printf("%llu\n", pmem->counter);

while (true) {

      uint64_t counter = pmem->counter + 1;

      pmem_memmove_nodrain(&pmem->counter, &counter, sizeof(counter));
      pmem_drain();

}

We could then see if, after a power failure, the counter value is 0 or some other value.

[Igor]

Hi Steve,

could you also please check if the file ("n:\test.pfr" in your example) actually exists after the server reboot but before calling "PowerFailRepro.exe readonlycheck n:\test.pfr 1000000"?
I'm not sure why the file would not be there but I believe, "No transactions completed or canceled!" can be printed also in case, the file does not exist.

Igor

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/148b1e6b-d3fc-4be2-97ae-47e26d452df1n%40googlegroups.com.

[steve]

On Tue, 15 Sep 2020 17:58:00 +0200, Igor <chori...@gmail.com> wrote:

>Hi Steve,
>
>could you also please check if the file ("n:\test.pfr" in your example)
>actually exists after the server reboot but before calling
>"PowerFailRepro.exe readonlycheck n:\test.pfr 1000000"?
>I'm not sure why the file would not be there but I believe, "No
>transactions completed or canceled!" can be printed also in case, the file
>does not exist.

I've run the test under Windows.
The test file is there, and the value of the counter is *not* 0.
So the problem isn't that simple.
Which isn't surprising because if it were that simple, I would have expected it to show up in other people's results before this.
What's the next step?

>Igor
>
>On Tue, Sep 15, 2020 at 5:30 PM ppbb...@gmail.com <ppbb...@gmail.com>
>wrote:
>
>> I briefly looked at your code and concluded that the problem boils down to
>> the `TotalTransactionProcessingCount` counter value being zeroed out after
>> a power failure. And, if the reading of my code is correct, this counter is
>> never decremented. This makes me think that the same problem could be
>> reproduced by effectively boiling down this code to:
>>
>> struct pmem {
>> uint64_t counter;
>> };
>>
>> struct pmem *pmem = mmap(...);
>> printf("%llu\n", pmem->counter);
>> while (true) {
>> uint64_t counter = pmem->counter + 1;
>> pmem_memmove_nodrain(&pmem->counter, &counter, sizeof(counter));
>> pmem_drain();
>> }
>>
>> We could then see if, after a power failure, the counter value is 0 or
>> some other value.
>>

>> wtorek, 15 wrze?nia 2020 o 17:05:11 UTC+2 st...@steveheller.org
>> napisa?(a):

>> <https://groups.google.com/d/msgid/pmem/148b1e6b-d3fc-4be2-97ae-47e26d452df1n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
------------
Steve Heller

[ppbb...@gmail.com]

I just wanted to eliminate the possiblity of any obvious problems. We will try running your code internally to see if we can reproduce the problem, and if so, we will investigate further.

[Steve]

Thanks!

Steve Heller

On Sep 15, 2020, at 1:09 PM, ppbb...@gmail.com <ppbb...@gmail.com> wrote:

I just wanted to eliminate the possiblity of any obvious problems. We will try running your code internally to see if we can reproduce the problem, and if so, we will investigate further.

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/0105655b-81b9-4929-94d7-0f9f9a9d0014n%40googlegroups.com.