Interesting results from Persistence Inspector

[steve]

Hi all,

I ran my program until I was in the middle of a transaction and then exited via exit(1).
Then I reran the program, which noticed the interrupted transaction and restarted it.

Below is what I got from the inspector. Nothing about missing fences or out of order stores, just redundant memory fences.

Any ideas on how to figure out what is causing the power fail situation to be so different from interruption?

I'm sure the fact that killing the VM behaves in exactly the same way as an actual power failure is an important clue but I don't know what it means.

d:\ThreeMisses>pmeminsp report -- d:\ThreeMisses\UMVMTest\KVStoreBenchmark\x64\Release\KVStoreBenchmark.exe IFNL-n:\test.3ip OFNL-n:\test.3op
TIFNL-n:\test.3tip TOFNL-n:\test.3top RC-10M KL-8 VL-10 IPSA-3B IPSI-1B OPSA-1B OPSI-1B PER-1 OP-TP
#===============================================================================
# Diagnostic # 1: Redundant memory fence for persistence
#-------------------
Memory fence is redundant for persistence ordering but may be necessary for other purposes
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!CommitTransaction at threemisseswithtransactionsupport.cpp:278 -
0x1FD3E
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!CommitTransaction at threemisseswithtransactionsupport.cpp:278 -
0x1FD3C
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!Transactions at kvstorebenchmark.cpp:762 - 0x61B9
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!operator() at kvstorebenchmark.cpp:928 - 0x9126
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!callnewh at <unknown_file>:<unknown_line> - 0x4EB17
in c:\windows\system32\kernel32.dll!BaseThreadInitThunk at <unknown_file>:<unknown_line> - 0x1796E
in c:\windows\system32\ntdll.dll!RtlUserThreadStart at <unknown_file>:<unknown_line> - 0x6A26B


#===============================================================================
# Diagnostic # 2: Redundant memory fence for persistence
#-------------------
Memory fence is redundant for persistence ordering but may be necessary for other purposes
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!Transactions at kvstorebenchmark.cpp:851 - 0x6311
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!Transactions at kvstorebenchmark.cpp:854 - 0x632D
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!operator() at kvstorebenchmark.cpp:928 - 0x9126
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!callnewh at <unknown_file>:<unknown_line> - 0x4EB17
in c:\windows\system32\kernel32.dll!BaseThreadInitThunk at <unknown_file>:<unknown_line> - 0x1796E
in c:\windows\system32\ntdll.dll!RtlUserThreadStart at <unknown_file>:<unknown_line> - 0x6A26B


#===============================================================================
# Diagnostic # 3: Redundant memory fence for persistence
#-------------------
Memory fence is redundant for persistence ordering but may be necessary for other purposes
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!Open at umvmvector.h:184 - 0x20907
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!Open at umvmvector.h:187 - 0x20915
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!Transactions at kvstorebenchmark.cpp:865 - 0x65B0
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!operator() at kvstorebenchmark.cpp:928 - 0x9126
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!callnewh at <unknown_file>:<unknown_line> - 0x4EB17
in c:\windows\system32\kernel32.dll!BaseThreadInitThunk at <unknown_file>:<unknown_line> - 0x1796E
in c:\windows\system32\ntdll.dll!RtlUserThreadStart at <unknown_file>:<unknown_line> - 0x6A26B


#===============================================================================
# Diagnostic # 4: Redundant memory fence for persistence
#-------------------
Memory fence is redundant for persistence ordering but may be necessary for other purposes
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!Open at umvmvector.h:195 - 0x2097D
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!Open at umvmvector.h:196 - 0x20988
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!Transactions at kvstorebenchmark.cpp:865 - 0x65B0
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!operator() at kvstorebenchmark.cpp:928 - 0x9126
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!callnewh at <unknown_file>:<unknown_line> - 0x4EB17
in c:\windows\system32\kernel32.dll!BaseThreadInitThunk at <unknown_file>:<unknown_line> - 0x1796E
in c:\windows\system32\ntdll.dll!RtlUserThreadStart at <unknown_file>:<unknown_line> - 0x6A26B


#===============================================================================
# Diagnostic # 5: Redundant memory fence for persistence
#-------------------
Memory fence is redundant for persistence ordering but may be necessary for other purposes
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!~UMVMVector<struct
UserModeVirtualMemory::ConsistencyBlockPersistentStruct> at umvmvector.h:139 - 0x2047F
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!~UMVMVector<struct
UserModeVirtualMemory::ConsistencyBlockPersistentStruct> at umvmvector.h:139 - 0x2047D
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!Transactions at kvstorebenchmark.cpp:865 - 0x65B7
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!operator() at kvstorebenchmark.cpp:928 - 0x9126
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!callnewh at <unknown_file>:<unknown_line> - 0x4EB17
in c:\windows\system32\kernel32.dll!BaseThreadInitThunk at <unknown_file>:<unknown_line> - 0x1796E
in c:\windows\system32\ntdll.dll!RtlUserThreadStart at <unknown_file>:<unknown_line> - 0x6A26B


#===============================================================================
# Diagnostic # 6: Redundant memory fence for persistence
#-------------------
Memory fence is redundant for persistence ordering but may be necessary for other purposes
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!~UMVMVector<struct
UserModeVirtualMemory::ConsistencyBlockPersistentStruct> at umvmvector.h:140 - 0x204E4
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!~UMVMVector<struct
UserModeVirtualMemory::ConsistencyBlockPersistentStruct> at umvmvector.h:140 - 0x204EA
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!Transactions at kvstorebenchmark.cpp:865 - 0x65B7
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!operator() at kvstorebenchmark.cpp:928 - 0x9126
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!callnewh at <unknown_file>:<unknown_line> - 0x4EB17
in c:\windows\system32\kernel32.dll!BaseThreadInitThunk at <unknown_file>:<unknown_line> - 0x1796E
in c:\windows\system32\ntdll.dll!RtlUserThreadStart at <unknown_file>:<unknown_line> - 0x6A26B


#===============================================================================
# Diagnostic # 7: Redundant memory fence for persistence
#-------------------
Memory fence is redundant for persistence ordering but may be necessary for other purposes
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!operator struct
UserModeVirtualMemory::ConsistencyBlockPersistentStruct at umvmvector.h:354 - 0x2066C
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!operator struct
UserModeVirtualMemory::ConsistencyBlockPersistentStruct at umvmvector.h:354 - 0x20670
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!Transactions at kvstorebenchmark.cpp:865 - 0x65B7
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!operator() at kvstorebenchmark.cpp:928 - 0x9126
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!callnewh at <unknown_file>:<unknown_line> - 0x4EB17
in c:\windows\system32\kernel32.dll!BaseThreadInitThunk at <unknown_file>:<unknown_line> - 0x1796E
in c:\windows\system32\ntdll.dll!RtlUserThreadStart at <unknown_file>:<unknown_line> - 0x6A26B


#===============================================================================
# Diagnostic # 8: Redundant memory fence for persistence
#-------------------
Memory fence is redundant for persistence ordering but may be necessary for other purposes
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!Open at umvmvector.h:173 - 0x2076E
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!Open at umvmvector.h:173 - 0x20784
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!Transactions at kvstorebenchmark.cpp:865 - 0x65B7
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!operator() at kvstorebenchmark.cpp:928 - 0x9126
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!callnewh at <unknown_file>:<unknown_line> - 0x4EB17
in c:\windows\system32\kernel32.dll!BaseThreadInitThunk at <unknown_file>:<unknown_line> - 0x1796E
in c:\windows\system32\ntdll.dll!RtlUserThreadStart at <unknown_file>:<unknown_line> - 0x6A26B


#===============================================================================
# Diagnostic # 9: Redundant memory fence for persistence
#-------------------
Memory fence is redundant for persistence ordering but may be necessary for other purposes
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!Open at umvmvector.h:173 - 0x207D1
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!Open at umvmvector.h:173 - 0x207CF
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!Transactions at kvstorebenchmark.cpp:865 - 0x65B7
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!operator() at kvstorebenchmark.cpp:928 - 0x9126
in d:\threemisses\umvmtest\kvstorebenchmark\x64\release\kvstorebenchmark.exe!callnewh at <unknown_file>:<unknown_line> - 0x4EB17
in c:\windows\system32\kernel32.dll!BaseThreadInitThunk at <unknown_file>:<unknown_line> - 0x1796E
in c:\windows\system32\ntdll.dll!RtlUserThreadStart at <unknown_file>:<unknown_line> - 0x6A26B


Analysis complete. 9 diagnostic(s) reported.
------------
Steve Heller

[Michael Tutin]

Hi Steve,

Do you terminate application while running Inspector analysis or you observe a problem with persistence memory when you terminate the program without analysis? It is not quite clear from your description. In general, to have comprehensive Inspector analysis, application should run normally to completion to record all memory events for both ‘store’ and ‘recovery’ phases.

Out of order stores analysis can be enabled by adding -check-out-of-order-store option to the report command. It is turned off by default. 

If there are no problems reported, it would be worth trying -additional-diagnostics option for that report command. Maybe it will detect something new there.

[steve]

On Thu, 2 Jul 2020 05:55:53 -0700 (PDT), Michael Tutin <mtu...@gmail.com> wrote:

>Hi Steve,
>
>Do you terminate application while running Inspector analysis or you
>observe a problem with persistence memory when you terminate the program
>without analysis? It is not quite clear from your description. In general,
>to have comprehensive Inspector analysis, application should run normally
>to completion to record all memory events for both ‘store’ and ‘recovery’
>phases.

I had the program use exit(1) in the middle of a transaction while running it under inspector for the "before unfortunate event" phase. Then I ran the
recovery phase as "after unfortunate event" under the inspector.

>Out of order stores analysis can be enabled by adding

>*-check-out-of-order-store* option to the *report *command. It is turned

>off by default.
>If there are no problems reported, it would be worth trying

>*-additional-diagnostics* option for that *report *command. Maybe it will
>detect something new there.

Ok, I'll try that next.

Thanks.
------------
Steve Heller

[Niall Douglas]

On Thursday, July 2, 2020 at 2:03:40 PM UTC+1, steve wrote:

I had the program use exit(1) in the middle of a transaction while running it under inspector for the "before unfortunate event" phase. Then I ran the
recovery phase as "after unfortunate event" under the inspector.

exit() induces an orderly process exit. Destructors will be invoked, data will be synchronised out, cleanup performed.

abort() is less orderly, but still quite orderly.

Deliberately dereferencing a null pointer is getting much better, but the signal handlers still run, and they still do trivial cleanup.

Terminating one's own process id is better again by preventing signal handlers firing, but the OS still does cleanup.

This is why I said that you need to kill -9 a VM. Then its kernel does no cleanup, everything just halts exactly mid-flow. Any bad i/o ordering then becomes revealed.

Niall

[steve]

I powered down the VM via the "stop" button in Hyper-V. This acts exactly as turning off the power switch did: my data was corrupted.

I've also tried the same code with newer firmware, namely 01.02.00.5355. No change.
------------
Steve Heller