DAX volumes disappeared

[steve]

Hi all,

Last night when I shut my computer down in the normal manner, my three DAX pmem volumes were present and accounted for.

This morning when I turned it on, they were gone.
The regions were there and I was able to recreate the volumes, so obviously the devices are fine.
This is a test machine and I can recreate my test data easily, but imagine if it were real data.

How can I find out what happened and prevent it from happening again?

Thanks.
------------
Steve Heller

[Steve Scargall]

It's best to leave a system in a broken state to collect diagnostic data to help root cause the problem. We may have lost the ability to root cause the issue since the namespaces/volumes having been recreated. My first thought is the observation can occur if there was a dirty_shutdown. IIRC, Windows won't map the volumes and will require you to recreate them. Linux will keep the namespaces but attempts to access the data will result in an error. If this is a dirty_shutdown problem (as I think some of your previous issues may have been), this would imply your system isn't ADR compliant, which is mandatory for PMem. 

Persistent Memory Programming –Application Handling of DirtyShutdowns describes how the dirty_shutdown mechanism works and includes Linux and Windows information.

On Windows, check the Event Logs for clues

On Linux, check the /var/log/messages for clues

Also, check your BMC/IPMI logs for any potential issues that may have been reported during POST.

On both Linux and Windows, you can dump the health information using:

> ipmctl show -sensor

       LatchedDirtyShutdownCount

           (Default) The number of shutdowns without notification over the lifetime of the PMem module.

       UnlatchedDirtyShutdownCount

           (Default) The number of shutdowns without notification over the lifetime of the PMem module. This counter is the

           same as LatchedDirtyShutdownCount except it will always be incremented on a dirty shutdown, even if Latch System

           Shutdown Status was not enabled. 

The Windows Powershell 'Get-PmemDisk' command shows if an unsafe_shutdown occurred or not. See https://docs.microsoft.com/en-us/windows-server/storage/storage-spaces/deploy-pmem for an example.

/SteveS