Simulating a power failure

[steve]

Hi all,

I'm working on crash consistency for my hash table. I've gotten it to the point where I can kill the process either with Task Manager or by hitting
Ctrl-C and then when I start the program again, it takes up where it left off without losing anything.

But when I actually turn off the power to the machine at the power strip it's plugged into, the data is messed up when I restart.

Is there a way to simulate a power failure so I can relatively easily reproduce this issue? Ideally I'd like a tool that would let me say "kill this
process at this point as though there were a power failure". Does such a tool exist, and if not, is there anything else that might help with this
issue?

Thanks!
------------
Steve Heller

[Adrian Jackson]

Hi Steve,

I guess what you're missing is something that invalidates the data in the caches and stops it automatically getting flushed back to the persistent memory. Apologies if you've already been through this discussion, but can you do it inside a VM? Because something like VirtualBox allows you to power off the machine directly, i.e. VBoxManage controlvm #instancename poweroff. I'm not an expert on how well the persistent memory plays with VMs but given it's working at the hardware level it should do the sensible thing.

cheers

adrianj

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/m7bnffpcu4u5a1hkgumiv3slkrf3vcr2b4%404ax.com.

[steve]

On Tue, 30 Jun 2020 23:09:17 +0100, Adrian Jackson <adrianj...@gmail.com> wrote:

>Hi Steve,
>
>I guess what you're missing is something that invalidates the data in the
>caches and stops it automatically getting flushed back to the persistent
>memory. Apologies if you've already been through this discussion, but can
>you do it inside a VM? Because something like VirtualBox allows you to
>power off the machine directly, i.e. VBoxManage controlvm #instancename
>poweroff. I'm not an expert on how well the persistent memory plays with
>VMs but given it's working at the hardware level it should do the sensible
>thing.

That would probably work but I don't know how to access pmem from a VM. I know it is supposed to be possible but I've never gotten it to work. Would
you happen to know if there any detailed instructions on how to do it?

------------
Steve Heller

[Adrian Jackson]

I've not been through any of these, but I've seen:

https://software.intel.com/content/www/us/en/develop/articles/enhance-vmware-vms-with-intel-optane-persistent-memory.html

https://lenovopress.com/lp1224-enabling-intel-optane-dc-persistent-memory-in-a-linux-virtual-machine

cheers

adrianj

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/8benffl8bveqqs5i82jsrfts0nf5hjp220%404ax.com.

[st...@steveheller.org]

I'm working through these instructions right now:
https://blog.workinghardinit.work/2019/08/12/configure-persistent-memory-for-hyper-v/
Fortunately I have a Windows Server 2019 install that I can use for
testing.

[ppbb...@gmail.com]

For crash-consistency checking, our team uses pmreorder. The whole procedure is described in Chapter 12 in the Programming Persistent Memory book.

The short version is that deterministic failure-atomic algorithm testing requires far more control over the writes to PMEM than you can get by forcing a shutdown of a machine at a particular point in time. And I don't think VMs will help that much unless you are fully emulating the CPU - but that's just my guess.

So what we do instead is record all stores to PMEM, reorder them between barriers, and then verify the integrity of our data structures and algorithms for every possible memory state.

A similar solution is PMTest: https://pmtest.persistentmemory.org/

[steve]

On Wed, 1 Jul 2020 02:14:43 -0700 (PDT), "ppbb...@gmail.com" <ppbb...@gmail.com> wrote:

>For crash-consistency checking, our team uses pmreorder. The whole
>procedure is described in Chapter 12 in the Programming Persistent Memory
>book.

I'll try the Persistence Inspector. That seems most appropriate for my case.

>The short version is that deterministic failure-atomic algorithm testing
>requires far more control over the writes to PMEM than you can get by
>forcing a shutdown of a machine at a particular point in time. And I don't
>think VMs will help that much unless you are fully emulating the CPU - but
>that's just my guess.

Well, the good news is that I'm not having any problem reproducing a failure. It fails every time if I turn off the power while running the program.
However, if I hit Ctrl-C, everything works fine on a restart.

>So what we do instead is record all stores to PMEM, reorder them between
>barriers, and then verify the integrity of our data structures and
>algorithms for every possible memory state.

Yes, I can see how that would be very useful, but from a quick perusal of the documentation on pmreorder in the book, it doesn't seem to be applicable
to my situation. I'm not using your memory pool, persistent_ptr, or other pmdk data structures.

>A similar solution is PMTest: https://pmtest.persistentmemory.org/
>

>?roda, 1 lipca 2020 o 01:16:46 UTC+2 st...@steveheller.org napisa?(a):

------------
Steve Heller

[Niall Douglas]

On Tuesday, June 30, 2020 at 10:26:19 PM UTC+1, steve wrote:

I'm working on crash consistency for my hash table. I've gotten it to the point where I can kill the process either with Task Manager or by hitting
Ctrl-C and then when I start the program again, it takes up where it left off without losing anything.

That's sudden exit consistency. That can be made quite fast.

 

But when I actually turn off the power to the machine at the power strip it's plugged into, the data is messed up when I restart.

Is there a way to simulate a power failure so I can relatively easily reproduce this issue? Ideally I'd like a tool that would let me say "kill this
process at this point as though there were a power failure". Does such a tool exist, and if not, is there anything else that might help with this
issue?

The way I do it is to start 16 KVMs in parallel to run the test program, then kill -9 those KVMs after each a random period of time, then loop that whole thing for 24 hours. 

That's still not sudden power loss safety mind you. When you force kill a VM, the host OS still ensures whatever the VM wrote reaches storage.

If you really want to do it properly, you need a few dozen Raspberry Pis, each with their power connected to a software controller switch. You boot them, run your test program, randomly kill power, loop for 24 hours. Be aware that if you do do this properly, you'll be finding sudden power loss bugs in all sorts of places.

Me personally, I'd take it to VM sudden death consistency, advertise that, call it a day. Most folk nowadays run everything inside Docker or whatever anyway, so they only care about VM consistency. They assume that if the host ever panics or dies, you need to reset the entire system from the cloud.

Niall

[steve]

On Wed, 1 Jul 2020 05:53:58 -0700 (PDT), Niall Douglas <nialldo...@gmail.com> wrote:

>On Tuesday, June 30, 2020 at 10:26:19 PM UTC+1, steve wrote:
>>
>>
>> I'm working on crash consistency for my hash table. I've gotten it to the
>> point where I can kill the process either with Task Manager or by hitting
>> Ctrl-C and then when I start the program again, it takes up where it left
>> off without losing anything.
>>
>
>That's sudden exit consistency. That can be made quite fast.

Actually it slows down my program by about a factor of 10 compared to the version without consistency protection at all, but the performance is still
pretty good.

>> But when I actually turn off the power to the machine at the power strip
>> it's plugged into, the data is messed up when I restart.
>>
>> Is there a way to simulate a power failure so I can relatively easily
>> reproduce this issue? Ideally I'd like a tool that would let me say "kill
>> this
>> process at this point as though there were a power failure". Does such a
>> tool exist, and if not, is there anything else that might help with this
>> issue?
>>
>> The way I do it is to start 16 KVMs in parallel to run the test program,
>then kill -9 those KVMs after each a random period of time, then loop that
>whole thing for 24 hours.
>

>That's *still* not sudden power loss safety mind you. When you force kill a

>VM, the host OS still ensures whatever the VM wrote reaches storage.
>
>If you really want to do it properly, you need a few dozen Raspberry Pis,
>each with their power connected to a software controller switch. You boot
>them, run your test program, randomly kill power, loop for 24 hours. Be
>aware that if you do do this properly, you'll be finding sudden power loss
>bugs in all sorts of places.

Raspberry Pis can use Optane DC PM? Wow!

>Me personally, I'd take it to VM sudden death consistency, advertise that,
>call it a day. Most folk nowadays run everything inside Docker or whatever
>anyway, so they only care about VM consistency. They assume that if the
>host ever panics or dies, you need to reset the entire system from the
>cloud.

How do you define "VM sudden death consistency"? Is that when you force kill a VM as mentioned above?

>Niall
------------
Steve Heller

[Andy Rudoff]

Remember, the CPU caches are caching physical addresses.  Kill all the processes you want, the data is still sitting in the cache and any consistency check you run will find it.  That doesn't prove your program correctly flushed those stores to persistence.  I don't see how VMs help you unless you modify the VM to do something tricky with cacheable stores and that would be a pretty big project.

Your only choices to really test for crash consistency is to actually remove power (which is not a very good choice because it is hard to show test coverage) or use tools the analyze your persistent stores and/or mess with them so they don't appear to be persistent until you flush them.

This is tricky to do right (which is why we wrote libraries and tools and a book to help).

[steve]

On Wed, 1 Jul 2020 06:22:15 -0700 (PDT), Andy Rudoff <an...@rudoff.com> wrote:

>Remember, the CPU caches are caching physical addresses. Kill all the
>processes you want, the data is still sitting in the cache and any
>consistency check you run will find it. That doesn't prove your program
>correctly flushed those stores to persistence. I don't see how VMs help
>you unless you modify the VM to do something tricky with cacheable stores
>and that would be a pretty big project.

Ok, but if I power off the VM without shutting it down normally, shouldn't that at least be a better simulation of a power failure than hitting
ctrl-C?

>Your only choices to really test for crash consistency is to actually
>remove power (which is not a very good choice because it is hard to show
>test coverage) or use tools the analyze your persistent stores and/or mess
>with them so they don't appear to be persistent until you flush them.
>
>This is tricky to do right (which is why we wrote libraries and tools and a
>book to help).

I'm sure it is tricky. I'm going to use Persistence Inspector and see what that turns up.

------------
Steve Heller

[Andy Rudoff]

On Wednesday, July 1, 2020 at 7:27:56 AM UTC-6 st...@steveheller.org wrote:

On Wed, 1 Jul 2020 06:22:15 -0700 (PDT), Andy Rudoff <an...@rudoff.com> wrote:

>Remember, the CPU caches are caching physical addresses. Kill all the
>processes you want, the data is still sitting in the cache and any
>consistency check you run will find it. That doesn't prove your program
>correctly flushed those stores to persistence. I don't see how VMs help
>you unless you modify the VM to do something tricky with cacheable stores
>and that would be a pretty big project.

Ok, but if I power off the VM without shutting it down normally, shouldn't that at least be a better simulation of a power failure than hitting
ctrl-C?

In both cases, killing a VM and hitting control-C, any stores you did that are unflushed to persistence are sitting in the CPU caches and will be seen the next time you read those locations, yet they will be lost on power failure.  I don't see how using the VM changed the situation from the control-C case at all.

-andy

[Adrian Jackson]

As long as you overwrite the caches though the data would be gone. I know this is simpler said than done, but it is possible to invalidate the cpu caches and then wouldn't that get rid of this data after the VM has been powered off?

cheers

adrianj

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/a8c0b864-1052-4c01-848f-89eea01e11b5n%40googlegroups.com.

[Andy Rudoff]

On Wednesday, July 1, 2020 at 9:03:19 AM UTC-6 adrianj.jackson wrote:

As long as you overwrite the caches though the data would be gone. I know this is simpler said than done, but it is possible to invalidate the cpu caches and then wouldn't that get rid of this data after the VM has been powered off?

Normal system activity can evict dirty lines from the cache, which would end up hiding the missing flushes for those cache lines.  Even ignoring that, it isn't clear to me what you're suggesting.  You can't just invalidate the CPU caches since there's system state in there that has nothing to do with pmem.

Before we had tools, I wrote an intrusive consistency checker (required program changes) which is still available in the old "linux-examples" repo.  It was very simple: change MAP_SHARED to MAP_PRIVATE when you map the pmem, now all stores end up in anonymous memory instead of the pmem.  Then, when you call pmem_persist(), a modified version of that call uses the write() syscall to write the data in the specified range from anonymous memory into pmem.  Now you at least have a program whose stores won't show up in pmem unless a flush is performed.  Then I wrote a program that runs a test repeatedly, crashing it after every possible instruction and then running a consistency check afterwards.  Took many hours to run on a small test :-)  But it did prove the flushes were correct.

Of course, the pmemcheck tool ended up being a much better solution so we focused on that.

-andy

[Niall Douglas]

ARM lets you drop CPU caches, but I don't believe Intel can, even within ring 0.

In any case, what you need to test for is correct ordering by software. If the ordering as received by the hardware from software is correct, sudden power loss safety should naturally result, modulo hardware bugs.

CLWB et al is one way of enforcing ordering. But in pure algorithmic terms, fdatasync() is a close substitute, and sync_file_range() is identical. So you can test your algorithm's sequencing correctness, without persistent memory, via Monte Carlo testing.

Niall

To unsubscribe from this group and stop receiving emails from it, send an email to pm...@googlegroups.com.

[Andy Rudoff]

On Wednesday, July 1, 2020 at 9:24:15 AM UTC-6 nialldo...@gmail.com wrote:

ARM lets you drop CPU caches, but I don't believe Intel can, even within ring 0.

You can, the (privileged) instruction is called INVD.  Don't see how that helps here, since it would also discard things you care about.

-andy