Shutdown behavior and flash storage devices

43 views
Skip to first unread message

Dave Nadler

unread,
Jul 24, 2021, 11:19:42 AMJul 24
to
Hi All - I'm wondering what other folks do about this issue...

Consumer flash storage devices (USB memory stick, SD card, etc)
have a nice internal wear-leveling controller. When one does a write
operation, lots of sectors may be internally rejiggered to provide
uniform wear (so things that are never rewritten from the application
point of view are actually moved around and rewritten). This activity is
invisible from normal application's point of view, but takes some time.
If the device is powered off during such activity, data corruption...
So, its necessary to provide the device some time after the last write
before it is powered off. Plenty of embedded stuff I've seen too often
corrupts memory at power off, and of course the device manufacturers
blame the memory manufacturers...

So, how do you avoid this kind of corruption in your designs?

Thanks,
Best Regards, Dave

Theo

unread,
Jul 24, 2021, 11:37:26 AMJul 24
to
If it's a USB stick, SCSI has an UNLOAD command which is what you do when
you eject removable media like a CD. I would hope that a USB stick (USB
mass storage = SCSI protocol) would use it as a hint to flush its write
cache and get ready to be removed, but I don't actually know if it does.

Maybe they can also detect the USB signalling going away, which is what
happens when you pull the stick, a few ms before the power disconnects, due
to the layout of the USB A connector.

Enterprise SSDs have power loss protection so you can pull the power at any
time and they'll tidy up nicely using stored energy in a bank of capacitors.

Theo

Don Y

unread,
Jul 24, 2021, 9:10:41 PMJul 24
to
Two (transient) copies of each "chunk" of data. Update the second
copy (on or before impending power fail). When complete, toggle
a pointer to reference it as the most recent (assuming you don't
have metadata that automatically provides that sort of information).

The "first" copy can now be removed and its space reallocated to
some other "second copy"

Then, move on to the next "chunk". Design your algorithms so that
you can tolerate some chunks being "stale" (not updated in time)
wrt the others.

Dimiter_Popoff

unread,
Jul 25, 2021, 7:18:27 AMJul 25
to
Don, this strategy is obviously good (standard?) for say disk writes
etc., but I don't know if it will be OK for flash as it involves erase
then write, AFAIK "erase" is the most killing part. Perhaps they do
a lot of "refresh", i.e. read while they can then rewrite the same
locations? I don't really know, just wondering how they do it.

Dave Nadler

unread,
Jul 25, 2021, 8:50:05 AMJul 25
to
On 7/24/2021 9:10 PM, Don Y wrote:
>> So, how do you avoid this kind of corruption in your designs?
>
> Two (transient) copies of each "chunk" of data.  Update the second
> copy (on or before impending power fail).  When complete, toggle
> a pointer to reference it as the most recent (assuming you don't
> have metadata that automatically provides that sort of information).
>
> The "first" copy can now be removed and its space reallocated to
> some other "second copy"
>
> Then, move on to the next "chunk".  Design your algorithms so that
> you can tolerate some chunks being "stale" (not updated in time)
> wrt the others.

Don, I don't think that works. The entire contents of the volume can
be corrupted if power is removed during wear-leveling operations that
are invisible to the host OS. And I guess you are assuming no file
system is in use (because file system internals would still get corrupted)?

Don Y

unread,
Jul 25, 2021, 12:20:38 PMJul 25
to
Keep a spare block (or two) that are already erased, on hand
(this cuts into the capacity of the device but not in a meaningful way).

As you need to write data to the device, select a page from the
erased block to accept your data. When the number of "clean pages"
falls below a threshold, schedule another (free!) block to be erased.

If the erasure is interrupted, then you see the "free" block as
free-but-not-erased.

If the page write is interrupted, then you see the page as unclean
*and* the data it WANTED to contain as still residing in the
"previous" location.

There is always going to be a window in which you can get screwed
by unfortunate power cycling. But, you can shrink this window
(at some manageable cost).

If, instead, you only perform the program/erase/write cycle
on the "parameter block" when you need to update the "parameters",
then you have to move the existing parameters into volatile storage.
Then, program/erase the block. Then write the parameters back.
Then, verify the write.

This leaves the parameters (in volatile memory) "exposed" for a long
time -- seconds per block. If you "set up" the write operation ahead
of time, you can trim this to a millisecond, or so.

Don Y

unread,
Jul 25, 2021, 12:30:45 PMJul 25
to
[I don't use filesystems; do you use a filesystem to manage your *RAM*? :> ]

I use "raw" devices and have the FTL (and wear-leveling) in my firmware.
So, I know what the hardware is being asked to do when I talk to it;
there's no "black magic" sitting in the middle that I can't quantify/qualify.

If you're going to use a "smart" device, then you have to come up with a
strategy that lets you recover from arbitrary levels of corruption
(because you don't know what that device is going to be doing when the
fit hits the shan).

I suspect you will see your losses limited to a single block (or so) as
it would require more resources to transfer a block's contents into
RAM while another block is nominated to receive them.

There might be different strategies employed by different device vendors.
E.g., one may defer the program/erase of that block until a later time;
another may opt to do it "now".

I don't see how you can expose enough of the internals of a "smart" device
in a manufacturer/model-independent manner. I.e., even storing a hash with
each "file", there's no guarantee as to WHERE that will reside on the device.
And, when it might "move" (be exposed to loss).

[You also have to be prepared for "good data" being lost in such a reshuffling;
in my case, I can ensure "what I have, I keep (but may lose something NEW)"]

If your needs are for things like configuration parameters (presumably few),
can you consider an EEPROM? Use FLASH for bigger things -- that tend to
change less frequently (and in more user-visible ways)?

Kent Dickey

unread,
Jul 25, 2021, 2:53:40 PMJul 25
to
It is not necessary to do this. Here's a very simple way to recover with
no data loss ever. It is simply "log-only" or "journal-only" storage.

The device keeps 10% capacity reserved (pre-erased) (or less, this is a
performance number). The driver has enough RAM to track where every logical
block is (this isn't much RAM). And then writing is always done as a
logfile--write whatever the user data is starting at block 0, and move up.
After writing user data, write a log entry (this can be just right after the
user data). Writes never go to the block the user indicates, they always go
only to the log pointer. Reading looks up where that block's latest copy is,
and reads that.

When you're at less than 10% space left (near the end of the device for
first-pass writes), you simply read from +10% from the current log write
position, and write it to the log (compacting it to avoid re-writing any
blocks which are now obsolete) until you free up space. Since many blocks are
rewritten, when we compact them we free space. Once copied, then erase the
blocks which were compacted. Then just keep growing the "log" through the
newly erased blocks.

Upon power up, scan entire device for the log info, and rebuild the RAM index.

Power can be removed at any time, and fully recovered, although obviously the
most recent data may be lost, but we can roll back to the last consistent
state just like journaled file systems (the log entries need a checksum so
we can validate them). Wear leveling is achieved by design.

There are lots of small ways to improve the above, so do that.

I have no idea what actual devices do, I suspect it's something much more
complex and less robust.

Kent

Dimiter_Popoff

unread,
Jul 25, 2021, 3:51:52 PMJul 25
to
If you have no knowledge of how the flash medium is written - which
would normally be the case with USB sticks or SD cards etc. - the most
efficient way to mitigate data loss probability is to do as much
write cacheing as you can afford at system level. Of course again
you will be prone to data loss on unexpected power loss but updating
the write cache will be done at (much) longer intervals and in
relatively brief bursts, thus the probability to be hit gets lower.
The rest can only be got rid of by having large enough caps on
the power supply and an early warning that power has been lost
(usually at the input of the stepdowns and you are running on fumes.

Of course you can employ various strategies to mimick flash drive
behaviour at a higher level, Don suggested one, Kent also did,
but these mean having different drivers for different media and
all the consequences of that. I prefer to talk to standard SCSI
or ATA behaving things and leave it to them to handle their
unique specifics. [I remember how I *had to* introduce write
cacheing at system level, in the mid 90-s I started to do backups
under DPS to magneto-optical devices; early days DPS would
simply update its CAT (Cluster Allocation Table) every time
it got modifier and these sectors got corrupted before the backup
would finish.... :-) ].

Dimiter

======================================================
Dimiter Popoff, TGI http://www.tgi-sci.com
======================================================
http://www.flickr.com/photos/didi_tgi/

Don Y

unread,
Jul 25, 2021, 4:20:12 PMJul 25
to
You're missing the point/role of the internal controller in the mix.

Imagine 8 blocks of memory in the device. Blocks 1, 2 and 3 (no special
significance) contain highly static code/data. They get written *once*
when the device is manufactured. So, each has seen *1* program/erase/write
cycle.

Meanwhile, blocks 4-8 are constantly hammered on. The code is constantly
updating the "files" that occupy those blocks (the files don't TOUCH the
first 3 blocks, by definition).

After 1,000 updates, blocks 4-8 have seen 1,000 program/erase/write cycles.

The underlying FLASH device will have a MAXCYCLES specified, based on the
technology used (SLC, MLC, TLC, QLC, etc.), process geometry, etc.
For consumer devices, this number tends to be lower -- because consumers
tend to want to purchase "big" instead of "durable" (so, MLC/TLC/etc.
technology).

Assume the MAXCYCLES is 1,002 (actual numbers are unimportant -- if Dave feels
he will be taxing the medium over the life of his design).

So, after two more updates, the device is *broken*. Because blocks 4-8 will
each have hit their 1,002 cycle limit and stopped working (yeah, I know it's
not a brick wall; but there is *some* number at which ECC errors prove
unmanageable and the controller marks those blocks (4-8) as bad.

Meanwhile, 1,2 and 3 each have 1,001 cycles of wear available -- that they
aren't (and WON'T!) be using. Had those 3,003 cycles been "shared" among
all of the blocks, then the files being stored in 4-8 would still be
writable (even though the next write might be into blocks 1, 3, 5, 6 and 8).

So, the controller has to deliberately *move* a copy of the data in blocks
1, 2 and/or 3 into 4-8 -- consuming one cycle of 4-8. But, freeing up
1001 cycles in 1, 2, 3!

You, on the outside, can't tell how it is doing this, when it is doing this
or even *if* it is doing this! So, you can't predict what parts of the flash
will be corrupted as power goes away. Maybe the VTOC gets hosed (so you can't
even FIND the files). Or, some bookkeeping metadata used by the controller.

The logging idea works for magnetic media where what's there, stays there,
and all you have to worry about is whether or not the *new* stuff made it in
"under the wire", or not.

As an analogy, next time your RAID array is rebuilding, cycle power.
See how easily it sorts out WHERE it was, in the process (and think
about the resources that it spent to make that possible -- all in a $5
thumb drive??)

Don Y

unread,
Jul 25, 2021, 4:27:09 PMJul 25
to
On 7/25/2021 12:51 PM, Dimiter_Popoff wrote:
> If you have no knowledge of how the flash medium is written - which
> would normally be the case with USB sticks or SD cards etc. - the most
> efficient way to mitigate data loss probability is to do as much
> write cacheing as you can afford at system level.

One would assume you'd adopt such a strategy regardless -- as the
durability of FLASH is limited. (write a piece of code that
just keeps rewriting the contents of FLASH and see how long before
you get write/verify errors)

> Of course again
> you will be prone to data loss on unexpected power loss but updating
> the write cache will be done at (much) longer intervals and in
> relatively brief bursts, thus the probability to be hit gets lower.

The usual trick is to just have a mirror of the FLASH contents
and flush that to the media when you're pretty sure the ship
is sinking.

But, if this is *large*, then you risk only partial writes before
power drops to an unusable/unreliable level.

> The rest can only be got rid of by having large enough caps on
> the power supply and an early warning that power has been lost
> (usually at the input of the stepdowns and you are running on fumes.

I'm not sure that will work with consumer devices. You have no way of
knowing how long it will take to reliably flush ALL of the data
out to the physical store.

Remember, if you have a USB interface, then the interface limits the
write rate as well as the media in the device.

> Of course you can employ various strategies to mimick flash drive
> behaviour at a higher level, Don suggested one, Kent also did,
> but these mean having different drivers for different media and
> all the consequences of that.

Exactly. I use "bare" devices so I can see how they respond -- because
the datasheet gives me hard limits.

So, I have to worry about a write *failing* (in which case, it's as if
I lost power prematurely) but not *other* "unchanged" data being corrupted
in the process (neglecting read/write disturb events).

> I prefer to talk to standard SCSI
> or ATA behaving things and leave it to them to handle their
> unique specifics. [I remember how I *had to* introduce write
> cacheing at system level, in the mid 90-s I started to do backups
> under DPS to magneto-optical devices; early days DPS would
> simply update its CAT (Cluster Allocation Table) every time
> it got modifier and these sectors got corrupted before the backup
> would finish.... :-) ].

I think much of that is inconsistent with Dave's idea of "consumer"
(which I read as "inexpensive")

Dimiter_Popoff

unread,
Jul 25, 2021, 5:00:32 PMJul 25
to
On 7/25/2021 23:27, Don Y wrote:
> On 7/25/2021 12:51 PM, Dimiter_Popoff wrote:
> ......
>
>> Of course again
>> you will be prone to data loss on unexpected power loss but updating
>> the write cache will be done at (much) longer intervals and in
>> relatively brief bursts, thus the probability to be hit gets lower.
>
> The usual trick is to just have a mirror of the FLASH contents
> and flush that to the media when you're pretty sure the ship
> is sinking.
>
> But, if this is *large*, then you risk only partial writes before
> power drops to an unusable/unreliable level.

Well I think tens of gigabytes for small drives nowadays, what can you
do. Cacheing it all just is not feasible.
What DPS does is cache directories and CATs on a 4k piece basis
(regardless of the disk block size, 512 2048, 4k) and maintains a
bitmap for its dirty regions - so when it updates to the medium
it writes at least 4k and no more than twice 4095 excessive bytes
or something like that.

>
>> The rest can only be got rid of by having large enough caps on
>> the power supply and an early warning that power has been lost
>> (usually at the input of the stepdowns and you are running on fumes.
>
> I'm not sure that will work with consumer devices.  You have no way of
> knowing how long it will take to reliably flush ALL of the data
> out to the physical store.

Oh with consumer devices you have not built yourself it will probably
never work, of course. They are unlikely to give you an early warning
in the first place. If you want to be safer you just use good rotating
disks.... I think I read somewhere they do their soft shutdown using the
energy in the rotor (and do that since times immemorial), but I have
never verified this.

Don Y

unread,
Jul 25, 2021, 8:35:40 PMJul 25
to
On 7/25/2021 2:00 PM, Dimiter_Popoff wrote:
> On 7/25/2021 23:27, Don Y wrote:
>> On 7/25/2021 12:51 PM, Dimiter_Popoff wrote:
>> ......
>>
>>> Of course again
>>> you will be prone to data loss on unexpected power loss but updating
>>> the write cache will be done at (much) longer intervals and in
>>> relatively brief bursts, thus the probability to be hit gets lower.
>>
>> The usual trick is to just have a mirror of the FLASH contents
>> and flush that to the media when you're pretty sure the ship
>> is sinking.
>>
>> But, if this is *large*, then you risk only partial writes before
>> power drops to an unusable/unreliable level.
>
> Well I think tens of gigabytes for small drives nowadays, what can you
> do. Cacheing it all just is not feasible.

Right. OTOH, if your software is pushing large objects to long term
storage, it likely WANTS them to be written, there. Caching is
a more prudent strategy for small things that may change often
(e.g., configuration parameters, "state", etc.) Pushing them
to the physical store is LOGICALLY correct (from the application's
standpoint) but usually a performance hit and a durability hit
(for media that wear).

> What DPS does is cache directories and CATs on a 4k piece basis
> (regardless of the disk block size, 512 2048, 4k) and maintains a
> bitmap for its dirty regions - so when it updates to the medium
> it writes at least 4k and no more than twice 4095 excessive bytes
> or something like that.

I only have persistent memory in my RDBMS -- everything else is
just RAM (save for small bootloaders). But, the RDBMS node can
afford to have gobs of disk cache to avoid physical writes.

And, I can use different "tablespaces" (the "disk" on which
particular "tables" reside) to deliberately avoid backing certain
tables (like temporary ones) to durable memory.

>>> The rest can only be got rid of by having large enough caps on
>>> the power supply and an early warning that power has been lost
>>> (usually at the input of the stepdowns and you are running on fumes.
>>
>> I'm not sure that will work with consumer devices. You have no way of
>> knowing how long it will take to reliably flush ALL of the data
>> out to the physical store.
>
> Oh with consumer devices you have not built yourself it will probably
> never work, of course. They are unlikely to give you an early warning
> in the first place. If you want to be safer you just use good rotating
> disks.... I think I read somewhere they do their soft shutdown using the
> energy in the rotor (and do that since times immemorial), but I have
> never verified this.

I think the first step is questioning what you *need* to store.

E.g., I trade memory for time with a lot of my algorithms -- pulling
data from disk and using that to build large structures in RAM that
I can use to expedite the algorithms. It would be *nice* to be
able to save those structures in their "large" state -- to save
the time/effort of having to rebuild them when I restart those
applications. But, this gets to be costly if you try to adopt
that practice universally (with all applications).

And, of course, making sure you can "recognize" data that is
corrupt...

Jaded Hobo

unread,
Jul 27, 2021, 1:47:48 PMJul 27
to
A number of upper tier memory device manufacturers sell flash drives,
USB sticks, SS Sata drives and M2 drives with "UPS" capabilities. When
they sense a loss of power they will have just enough onboard juice to
power down in a controller manner.

I have used drives from InnoDisk (
https://www.innodisk.com/en/products/flash-storage/ssd ) on Linux boxes
inside machines with hard on/off switches. With regular drives we'd see
boot problems after 20 to 30 power cycles, with the "UPS" protected ones
we could do hundreds of hard cycles without ill effect.

Antoon

Don Y

unread,
Jul 27, 2021, 2:27:13 PMJul 27
to
What's the (typical) price premium for that "feature"?
Reply all
Reply to author
Forward
0 new messages