fflush and fat fs

[hachi1337]

Hi,

I have a program that logs data to sdcard - a lot. For each write I use fwrite followed by the fflush (in one second there are about 50 of such writes). Every operation ends with success (including fopen). When I let program end by itself, the data is there on SD card, BUT when I force power off the data is not there. It is 100% reproducible. I can wait even 1 minute - which should store a few megabytes of data, but when force power off occurs that whole data is lost - it's like fat table is not updated?

I can reproduce this issue with following code:

====] CODE [====
unsigned long long i;
FILE *f;
if ((f = fopen("/mnt/sd/data", "a")) == NULL)
{
perror("open fail");
return 1;
}

const char *d = "data store\n";

for (i = 0;; ++i)
{
if (fwrite(d, strlen(d), 1, f) != 1)
perror("write error");

if (fflush(f) != 0)
perror("fflush error");
}

return 0;
====] END CODE [====

Just run it, wait however long, even 5 minuts. Force power off (unplug directly) then power back on and check if /mnt/sd/data has something inside - its empty on by board.

But then, add "i != 1000" into for loop, and run it again without powering board off - oh look, the data is there!

Am I missing something in fat configuration? Or maybe should I fclose file for fat table to be updated? That wouldn't be good.

I appreciate any input on this.

[patacongo]

You should enable CONFIG_DEBUG_FS_ERROR,  CONFIG_DEBUG_FS_WARN, and CONFIG_DEBUG_FS_INFO and see what is happening.

Data loss on power off is always possible if the directory entry is never updated.

Note that you continue writing even after an error is reported.  What happens if you break out of the loop after an error is reported.

[patacongo]

Hmmm.. fflush() just issues a write() to write the buffered C data.  It does not call fsync() so the directory entry would never be written to the SD card in this test case: The data would be continuously written, fail each time, but the directory entry is never written so when you power cycle, the directory entry still says that the length of the file is zero.

fsync() will be called when you call fclose().  So I would expect the huge file to be there if you exit the loop on an error and call fclose().

[patacongo]

This is a pretty good explanation:  https://lwn.net/Articles/457667/  although its terminology to distinguish writing to the file system, writing to the driver, and writing the hardware are a little imprecise.  Also, ignore discussion related to O_SYNC and D_SYNC; these are not supported by NuttX:

Basically it is like this:

fflush() dumps the C buffered I/O to the file system.  That data may not be cached in memory (with small writes and FAT it definitely will be cached in memory).  That includes both the cached directory entry and the cached file data.

As the LWN article suggests, if you want to flush the data from file system to the block driver, then you need to call fsync() or fclose().

NOTE:  Just because the data has been written to the block driver, it is still not in hardware.  That requires a little additional time but will happen a little latter.  If you were to power off immediately after fsync(), data would still be lost.  But a few hundred milliseconds after fsync(), it should be okay.

So my conclusion is the NuttX behavior is correct but your example is wrong.  You are getting the expected result if you never fsync() the file.

[patacongo]

Just a warning.  If you were to call fsync() after each fflush() you would completely destroy your performance and perhaps your SD card as well!

You are currently writing 11 bytes per loop.  A sector is 512 bytes.  No actually write to hardware occurs until the full 512b buffer is full, about 46.5 times through the loop.  If you fsync'ed each time through the link, you would write that data sector 46.5 times per loop.  Worse, you would also the directory sector 46.5 times.  That would be (a little more than) 46.5 Kb of data writes.  And even worse, I think it would also erase each sector before writing it (not sure of that).  In any case, it would be very slow.

The underlying SD technology is probably NAND and the expected write life is possibly not so high.  SD internal logic uses and ECC and sparing logic so it generally tolerates a pretty high level of failures (with delays up to 5 seconds ... really! ...  for the worst case sparing delay).  So your SD card would probably become unusable after you ran such a test several times.

[hachi1337]

Hi,

Thank you for your answer.

> Data loss on power off is always possible if the directory entry is never updated.

> Note that you continue writing even after an error is reported.  What happens if 

> you break out of the loop after an error is reported.

My example as you notice is just an example to reproduce this problem. And 

thing is these functions *never* fail - so there is nothing really a reason

to break, those prints are only informational and a proof functions return 0.

If there was an error I would definately debug it.

I can live with small data loss, it's obvious to loose some data on power 

loss. And by small I mean few of kilobytes, but here I loose *whole* file 

with many megabytes if it is not closed.

> So my conclusion is the NuttX behavior is correct but your example is wrong. 

> You are getting the expected result if you never fsync() the file.

So if I understand correctly - I can open file (mode 'a'), put in there hundreds of 

megabytes calling *only* fwrite(), all of which return SUCCESS (there is not a single

error in whole execution) but if I never call fsync() (or fclose()) and power lose 

occurs I lost *whole* file, because only fsync() can update file entry on FAT?

I expected things to work like that (consider calling only fwrite() in a loop):

1) on fwrite() data is copied to some internal library buffer - ok

2) as data written is less than 512 bytes (or different value depending on 

library buffer size) nothing else happens

3) if buffer fills to certain threshold (like that 512 bytes) data is flushed

from library to kernel buffers.

4) when kernel buffer overflows, kernel will send data to SD card.

5) after dumping data to sd card, FAT entry is updated to reflect changes

And it appears point 5 is not done. It will be done once fclose() is called.

Is this correct behaviour of stdio and writing to file?

Later I will turn on those debugs and see what happens.

[patacongo]

My example as you notice is just an example to reproduce this problem. And 

thing is these functions *never* fail - so there is nothing really a reason

to break, those prints are only informational and a proof functions return 0.

These functions should return errors if the card is full.  It may be that the error occurs asynchronously, much later and so it not detected.

I can live with small data loss, it's obvious to loose some data on power 

loss. And by small I mean few of kilobytes, but here I loose *whole* file 

with many megabytes if it is not closed.

Yes, that is unfortunate, but normal behavior for FAT if you never fsync.

 

 

So if I understand correctly - I can open file (mode 'a'), put in there hundreds of 

megabytes calling *only* fwrite(), all of which return SUCCESS (there is not a single

error in whole execution) but if I never call fsync() (or fclose()) and power lose 

occurs I lost *whole* file, because only fsync() can update file entry on FAT?

Yes, that is correct.  Or O_OSYNC if it is supported (it is not).

Returning success may not be correct, depending upon when the error occurs.  I would think that FAT should be smart enough not to allocate sectors past the end of the SD card so any absence of error does sound like a bug.

I would not expect fwrite() to return an error since it only writes into a application buffer, but fflush() should return and error; the write() should fail when the media is full.

 

I expected things to work like that (consider calling only fwrite() in a loop):

1) on fwrite() data is copied to some internal library buffer - ok

2) as data written is less than 512 bytes (or different value depending on 

library buffer size) nothing else happens

3) if buffer fills to certain threshold (like that 512 bytes) data is flushed

from library to kernel buffers.

4) when kernel buffer overflows, kernel will send data to SD card.

5) after dumping data to sd card, FAT entry is updated to reflect changes

And it appears point 5 is not done. It will be done once fclose() is called.

Is this correct behaviour of stdio and writing to file?

No, it would be an incorrect behavior if it did 5.  Meta data (i.e., the directory entry and the FAT) is not flushed to hardware until fysnc or fclose is called. This is why you have to unmounts a FAT volume on Windows or Linux before you remove it.

[hachi1337]

Well Greg, no surprise that you were right;)

For future refference from fsync(2)

> As well as flushing the file data, fsync() also flushes

> the metadata information associated with the file

But also as word of warning

> Calling fsync() does not necessarily ensure that the entry in

> the directory containing the file has also reached disk.  For

> that an explicit fsync() on a file descriptor for the directory 

> is also needed.

But in nuttx it seems calling fsync fixes this "issue" and data is

indeed on SD card after power lose. Looks like I have to add 

parameter "how many bytes of data are you willing to lose";) and

manually call fsync after that ammount of data has been stored.

Thanks for fixing me Greg!

[hachi1337]

Too bad there isn't pure c89 sollution and one needs to use OS specific functions.

Well... I guess I can always close and reopen file.

[Sebastien Lorquet]

 

Hello

Ensuring data has been written to disk in proper order is a typical requirement of embedded platforms.

It is not usually the case on normal desktop/server platforms, where performance and caching is more important than actual non volatile storage device access, in order to reduce IO operations and improve disk throughput.

That's why, in embedded devices, we need file systems that ensure this device consistency as best as possible. This is a responsibility of the filesystem, not of the C language.

That's also why FAT family filesystems (and SD cards) are a total nonsense in embedded devices where power can be shut down without any warning or backup power.

SmartFS attempts to go in this direction but I am not aware of the offered consistency guarantees.

littlefs from mbedOS was promising in this domain by promising atomic FS updates as well as wear leveling, but I lack time and no one offered to reimplement it for nuttx. Specs are available.

Sebastien

--
You received this message because you are subscribed to the Google Groups "NuttX" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nuttx+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

__._,_.___

---

Posted by: Sebastien Lorquet <seba...@lorquet.fr>

---

Reply via web post

•

Reply to sender

•

Reply to group

•

Start a New Topic

•

Messages in this topic (1)

---

Have you tried the highest rated email app?

With 4.5 stars in iTunes, the Yahoo Mail app is the highest rated email app on the market. What are you waiting for? Now you can access all your inboxes (Gmail, Outlook, AOL and more) in one place. Never delete an email again with 1000GB of free cloud storage.

---

Visit Your Group

• Privacy • Unsubscribe • Terms of Use

.

__,_._,___