File system gets damaged undr Android (with a solution)

[Lurker]

I'm working on a one more port of exFAT for Android. The project is here:
https://github.com/Lurker00/Android-fs

Probably the problem I faced is only Android related, or even particular Android build related, but nevertheles...

Digging the source code, I've found out that cmap (FAT) is flushed only in exfat_flush() function, which is called only from fuse_exfat_fsync() and exfat_unmount(). So, cmap is updated only on fsync or umnount events.

The problem is that under Android it is never called: Android does not call fsync, and it does not unmount the device on reboots and power offs! So, taking exFAT code as it is, any write causes file system corruption! That's the problem!

Even if Android would unmount the device on reboots and power offs, it is not a solution: a user may remove SD card without manual unmount, or accidental power loss may happen. So, I'm seeking for a solution that keeps the file system consistent, but does not update the FAT too often.

Currently, I've implemented cmap flush in fuse_exfat_release(), but, to keep compatible functionality, I do it only if the device was mounted with "sync" option (not in use by exFAT, as I understand). So far so good: I've seen lost clusters only once, on an accidental reboot, when an application kept its file opened.

My questions are:
1. Why, in general, it is implemented this way? Does desktop Linux call fsync often enough to prevent file system corruption?
2. Could anybody more experienced suggest me a better place(s) to flush cmap?
3. Am I right thinking that "sync" option has no effect in original exFAT code?

In particular, I'm thinking if I can flush cmap on each exfat_flush_node(), believing that flash device driver is smart enough to delay actual writes and it will not drain the flash write cycles too fast...

Thanks in advance!

[Andrew Nayenko]

Hi,

If you don't mind I'll replay to both of your messages in a single email.

> I'm working on a one more port of exFAT for Android. The project is
> here: https://github.com/Lurker00/Android-fs
>
> Probably the problem I faced is only Android related, or even
> particular Android build related, but nevertheles...
>
> Digging the source code, I've found out that cmap (FAT) is flushed
> only in exfat_flush() function, which is called only from
> fuse_exfat_fsync() and exfat_unmount(). So, cmap is updated only on
> fsync or umnount events.
>
> The problem is that under Android it is never called: Android does
> not call fsync, and it does not unmount the device on reboots and
> power offs! So, taking exFAT code as it is, any write causes file
> system corruption! That's the problem!

That's a severe problem. Fuse-exfat expects that filesystems are cleanly
unmounted.

> Even if Android would unmount the device on reboots and power offs,
> it is not a solution: a user may remove SD card without manual
> unmount, or accidental power loss may happen. So, I'm seeking for a
> solution that keeps the file system consistent, but does not update
> the FAT too often.
>
> Currently, I've implemented cmap flush in fuse_exfat_release(), but,
> to keep compatible functionality, I do it only if the device was
> mounted with "sync" option (not in use by exFAT, as I understand). So
> far so good: I've seen lost clusters only once, on an accidental
> reboot, when an application kept its file opened.
>
> My questions are:
> 1. Why, in general, it is implemented this way?
> Does desktop Linux call fsync often enough to prevent file system
> corruption?

It depends on software that works with files. If it calls sync(2) or
synfs(2), fuse-exfat does sync data to the device. But often there in
only on sync--on unmount.

> 2. Could anybody more experienced suggest me a better place(s) to
> flush cmap?

If you sync only on file close can loose data in many other cases
(metadata changes, truncation, file removal etc.).

> 3. Am I right thinking that "sync" option has no effect in original
> exFAT code?

Yes, fuse-exfat does not support it.

> In particular, I'm thinking if I can flush cmap on each
> exfat_flush_node(), believing that flash device driver is smart
> enough to delay actual writes and it will not drain the flash write
> cycles too fast...

The underlying device *must* perform physical writing when you issue
sync. That's the point of sync.

Adding sync into exfat_flush_node() will indeed make things more
reliable. But it's a *very* expensive procedure.

> I've found out that exFAT source code relies on
> _FILE_OFFSET_BITS=64, which should force the C library using off64_t
> and lseek64/pread64/pwrite64 instead of off_t/lseek/pread/pwrite. The
> problem is that Android, by default, used Bionic libc, which ignores
> this define!

Unfortunately Bionic is a second class citizen in the Android world.

> I've seen a solution to typedef off_t as a 64-bit value before
> including any other header, but it breaks standard stream I/O
> functions, like fprintf. So, I've replaced all occurences of off_t
> with off64_t, and added conditional calls to
> lseek64/pread64/pwrite64.
>
> But why exFAT is written such an non-reliable way? Why not to use
> required 64-bit file system calls and types explicitly?

Let's first define terms. The ability to work with files larger than 2GB
is called Large File Support (LFS). Native LFS means 64-bit off_t.
Transitional LFS means that off_t remains 32-bit, but in addition there
is 64-bit off64_t. As the name suggests, Transitional LFS was a
temporary solution while operating systems transition to Native LFS. And
they did make this transition many years ago.

Then came Android. Whatever reasons the architects had (if any) to
abandon Native LFS, it was just stupid.

> If there is a reason, then why not to use a non-standard typedef, and
> wrap those calls, to localize this for the sake of easy porting?

The reason to not use non-standard things is that they are not standard.
Native LFS is supported by all modern systems except Android.
Transitional LFS is supported by GNU/Linux and Bionic/Linux, but not Mac
OS X. I thought about using loff_t/off64_t and making wrappers for the
system functions that use it, i.e. implementing functionality that must
be in libc. This does not look very good. Any other ideas?

--
Andrew Nayenko <res...@gmail.com>

[Lurker]

If you don't mind I'll replay to both of your messages in a single email.

It was a bad idea: the topics are quite different.
 

That's a severe problem. Fuse-exfat expects that filesystems are cleanly
unmounted.

Good assumption for occasional use of a card reader or USB stick, but inacceptable for heavy use of a microSD in a battery powered device :(

> 2. Could anybody more experienced suggest me a better place(s) to
> flush cmap?

If you sync only on file close can loose data in many other cases
(metadata changes, truncation, file removal etc.).

If I understand correctly, fuse_exfat_release is called on directories updates as well, isn't it? If it does not, then yes, I have to find more places to flush the cmap (free cluster bitmap).

Adding sync into exfat_flush_node() will indeed make things more
reliable. But it's a *very* expensive procedure.

I never asked about "sync" operation for underlying device. There is no a problem with underlying device per se. The problem is to keep that "cmap" consistent. It is cmap that (originally) is written into the flash card not so often (almost never under Android).
 
OK, I just create a directory "test", rebooted the device, and exaftfsck said, "cluster for 'test' is not allocated". So, I need to flush cmap in some other places as well. That's what I wanted to know... It seems, exfat_flush_node function is the safest place!

 I thought about using loff_t/off64_t and making wrappers for the
system functions that use it, i.e. implementing functionality that must
be in libc. This does not look very good. Any other ideas?

No. But, as for me, it is not a good idea to expect identical clib implementation on each and every platform, even if they all are based on Linux. It is very common in embedded world to cut down functionality. So, I'd go for loff_t and wrappers ;)

[Andrew Nayenko]

>> That's a severe problem. Fuse-exfat expects that filesystems are cleanly
>> unmounted.
>
> Good assumption for occasional use of a card reader or USB stick, but
> inacceptable for heavy use of a microSD in a battery powered device :(

This assumption exists since UNIX ages. Filesystems must be unmounted.
Otherwise it's impossible to ensure good performance (as you might
have noticed). Unfortunately DOS did not have such concept probably
because it was a single-task system (unlike UNIX).

You are right that hardware and software failures cause unclean
dismounts. And it's better to have some protection when this happens.
That's why most modern filesystems use a journal.

>> > 2. Could anybody more experienced suggest me a better place(s) to
>> > flush cmap?
>>
>> If you sync only on file close can loose data in many other cases
>> (metadata changes, truncation, file removal etc.).
>
> If I understand correctly, fuse_exfat_release is called on directories
> updates as well, isn't it?

No, it's not called in this case.

>> Adding sync into exfat_flush_node() will indeed make things more
>> reliable. But it's a *very* expensive procedure.
>
> I never asked about "sync" operation for underlying device. There is no a
> problem with underlying device per se. The problem is to keep that "cmap"
> consistent. It is cmap that (originally) is written into the flash card not
> so often (almost never under Android).

You have to make "sync" to make kernel physically write data to the
device. In the case of force eject there is no difference whether
updated cmap has been put into kernel's page cache or remained in the
fuse-exfat process' memory. It will be lost. So you have to enforce
physical writes after each FS modification to ensure consistent state.

> OK, I just create a directory "test", rebooted the device, and exaftfsck
> said, "cluster for 'test' is not allocated". So, I need to flush cmap in
> some other places as well. That's what I wanted to know... It seems,
> exfat_flush_node function is the safest place!

I think so.

>> I thought about using loff_t/off64_t and making wrappers for the
>> system functions that use it, i.e. implementing functionality that must
>> be in libc. This does not look very good. Any other ideas?
>
> No. But, as for me, it is not a good idea to expect identical clib
> implementation on each and every platform, even if they all are based on
> Linux. It is very common in embedded world to cut down functionality. So,
> I'd go for loff_t and wrappers ;)

When I started the project in 2009 I didn't think about embedded. :)
Anyway, good luck!

--
Andrew Nayenko <res...@gmail.com>

[Lurker]

I've added flushing of "cmap" into exfat_flush_node(), and then, whatever operations I tried, it survived reboots.

I don't care about hot ejects, and will not insert any syncs. The main problem was power offs and reboots, i.e. usual user activity.

Thanks a lot for your explanations, and, of course, for the project itself!

[Lurker]

I've read Samsung' exFAT driver:
https://github.com/wanam/Adam-Kernel-GS4-LTE/tree/master/drivers/exfat

They don't mind to write down cluster bitmap every time they allocate or free a cluster! Also, they update superblock with dirty/clean flag (EXFAT_STATE_MOUNTED in your code) before and after every file system change...

So I've implemented cmap flush on node flush, unlink/rmdir, file growth, i.e. right after any change in cmap, and feel even better :)