DeviceStorage and file names that are illegal on the underlying file-system

[Andrew Sutherland]

On https://bugzil.la/1140171 we're encountering the problem where
DeviceStorage is refusing to save a filename with a double-quote
character to the VFAT-formatted sdcard. (This isn't surprising,
although it's surprising we went this far without encountering the
problem! Note that I can happily download and save the file on
b2g-desktop on linux because of the awesome power of ext4.)

Prior-spec art seems to be
http://www.w3.org/TR/FileAPI/#file-constructor which does specifying
that a normalization step should be performed on the name. (It says
replace all "/" characters with ":" characters. Since ":" is also
illegal on VFAT, we probably don't want that exactly.) The other file
APIs that I thought existed appear to have all been retracted.

Although it is quite feasible for the Gaia email app to perform a
filename normalization to VFAT, I thought it would be worth discussing
whether it would be advisable for DeviceStorage to automatically
normalize the filename to something legal. I believe callers have
already voided their warranty if they fail to look at the
evt.target.result they get back from addNamed and use that, so this
could work for existing consumers even.

Arguments in favor of having DeviceStorage do it would be:
- We might not be using VFAT in all cases. Other file-systems could
have different rules. (Maybe HFS+ is crazy?)
- It's unlikely all callers will think to test with filenames that are
illegal on VFAT, so not doing it in DeviceStorage is just helping
developers shoot themselves and their/our users in their feet.
- There can be other complications like MAX_PATH on Windows that might
require name truncation, especially since non-chrome JS code has no
chance of knowing the mount points/etc. in play.
- Non-security unicode issues where non-ASCII things don't work.
- Complicated hypothetical security/attack unicode issues like
http://en.wikipedia.org/wiki/IDN_homograph_attack.

In the event there are emerging standards that DeviceStorage will
converge to/etc., I think we should just do what they're doing and I'm
happy for people to point me at them.

Thanks,
Andrew

[Ehsan Akhgari]

It seems that the new FileSystem spec also points to the File
Constructor spec for file name normalization. CCing Arun.

I think it's pretty hard to come up with a future proof normalization
scheme that works across all filesystems, and it should probably be up
to the implementation to protect against some of these unacceptable
characters. The interesting thing about DeviceStorage is that these
files are also expected to be visible to humans, so we should think
about whether saving the file under a different name than the one
provided to the API makes sense. I personally don't think that's a big
issue.

> _______________________________________________
> dev-webapi mailing list
> dev-w...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-webapi

[Jonas Sicking]

I don't think any W3C spec is going to address our needs here. W3C has
explicitly stayed away from allowing writes to the user's filesystem
for various security reasons, and so won't need to tackle this issue.

I think the question here is how we want our DeviceStorage API to
behave. I see two possible options:

1. Silently change the filename to something which can be written to
the filesystem.
2. Fire an error event (which possibly includes a suggested fixed-up name).

I don't really have any strong opinion. Doing 1 definitely runs the
risk of having application databases and data structures get
out-of-sync with what's actually on the filesystem.

So I think 2 would get my vote.

/ Jonas

[Arun Ranganathan]

So when File API (best accessed here:http://dev.w3.org/2006/webapi/FileAPI/
and not as "TR") disallowed "/" it was just reifying existing behavior.
That is, conversion of "/" to ":".

But the VFAT issue is clearly a problem that we didn't anticipate :(

I'm happy to modify this, but some suggestions would be good. In general,
we take UTF16 and try and disallow reserved chars. Maybe we should have an
expanded list of reserved chars that we disallow. Not 100% sure we can be
normative about this in the spec., as Jonas says.

[Ehsan Akhgari]

Dropbox seems to have this list <https://www.dropbox.com/help/145> which we
can probably use. Credits for finding it go to Arun!

--
Ehsan