Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

SD Card File Management user stories and API proposal

48 views
Skip to first unread message

Ray You

unread,
May 15, 2013, 5:59:30 AM5/15/13
to dev-w...@lists.mozilla.org
Hi all,

Exploration and file operation on SD Card has been a required feature for a
long time. The user stories document is at
https://wiki.mozilla.org/User_Stories_of_SD_Card_File_Management_Requirement

Below is my API proposal. navigator.mozSDCard is the entrance.
It represents the root directory of the SD Card.

The rest is almost the same as FileSystem API:
http://www.w3.org/TR/file-system-api/

partial interface Navigator {
readonly attribute FileSystem mozSDCard;
};

interface FileSystem {
readonly attribute DOMString name;
readonly attribute DirectoryEntry root;
};

interface Entry {
readonly attribute boolean isFile;
readonly attribute boolean isDirectory;
void getMetadata (MetadataCallback successCallback, optional
ErrorCallback errorCallback);
readonly attribute DOMString name;
readonly attribute DOMString fullPath;
readonly attribute FileSystem filesystem;
void moveTo (DirectoryEntry parent, optional DOMString newName, optional
EntryCallback successCallback, optional ErrorCallback errorCallback);
void copyTo (DirectoryEntry parent, optional DOMString newName, optional
EntryCallback successCallback, optional ErrorCallback errorCallback);
void remove (VoidCallback successCallback, optional ErrorCallback
errorCallback);
void getParent (EntryCallback successCallback, optional ErrorCallback
errorCallback);
};

interface DirectoryEntry : Entry {
DirectoryReader createReader ();
void getFile (DOMString path, optional Flags options, optional
EntryCallback successCallback, optional ErrorCallback errorCallback);
void getDirectory (DOMString path, optional Flags options, optional
EntryCallback successCallback, optional ErrorCallback errorCallback);
void removeRecursively (VoidCallback successCallback, optional
ErrorCallback errorCallback);
};

interface DirectoryReader {
void readEntries (EntriesCallback successCallback, optional ErrorCallback errorCallback);
};

interface FileEntry : Entry {
};

interface Metadata {
readonly attribute Date modificationTime;
readonly attribute unsigned long long size;
};

dictionary Flags {
boolean create;
boolean exclusive;
};

[Callback=FunctionOnly]
interface EntryCallback {
void handleEvent (Entry entry);
};

[Callback=FunctionOnly]
interface EntriesCallback {
void handleEvent (Entry[] entries);
};

[Callback=FunctionOnly]
interface MetadataCallback {
void handleEvent (Metadata metadata);
};

[Callback=FunctionOnly]
interface FileCallback {
void handleEvent (File file);
};

[Callback=FunctionOnly]
interface VoidCallback {
void handleEvent ();
};

[Callback=FunctionOnly]
interface ErrorCallback {
void handleEvent (DOMError err);
};


--

Best regards,
尤睿(Ray You)





Mounir Lamouri

unread,
May 15, 2013, 6:24:49 AM5/15/13
to dev-w...@lists.mozilla.org, Doug Turner, Jan Varga, Jonas Sicking
Hi Ray,

When I thought about sdcard management in the past I thought that we
could simply have something like navigator.getExternalStorage(id) or
navigator.getExternalStorages() and that we could expose an API that is
very close to the Device Storage API. The advantage is that the same API
could be easily used for any kind of external storage. Why not going
that way?

Also, if we want to somewhat expose the filesystem, we should probably
plug this into the Filesystem API Mozilla recently proposed to the
WebApps WG:
http://lists.w3.org/Archives/Public/public-webapps/2013AprJun/0382.html

I think Jan, Jonas or Doug might have interesting insights on this topic.

Cheers,
--
Mounir
> _______________________________________________
> dev-webapi mailing list
> dev-w...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-webapi

Ray You

unread,
May 16, 2013, 6:02:42 AM5/16/13
to dev-w...@lists.mozilla.org, Mounir Lamouri, hongtang, Doug Turner, Jan Varga, Jonas Sicking, Pin Zhang
Hi Mounir,

It was a few months ago when I started working on SD Card File
Management API
which is originally requested from our partner Spreadtrum, a chipset
company in
China.

At that time, some recently added features of Device Storage API were absent
(Bug 858416 just landed last week), and I found that Google have
proposed and
implemented a Filesystem API which I thought fit our requirements very well.
That's why I designed the API heavily based on Google's proposal.

Device Storage API was originally designed to manage media files of image,
music and video types specially. Even after Bug 858416 landed, there still
lacks some key interfaces when it comes to SD card file management:

- Directory interface
- File operations: copy, move and rename
- Directory traverse: get direct children of a directory

The Filesystem API we proposed looks great, which at this moment I prefer
plugging SD Card File Management API into. I am also willing to
contribute to
it.

Ray You

unread,
May 16, 2013, 10:43:44 AM5/16/13
to dev-w...@lists.mozilla.org, hongtang
> The Filesystem API we proposed looks great, which at this moment I
prefer
> plugging SD Card File Management API into. I am also willing to
contribute to
> it.

Sorry if I cause any confusion in my previous email. Here by "we" I mean
Mozilla.

I mean the Filesystem API Mozilla recently proposed to the WebApps WG
looks great:

http://lists.w3.org/Archives/Public/public-webapps/2013AprJun/0382.html

And I prefer plugging SD Card File Management API into it. I am also
willing to contribute
to it.

--

Best regards,
Ray You
> ???Ray You?

Jonas Sicking

unread,
May 16, 2013, 11:57:31 PM5/16/13
to Ray You, Dave Hylands, dev-webapi
Why doesn't the current DeviceStorage API be enough to address the use
cases here?

Though I don't know what getDeviceStorage("sdcard") and
getDeviceStorageas("sdcard") does now with the new support for multiple
storage areas.

/ Jonas
On May 15, 2013 3:00 AM, "Ray You" <ry...@mozilla.com> wrote:

> Hi all,
>
> Exploration and file operation on SD Card has been a required feature for a
> long time. The user stories document is at
> https://wiki.mozilla.org/User_**Stories_of_SD_Card_File_**
> Management_Requirement<https://wiki.mozilla.org/User_Stories_of_SD_Card_File_Management_Requirement>
>
> Below is my API proposal. navigator.mozSDCard is the entrance.
> It represents the root directory of the SD Card.
>
> The rest is almost the same as FileSystem API:
> http://www.w3.org/TR/file-**system-api/<http://www.w3.org/TR/file-system-api/>
> 尤睿(Ray You)
>
>
>
>
>
> ______________________________**_________________
> dev-webapi mailing list
> dev-w...@lists.mozilla.org
> https://lists.mozilla.org/**listinfo/dev-webapi<https://lists.mozilla.org/listinfo/dev-webapi>
>

Ray You

unread,
May 17, 2013, 8:13:56 PM5/17/13
to Jonas Sicking, Dave Hylands, dev-webapi
Hi Jonas,

I think following are necessary for Device Storage API to cover SD Card
file management user cases entirely:

1. Ability to create a directory
2. Knowledge of existence of empty directories
3. Ability to tell if a given path is a directory or a file
4. Ability to enumerate only the direct children of a directory
5. Ability to copy, move and rename a file or a directory
6. Ability to get attributes of a directory like modification time

--

Best regards,
Ray You

On 5/17/13 11:57 AM, Jonas Sicking wrote:
>
> Why doesn't the current DeviceStorage API be enough to address the use
> cases here?
>
> Though I don't know what getDeviceStorage("sdcard") and
> getDeviceStorageas("sdcard") does now with the new support for
> multiple storage areas.
>
> / Jonas
>
> On May 15, 2013 3:00 AM, "Ray You" <ry...@mozilla.com
> <mailto:ry...@mozilla.com>> wrote:
>
> Hi all,
>
> Exploration and file operation on SD Card has been a required
> feature for a
> long time. The user stories document is at
> https://wiki.mozilla.org/User_Stories_of_SD_Card_File_Management_Requirement
>
> Below is my API proposal. navigator.mozSDCard is the entrance.
> It represents the root directory of the SD Card.
>
> The rest is almost the same as FileSystem API:
> _______________________________________________
> dev-webapi mailing list
> dev-w...@lists.mozilla.org <mailto:dev-w...@lists.mozilla.org>
> https://lists.mozilla.org/listinfo/dev-webapi
>

Jonas Sicking

unread,
May 18, 2013, 6:28:14 PM5/18/13
to Ray You, Dave Hylands, dev-webapi
On Fri, May 17, 2013 at 5:13 PM, Ray You <ry...@mozilla.com> wrote:
> Hi Jonas,
>
> I think following are necessary for Device Storage API to cover SD Card file
> management user cases entirely:
>
> 1. Ability to create a directory
> 2. Knowledge of existence of empty directories
> 3. Ability to tell if a given path is a directory or a file
> 4. Ability to enumerate only the direct children of a directory
> 5. Ability to copy, move and rename a file or a directory
> 6. Ability to get attributes of a directory like modification time

If we change our DeviceStorage API to use the FileSystem spec proposed here

http://lists.w3.org/Archives/Public/public-webapps/2013AprJun/0382.html

then all of these use cases are available except the last one. I.e. we
wouldn't change the fundamental model of DeviceStorage, but rather
just change the details of the file management part of it. So the
getDeviceStorage and getDeviceStorages functions would work
essentially as they do now.

The only think that the current filesystem proposal does not support
is requirement 6 above. Can you provide more details for what that
information would be useful for?

Would you be interested in implementing this API?

There's definitely a few details to iron out, but that should be
pretty quick once we know we have someone that has the time to work on
it. We should also coordinate with the owners of the DeviceStorage API
of course.

/ Jonas

> Best regards,
> Ray You
>
> On 5/17/13 11:57 AM, Jonas Sicking wrote:
>
> Why doesn't the current DeviceStorage API be enough to address the use cases
> here?
>
> Though I don't know what getDeviceStorage("sdcard") and
> getDeviceStorageas("sdcard") does now with the new support for multiple
> storage areas.
>
> / Jonas
>
>> https://lists.mozilla.org/listinfo/dev-webapi
>
>

Ray You

unread,
May 20, 2013, 2:46:59 AM5/20/13
to Jonas Sicking, Dave Hylands, dev-webapi, hongtang
Hi Jonas,

Please see my inline comments below.

--

Best regards,
Ray You

On 5/19/13 6:28 AM, Jonas Sicking wrote:
> On Fri, May 17, 2013 at 5:13 PM, Ray You <ry...@mozilla.com> wrote:
>> Hi Jonas,
>>
>> I think following are necessary for Device Storage API to cover SD Card file
>> management user cases entirely:
>>
>> 1. Ability to create a directory
>> 2. Knowledge of existence of empty directories
>> 3. Ability to tell if a given path is a directory or a file
>> 4. Ability to enumerate only the direct children of a directory
>> 5. Ability to copy, move and rename a file or a directory
>> 6. Ability to get attributes of a directory like modification time
> If we change our DeviceStorage API to use the FileSystem spec proposed here
>
> http://lists.w3.org/Archives/Public/public-webapps/2013AprJun/0382.html
>
> then all of these use cases are available except the last one. I.e. we
> wouldn't change the fundamental model of DeviceStorage, but rather
> just change the details of the file management part of it. So the
> getDeviceStorage and getDeviceStorages functions would work
> essentially as they do now.

Yes, I agree.

> The only think that the current filesystem proposal does not support
> is requirement 6 above. Can you provide more details for what that
> information would be useful for?

The directory attribute needed is modification time, which will be used
similarly to that of the file and useful in the following cases:

1. With directory modification time displayed, it's easy to tell whether
new changes occur within a directory, without going in and check every
file.
2. When browsing photo directories, users can get quick idea of the time
a group of photos are taken.
3. File managers on PC or Android phone display directory modification
time. Users will keep the same experience with our file manager.

> Would you be interested in implementing this API?

Yes, I am interested and would like to implement this API. I can start
working on it next month after finishing the task at hand. Will it be OK?

Jonas Sicking

unread,
May 20, 2013, 6:14:46 PM5/20/13
to Ray You, Dave Hylands, dev-webapi, hongtang
The reason I'm a bit reluctant to expose this information is that one
of the pieces of feedback that we got from Google based on them
implementing a FileSystem API is that metadata like this was some of
the more painful parts to implement, while at the same time providing
relatively little value to authors.

Keep in mind that we'd like to reuse this FileSystem API not just for
DeviceStorage, but also for things like sandboxed filesystem for apps,
drag'n'drop of user files on desktop platforms, reading of .zip files
etc.

So I'd prefer to hold off on adding directory metadata for now and
instead add it once application authors ask for it.

However we should also find ways to enable applications to quickly
read metadata information from files and be notified when this
changes. This way we can solve your usecases 1 and 2 above without
having to rely on directory metadata.

>> Would you be interested in implementing this API?
>
>
> Yes, I am interested and would like to implement this API. I can start
> working on it next month after finishing the task at hand. Will it be OK?

Yup, definitely.

/ Jonas

James Ho

unread,
May 20, 2013, 8:17:00 PM5/20/13
to Jonas Sicking, Jinghua Zhang, Dave Hylands, dev-webapi, Ray You, hongtang
See comments inline.
Being able to manipulate directories of SD card directly (as a file manager app) is a common (mostly a must-have) application requirement in Asia, especially in China and many others. People are so used to treat SD cards as their daily storage. This is why Ray asks for it.

Adding Jinghua for potentially more comments and contexts...:)

Jonas Sicking

unread,
May 20, 2013, 8:59:35 PM5/20/13
to James Ho, Dave Hylands, dev-webapi, Ray You, Jinghua Zhang, hongtang
Yup. I think that's totally fine. The only thing I'd rather not
support is last-modified information for directory objects.

The filesystem API proposed allows full ability to add/remove/rename
directories, including empty directories. And it supports getting
metadata about files.

If getting last-modified information for directories is also
important, please let me know.

/ Jonas

Ray You

unread,
May 21, 2013, 1:27:55 AM5/21/13
to Jonas Sicking, Dave Hylands, dev-webapi, hongtang
Hi Jonas,

Got what you mean. We can hold it off for the moment. Modification time
of a directory is not that critical in the whole project.

--

Best regards,
Ray You

Yuan Xulei

unread,
Jun 14, 2013, 3:13:11 AM6/14/13
to Jonas Sicking, Ray You, dev-webapi, James Ho, hongtang, Dave Hylands, Jan Varga
Hi Jonas,

Ray You and me are colleagues in Beijing Office. We would like to work
together to implement the filesystem API Mozilla
proposed(http://lists.w3.org/Archives/Public/public-webapps/2013AprJun/0382.html)
and we can start to work next Monday.

Ray has made a tentative implementation of Google filesystem API for SD
card file management. See Bug
871876(https://bugzilla.mozilla.org/show_bug.cgi?id=871876) for details.

We plan to implement Mozilla's filesystem API based on the tentative
implementation, as the interfaces and functions of the two are similar.

How can we start and coordinate with other people related?


Thanks,
Yuan

Jonas Sicking

unread,
Jul 31, 2013, 6:14:20 PM7/31/13
to Yuan Xulei, Ray You, dev-webapi, James Ho, hongtang, Dave Hylands, Jan Varga
Hi Yuan,

I fell hopelessly behind in this mailing list.

What is your current status?

The latest draft is available at [1]

More detailed description of the API is available in [2]. The API in
[2] is a bit outdated, but the description still applies. [1] is the
more updated API.

[1] https://etherpad.mozilla.org/filesystem-api-updated
[2] http://lists.w3.org/Archives/Public/public-webapps/2013JulSep/0071.html

/ Jonas

"Yuan Xulei(袁徐磊)"

unread,
Aug 1, 2013, 12:06:37 AM8/1/13
to Jonas Sicking, Ray You, dev-webapi, James Ho, hongtang, Dave Hylands, Jan Varga
Hi Jonas,

Ray has implemented most parts for DeviceStorage that doesn't require
FileHandle.
We need about a week or so to align the implementation to the lastest
draft. We're
going to provide our patch for feedback next week.

I and Ray're following the discussion of the API in public-webapps. It
is really tough
work to create a new API, but we can see the success and we'll all
support you.

Yuan

Jonas Sicking

unread,
Aug 1, 2013, 2:11:35 AM8/1/13
to Yuan Xulei(袁徐磊), Ray You, dev-webapi, James Ho, hongtang, Dave Hylands, Jan Varga
Hi Yuan,

That's great to hear!

A could of things that are not obvious with the API:

* You can pass relative path to most functions on the Directory
interface. I.e. you can call dir.createFile("foo/bar/file.txt") or
dir.move("foo/bar.txt", "other/dir/newname.txt"). However none of the
segments can be ".." or "." and the path can't start with "/". If it
does we should throw an exception.
* The locking around FileHandle is not very obvious. It works a lot
like IndexedDB transactions (except that there is no rollback if
there's an error of course). This means that you have to do something
like this:

The FileHandle has an 'active' flag. This flag is initially set to false.
When we resolve the promise returned from openRead/openWrite we do the
following steps:
1. Create a FileHandle object representing the opened file
2. Set the 'active' flag on the FileHandle to true
3. *Synchronously* resolve the promise using the created FileHandle.
4. Set the 'active' flag to false

This means that while the callbacks registered on the promise are
called, the 'active' flag is true. Note that all these steps happen on
the main thread and that we never return to the event loop in between
any of the steps. It's important that we never return to the event
loop with the 'active' flag set to true.

When readText/readBinary/getFile/write/setSize/flush is called do the following:
1. If the 'active' flag is false, throw an exception and do nothing else
2. Queue send a message to the IO thread to do the requested work.
3. Create a Promise and return it.
4. Once the work is done send a message back to the main thread with the result.
5. Set the 'active' flag on the FileHandle to true
6. Synchronously resolve the promise using the result from the IO thread
7. Set the active flag on the FileHandle to false.
8. If at this point, there are no pending operations for the
FileHandle, close the FileHandle and let the next pending open*
operation start running (if there are any).

Again, steps 5-7 are done without returning to the event loop, this
ensures that we never return to the event loop with the 'active' flag
set to true.

The reason for this complex setup is partially described here:
http://lists.w3.org/Archives/Public/public-webapps/2013AprJun/0217.html

/ Jonas

Yuan Xulei

unread,
Aug 1, 2013, 11:57:56 PM8/1/13
to Jonas Sicking, Ray You, dev-webapi, James Ho, hongtang, Dave Hylands, Jan Varga
Hi Jonas,

Thanks for your clarification.

A few other things we need your information as well:

1. Definition of AbortableProgressPromise

2. Definition of EventStream

3. removeDeep can take File as argument. Is is a mistake
or can it also remove a single file?

On 08/01/2013 02:11 PM, Jonas Sicking wrote:
> Hi Yuan,
>

Jonas Sicking

unread,
Aug 2, 2013, 1:41:10 AM8/2/13
to Yuan Xulei, Ray You, dev-webapi, James Ho, hongtang, Dave Hylands, Jan Varga
On Thu, Aug 1, 2013 at 8:57 PM, Yuan Xulei <xy...@mozilla.com> wrote:
> Hi Jonas,
>
> Thanks for your clarification.
>
> A few other things we need your information as well:
>
> 1. Definition of AbortableProgressPromise

AbortableProgressPromise is a subclass of Promise

callback VoidAnyCallback = void (optional any value);
interface AbortableProgressPromise : AbortablePromise {
void abort();
AbortableProgressPromise progress(VoidAnyCallback);
};

The progress() function always returns 'this'. I.e. x.progress() ===
x; This way you can write:

doAsyncStuff().progress(...).then(...);
or
doAsyncStuff().then(...);

Both do the same thing, except the first allows you to get callbacks
to be informed about progress.

> 2. Definition of EventStream

This one is much more tricky. We are debating what this interface looks like in

https://github.com/slightlyoff/Promises/pull/37

I'm not sure if the class is supposed to be called EventStream or
Cursor. But the harder problem is what exactly the API is supposed to
look like. I think for now we could do something like:

callback ForEachCallback = Promise? (optional any value);
interface Cursor {
Promise<void> forEach(ForEachCallback);
};

The callback registered using forEach is called for each iterated
file/directory. The returned Promise is resolved once we have iterated
all files, or if an error occurs.

> 3. removeDeep can take File as argument. Is is a mistake
> or can it also remove a single file?

It is intentional. You can use removeDeep to remove a single file too
yeah. That way you can always use removeDeep for all removals. You
only need to use remove() if you want to make sure that you are not
accidentally removing a non-empty directory.

/ Jonas

"Yuan Xulei(袁徐磊)"

unread,
Aug 2, 2013, 2:38:03 AM8/2/13
to Jonas Sicking, Ray You, dev-webapi, James Ho, hongtang, Dave Hylands, Jan Varga
Jonas, thanks for your help. We're clear now.

Yuan

Jonas Sicking

unread,
Aug 28, 2013, 5:32:36 AM8/28/13
to Yuan Xulei(袁徐磊), Ray You, dev-webapi, James Ho, hongtang, Dave Hylands, Jan Varga
Hi Yuan and Ray,

I'm not sure if you've followed all the discussions that has happened
on the W3C lists. Not much has changed in the API. The main thing is
that we've tentatively dropped the copy() function since copying files
can already be done using the createFile() function.

I'm always keeping the latest version of the API up-to-date here:
https://etherpad.mozilla.org/filesystem-api-updated

What is the current status with regards to the DeviceStorage
integration? And have you guys looked at all at exposing a sandboxed
filesystem yet?

/ Jonas

"Yuan Xulei(袁徐磊)"

unread,
Aug 28, 2013, 6:32:24 AM8/28/13
to Jonas Sicking, Ray You, dev-webapi, James Ho, hongtang, Dave Hylands, Jan Varga
Hi Jonas,

Ray is following the discussions on the W3C lists. For the device
storage integration, we finishes most functions, but still need
some work to use DOMPromise. I'll create a bug for that today.
Patch will be provided for feedback this week.

We don't have time to look at sandboxed filesystem yet.

Yuan
0 new messages