"Device storage" --- storing devices?

[Chris Jones]

At some point this API had a name change. Sorry to post naming nits to a public newsgroup, but I really dislike "device storage". One hand it sounds like "storing devices". On another hand, all files are stored on a device so "device storage" is meaningless.

AIUI originally the point of the API was to allow web pages to store media files into well-known locations, be they particular "devices" like SD cards or special folders like ~/Pictures.

Did the naming discussion already take place somewhere that I missed? If not, can someone argue for "DeviceStorage"? r- from me ;). MediaStorage isn't perfect either but I prefer it.

Evolving this to storage of arbitrary files, in some sort of "file system" we might call it, sounds like scope creep from what was originally discussed.

Cheers,
Chris

[Rick Waldron]

I doubt anyone would say that "Web Storage" sounds like "storing webs";
storage on a device vs. storage on the web.


Rick

> _______________________________________________
> dev-webapi mailing list
> dev-w...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-webapi
>

[Thinker Li]

On Mar 15, 9:32 am, Chris Jones <cjo...@mozilla.com> wrote:
>
> Evolving this to storage of arbitrary files, in some sort of "file system" we might call it, sounds like scope creep from what was originally discussed.
>

There is diverse reasons for saving files locally and sharing with
other applications. For example, people save PDF/epub files attached
to an email locally. Is there any reason that *Storage serves only
media files?

[Jonas Sicking]

On Wed, Mar 14, 2012 at 6:32 PM, Chris Jones <cjo...@mozilla.com> wrote:
> At some point this API had a name change.  Sorry to post naming nits to a public newsgroup, but I really dislike "device storage".  One hand it sounds like "storing devices".  On another hand, all files are stored on a device so "device storage" is meaningless.
>
> AIUI originally the point of the API was to allow web pages to store media files into well-known locations, be they particular "devices" like SD cards or special folders like ~/Pictures.
>
> Did the naming discussion already take place somewhere that I missed?  If not, can someone argue for "DeviceStorage"?  r- from me ;).  MediaStorage isn't perfect either but I prefer it.
>

> Evolving this to storage of arbitrary files, in some sort of "file system" we might call it, sounds like scope creep from what was originally discussed.

I generally don't get that worked up over naming things, so if there's
consensus for calling this MediaStorage then I'm ok with that.

However I don't think that we should leave out storing things like
KeyNote presentations, Word documents or todo-lists, just because we
can't think of good name for the API. So whatever name we come up
with, I think the API should allow storing those types of files.

/ Jonas

[Chris Jones]

----- Original Message -----
> From: "Jonas Sicking" <jo...@sicking.cc>
> To: "Chris Jones" <cjo...@mozilla.com>
> Cc: "dev-webapi" <dev-w...@lists.mozilla.org>
> Sent: Friday, March 16, 2012 6:57:47 PM
> Subject: Re: "Device storage" --- storing devices?
>
> I generally don't get that worked up over naming things, so if
> there's
> consensus for calling this MediaStorage then I'm ok with that.
>

I don't either, generally. But some names are important when they need to quickly convey a specific meaning. Spec names fall into this category. I don't think DeviceStorage succinctly conveys that anymore. (A name like "Farglegarble" might be a good project name, but it's a bad spec name.)

----- Original Message -----
> From: "Thinker Li" <think...@gmail.com>
> To: mozilla-d...@lists.mozilla.org
> Sent: Thursday, March 15, 2012 10:50:23 PM
> Subject: Re: "Device storage" --- storing devices?
>
> On Mar 15, 9:32 am, Chris Jones <cjo...@mozilla.com> wrote:
> >

> > Evolving this to storage of arbitrary files, in some sort of "file
> > system" we might call it, sounds like scope creep from what was
> > originally discussed.
> >
>

> There is diverse reasons for saving files locally and sharing with
> other applications. For example, people save PDF/epub files attached
> to an email locally. Is there any reason that *Storage serves only
> media files?

> However I don't think that we should leave out storing things like
> KeyNote presentations, Word documents or todo-lists, just because we
> can't think of good name for the API. So whatever name we come up
> with, I think the API should allow storing those types of files.
>

My concern is the difference between what's being proposed here and a generic FileSystem API. I think the differences are
- for storing a new file, requested name isn't guaranteed to be honored
- there's no concept of a directory, hence no hierarchy
- files of certain MIME types are handled specially (or, MIME types are the directory)

The last point seems to motivate the other two, so maybe TypedStorage is appropriate? Or MIMEStorage?

Cheers,
Chris

[Ben Francis]

On Sat, Mar 17, 2012 at 8:08 AM, Chris Jones <cjo...@mozilla.com> wrote:

> My concern is the difference between what's being proposed here and a
> generic FileSystem API.

I'm interested in this too. I wouldn't want to add even more confusion to
the complicated landscape of offline storage technologies so the name
should convey the intended use. Unfortunately I don't think we're yet clear
on the intended use, i.e. whether it's specifically for images, audio and
video or for any type of file.

I think the differences are
> - for storing a new file, requested name isn't guaranteed to be honored
>

I don't particularly like this feature of the proposal, it seems to me we
should either have names or not, not "maybe" give a file a name.

> - there's no concept of a directory, hence no hierarchy
>

I do like this part of the proposal and I think this is its main
distinguishing feature from the alternative File API proposals. No need to
inflict the complexities of hierarchical file systems on the web.

> - files of certain MIME types are handled specially (or, MIME types are
> the directory)
>

We have discussed this as an option, but as I understand it this isn't part
of the current proposal. Types are just arbitrary strings, not MIME types.

I would call it FileStorage or MediaStorage depending on the scope we agree
on.

Or, as a more off-the-wall idea "TaggedStorage" and allow multiple names
per file (tagging is better than directories)! (Sorry, I don't mean to
de-rail this proposal, but it's clear that the scope isn't clear to
everyone).

Ben

--
Ben Francis
http://tola.me.uk

[Jonas Sicking]

On Sat, Mar 17, 2012 at 1:08 AM, Chris Jones <cjo...@mozilla.com> wrote:
> ----- Original Message -----
>> From: "Thinker Li" <think...@gmail.com>
>> To: mozilla-d...@lists.mozilla.org
>> Sent: Thursday, March 15, 2012 10:50:23 PM
>> Subject: Re: "Device storage" --- storing devices?
>>
>> On Mar 15, 9:32 am, Chris Jones <cjo...@mozilla.com> wrote:
>> >
>> > Evolving this to storage of arbitrary files, in some sort of "file
>> > system" we might call it, sounds like scope creep from what was
>> > originally discussed.
>> >
>>
>> There is diverse reasons  for saving files locally and sharing with
>> other applications.  For example, people save PDF/epub files attached
>> to an email locally.  Is there any reason that *Storage serves only
>> media files?
>
>> However I don't think that we should leave out storing things like
>> KeyNote presentations, Word documents or todo-lists, just because we
>> can't think of good name for the API. So whatever name we come up
>> with, I think the API should allow storing those types of files.
>>
>

> My concern is the difference between what's being proposed here and a generic FileSystem API.  I think the differences are

>  - for storing a new file, requested name isn't guaranteed to be honored

Indeed. For storing files on a B2G device, I'm not sure I see a need
for non-automatically-generated filenames. For storing on an external
USB key, filenames seems more useful.

>  - there's no concept of a directory, hence no hierarchy

That's not entirely true. You can still enumerate a directory using
the enumerate function.

It's also missing functions to deal with "directory objects". I.e. you
can't get a list of *just* the directories in a given subdirectory,
and there is no object which represents a directory. Nor can you do a
non-deep directory traversal.

But I'm not sure that there are terribly strong use cases for
organizing user data with such a directory focus.

>  - files of certain MIME types are handled specially (or, MIME types are the directory)

How so? That's not currently enforced by the API, simply encouraged by
it. But if you look at how user-data is handled by modern OSs they are
moving in the same direction by creating folders for common file
types.

And the intent is that it won't be focused on MIME types as they map
pretty poorly to what we humans think of as types, i.e. there are 9
different MIME types for Word documents, with new ones added as they
revise the format. And there is no MIME type for "fonts" or
"documents" and only sort-of a MIME type for "images".

> The last point seems to motivate the other two, so maybe TypedStorage is appropriate?  Or MIMEStorage?

I'd be ok with TypedStorage. Though I think it misses the main feature
of the API which is that it's storage area which is shared by all
applications. I.e. it's not application-specific like IndexedDB and
localStorage is.

/ Jonas

[Jonas Sicking]

On Mon, Mar 19, 2012 at 9:00 AM, Ben Francis <b...@krellian.com> wrote:

> On Sat, Mar 17, 2012 at 8:08 AM, Chris Jones <cjo...@mozilla.com> wrote:
> I think the differences are
>>  - for storing a new file, requested name isn't guaranteed to be honored
>

> I don't particularly like this feature of the proposal, it seems to me we
> should either have names or not, not "maybe" give a file a name.

Yeah, I'm not a big fan of it either. The problem that I'm struggling
with is that on some platforms, like Desktop, we are essentially
forced to deal with filenames, whereas on platforms like B2G I was
hoping to avoid them (I've never dealt with a mobile app which forces
users to deal with filenames).

But maybe that is overly ambitious. Maybe we should just expose "real"
filenames on all platforms, but make the filename argument optional in
which case the implementation generates a filename.

> Or, as a more off-the-wall idea "TaggedStorage" and allow multiple names
> per file (tagging is better than directories)! (Sorry, I don't mean to
> de-rail this proposal, but it's clear that the scope isn't clear to
> everyone).

How do we map this to a desktop filesystem where we really are writing
files directly to the filesystem. And where we want to support reading
existing files which are identifiable mostly by filename.

Likewise, I was hoping to use the same API for accessing USB-keys as
to reduce storage-API-proliferation.

/ Jonas

[Sinker]

On Tuesday, March 20, 2012 6:32:19 AM UTC+8, Jonas Sicking wrote:
> > Or, as a more off-the-wall idea "TaggedStorage" and allow multiple names
> > per file (tagging is better than directories)! (Sorry, I don't mean to
> > de-rail this proposal, but it's clear that the scope isn't clear to
> > everyone).
>
> How do we map this to a desktop filesystem where we really are writing
> files directly to the filesystem. And where we want to support reading
> existing files which are identifiable mostly by filename.

We should forget hierarchical filesystems here. This API is a place to distribute files among applications in a device. But, it is not the only source and destination of files. Applications should use other filesystem APIs to access filesystems, but they use this API for sharing files with other applications. We need this API because others don't help applications to share and enumerate files. But, it is not necessary to manage all files in this way.

I suggest to import files from USB if user really want to use files from USB keys in this way. We can also provides a view for files with well-known types from USB keys to help users or applications. But, it is the responsibility of applications to recognize unknown files.

[Jonas Sicking]

On Tue, Mar 20, 2012 at 7:35 AM, Sinker <think...@gmail.com> wrote:
> On Tuesday, March 20, 2012 6:32:19 AM UTC+8, Jonas Sicking wrote:

>> > Or, as a more off-the-wall idea "TaggedStorage" and allow multiple names
>> > per file (tagging is better than directories)! (Sorry, I don't mean to
>> > de-rail this proposal, but it's clear that the scope isn't clear to
>> > everyone).
>>
>> How do we map this to a desktop filesystem where we really are writing
>> files directly to the filesystem. And where we want to support reading
>> existing files which are identifiable mostly by filename.
>

> We should forget hierarchical filesystems here.  This API is a place to distribute files among applications in a device.  But, it is not the only source and destination of files.  Applications should use other filesystem APIs to access filesystems, but they use this API for sharing files with other applications.  We need this API because others don't help applications to share and enumerate files.  But, it is not necessary to manage all files in this way.
>
> I suggest to import files from USB if user really want to use files from USB keys in this way.  We can also provides a view for files with well-known types from USB keys to help users or applications.  But, it is the responsibility of applications to recognize unknown files.

My hope was actually that we would *only* need this API for files
stored "centrally" on a device and thus shared between applications.
I.e. I was hoping that we wouldn't need a filesystem API exposed to
webapps/webpages.

What problems are you hoping to solve by exposing a filesystem API
which isn't solved by the current proposal? I.e. what applications do
you think people couldn't build with the current proposal, but could
build if we had a full filesystem API?

As for files from USB devices, I think we can add an API which allows
an app to enumerate plugged in USB-keys and memory cards, and get a
DeviceStorage object representing a USB-key or memory card. That way
the app can both read and write data to such locations.

/ Jonas

[Sinker]

On Wednesday, March 21, 2012 2:59:11 PM UTC+8, Jonas Sicking wrote:
> On Tue, Mar 20, 2012 at 7:35 AM, Sinker <think...@gmail.com> wrote:
> > On Tuesday, March 20, 2012 6:32:19 AM UTC+8, Jonas Sicking wrote:
> >> > Or, as a more off-the-wall idea "TaggedStorage" and allow multiple names
> >> > per file (tagging is better than directories)! (Sorry, I don't mean to
> >> > de-rail this proposal, but it's clear that the scope isn't clear to
> >> > everyone).
> >>
> >> How do we map this to a desktop filesystem where we really are writing
> >> files directly to the filesystem. And where we want to support reading
> >> existing files which are identifiable mostly by filename.
> >
> > We should forget hierarchical filesystems here.  This API is a place to distribute files among applications in a device.  But, it is not the only source and destination of files.  Applications should use other filesystem APIs to access filesystems, but they use this API for sharing files with other applications.  We need this API because others don't help applications to share and enumerate files.  But, it is not necessary to manage all files in this way.
> >
> > I suggest to import files from USB if user really want to use files from USB keys in this way.  We can also provides a view for files with well-known types from USB keys to help users or applications.  But, it is the responsibility of applications to recognize unknown files.
>
> My hope was actually that we would *only* need this API for files
> stored "centrally" on a device and thus shared between applications.
> I.e. I was hoping that we wouldn't need a filesystem API exposed to
> webapps/webpages.
>
> What problems are you hoping to solve by exposing a filesystem API
> which isn't solved by the current proposal? I.e. what applications do
> you think people couldn't build with the current proposal, but could
> build if we had a full filesystem API?

I don't think we need full filesystem API for most applications, because I think files should be imported into DeviceStorage or alike for a mobile device. But, we still need to import/export files from/to filesystems.

>
> As for files from USB devices, I think we can add an API which allows
> an app to enumerate plugged in USB-keys and memory cards, and get a
> DeviceStorage object representing a USB-key or memory card. That way
> the app can both read and write data to such locations.
>

As USB-keys going bigger and bigger, I don't think we can implement DeviceStorage that is efficient enough. Traditional filesystems are not designed for this purpose. They are optimized for traveling path but not for enumerating files of specific type. If an app try to find out all pdf files, we should scan all directories in an USB-key to find pdf files that may not exist. How long do you takes to run 'find . -name XXXX' in a mozilla-central repository on your desktop? (SSD maybe better)

[Jonas Sicking]

On Wed, Mar 21, 2012 at 8:57 AM, Sinker <think...@gmail.com> wrote:
> On Wednesday, March 21, 2012 2:59:11 PM UTC+8, Jonas Sicking wrote:

>> On Tue, Mar 20, 2012 at 7:35 AM, Sinker <think...@gmail.com> wrote:
>> > On Tuesday, March 20, 2012 6:32:19 AM UTC+8, Jonas Sicking wrote:
>> >> > Or, as a more off-the-wall idea "TaggedStorage" and allow multiple names
>> >> > per file (tagging is better than directories)! (Sorry, I don't mean to
>> >> > de-rail this proposal, but it's clear that the scope isn't clear to
>> >> > everyone).
>> >>
>> >> How do we map this to a desktop filesystem where we really are writing
>> >> files directly to the filesystem. And where we want to support reading
>> >> existing files which are identifiable mostly by filename.
>> >
>> > We should forget hierarchical filesystems here.  This API is a place to distribute files among applications in a device.  But, it is not the only source and destination of files.  Applications should use other filesystem APIs to access filesystems, but they use this API for sharing files with other applications.  We need this API because others don't help applications to share and enumerate files.  But, it is not necessary to manage all files in this way.
>> >
>> > I suggest to import files from USB if user really want to use files from USB keys in this way.  We can also provides a view for files with well-known types from USB keys to help users or applications.  But, it is the responsibility of applications to recognize unknown files.
>>
>> My hope was actually that we would *only* need this API for files
>> stored "centrally" on a device and thus shared between applications.
>> I.e. I was hoping that we wouldn't need a filesystem API exposed to
>> webapps/webpages.
>>
>> What problems are you hoping to solve by exposing a filesystem API
>> which isn't solved by the current proposal? I.e. what applications do
>> you think people couldn't build with the current proposal, but could
>> build if we had a full filesystem API?
>

> I don't think we need full filesystem API for most applications, because I think files should be imported into DeviceStorage or alike for a mobile device.  But, we still need to import/export files from/to filesystems.
>
>>

>> As for files from USB devices, I think we can add an API which allows
>> an app to enumerate plugged in USB-keys and memory cards, and get a
>> DeviceStorage object representing a USB-key or memory card. That way
>> the app can both read and write data to such locations.
>

> As USB-keys going bigger and bigger, I don't think we can implement DeviceStorage that is efficient enough.  Traditional filesystems are not designed for this purpose.  They are optimized for traveling path but not for enumerating files of specific type.  If an app try to find out all pdf files, we should scan all directories in an USB-key to find pdf files that may not exist.  How long do you takes to run 'find . -name XXXX' in a mozilla-central repository on your desktop? (SSD maybe better)

The DeviceStorage draft doesn't requiring enumerating files of a
specific type, so I'm not sure I'm following you here.

/ Jonas

[Sinker]

On Mar 22, 3:46 am, Jonas Sicking <jo...@sicking.cc> wrote:

> On Wed, Mar 21, 2012 at 8:57 AM, Sinker <thinker...@gmail.com> wrote:
> >> As for files from USB devices, I think we can add an API which allows
> >> an app to enumerate plugged in USB-keys and memory cards, and get a
> >> DeviceStorage object representing a USB-key or memory card. That way
> >> the app can both read and write data to such locations.
>
> > As USB-keys going bigger and bigger, I don't think we can implement DeviceStorage that is efficient enough.  Traditional filesystems are not designed for this purpose.  They are optimized for traveling path but not for enumerating files of specific type.  If an app try to find out all pdf files, we should scan all directories in an USB-key to find pdf files that may not exist.  How long do you takes to run 'find . -name XXXX' in a mozilla-central repository on your desktop? (SSD maybe better)
>
> The DeviceStorage draft doesn't requiring enumerating files of a
> specific type, so I'm not sure I'm following you here.

API does not, but Apps need to. It does not make better, it worse.
How about an PDF reader try to figure out all pdf files on a device?
Even API is not responsible for that, we should consider such very
common scenario.

[Jonas Sicking]

On Wed, Mar 21, 2012 at 6:52 PM, Sinker <think...@gmail.com> wrote:
> On Mar 22, 3:46 am, Jonas Sicking <jo...@sicking.cc> wrote:

>> On Wed, Mar 21, 2012 at 8:57 AM, Sinker <thinker...@gmail.com> wrote:
>> >> As for files from USB devices, I think we can add an API which allows
>> >> an app to enumerate plugged in USB-keys and memory cards, and get a
>> >> DeviceStorage object representing a USB-key or memory card. That way
>> >> the app can both read and write data to such locations.
>>
>> > As USB-keys going bigger and bigger, I don't think we can implement DeviceStorage that is efficient enough.  Traditional filesystems are not designed for this purpose.  They are optimized for traveling path but not for enumerating files of specific type.  If an app try to find out all pdf files, we should scan all directories in an USB-key to find pdf files that may not exist.  How long do you takes to run 'find . -name XXXX' in a mozilla-central repository on your desktop? (SSD maybe better)
>>
>> The DeviceStorage draft doesn't requiring enumerating files of a
>> specific type, so I'm not sure I'm following you here.
>

> API does not, but Apps need to.   It does not make better, it worse.
> How about an PDF reader try to figure out all pdf files on a device?

> Even API is not responsible for that, we should consider such very
> common scenario.

Hmm.. I guess I don't fully understand your concern then.

Are you saying that you think that the DeviceStorage API is slower
than a FileSystemAPI when performing operations like "find all .pdf
files"? Or are you saying that both the DeviceStorageAPI and a
FileSystemAPI are too slow at performing such an operation and that we
need to create something faster than both of them?

/ Jonas

[Jason Miller]

I would imagine he's saying that it is misleading to expose a simple API
for file search when it has extremely poor performance because it's
implemented on top of a filesystem that doesn't journal.
It seems like a bad idea to expose a simple API that does such a costly
operation, as it would very likely be abused (simply by being used).
Implementing search in a way that doesn't thrash the filesystem would
require some sort of journaling, even if it was a simple index built when
the storage is mounted, coupled with index updates triggered by any write
operations.

- Jason

[Sinker]

Yes, this what I was talking. In the real world, most USB-keys are in
FAT or NTFS. It is inefficient to enumerate files of specific type in
USB-keys or filesystems without some kind of index. And, it is not
easy to maintain an index for USB-keys.

[Jonas Sicking]

On Thu, Mar 22, 2012 at 8:15 PM, Jason Miller <ja...@developit.ca> wrote:
> I would imagine he's saying that it is misleading to expose a simple API for
> file search when it has extremely poor performance because it's implemented
> on top of a filesystem that doesn't journal.

The API doesn't contain any functionality for file search. Only for
file enumeration.

> It seems like a bad idea to expose a simple API that does such a costly
> operation, as it would very likely be abused (simply by being used).
> Implementing search in a way that doesn't thrash the filesystem would
> require some sort of journaling, even if it was a simple index built when
> the storage is mounted, coupled with index updates triggered by any write
> operations.

The current API doesn't intend to expose search, just like most basic
file systems (and the google FileSystemAPI) doesn't expose search. To
do search in most file systems you enumerate all files and then either
look at the file name, or at the file contents, or a combination of
the two, to determine which files you are interested in.

That's how you would use the proposed API too.

/ Jonas

[Jonas Sicking]

On Thu, Mar 22, 2012 at 9:14 PM, Sinker <think...@gmail.com> wrote:
> On Mar 23, 11:15 am, Jason Miller <ja...@developit.ca> wrote:

> Yes, this what I was talking.  In the real world, most USB-keys are in
> FAT or NTFS.  It is inefficient to enumerate files of specific type in
> USB-keys or filesystems without some kind of index. And, it is not
> easy to maintain an index for USB-keys.

Again, there seems to be some misconception that the current API
exposes functionality to enumerate files of a specific type. That is
not the case.

The intended use is to enumerate all files and then just act on the
ones that you are interested in.

It would be great if someone could let me know how I can improve the
description of the API to make this more clear.

/ Jonas

[Sinker]

On Saturday, March 24, 2012 8:13:57 AM UTC+8, Jonas Sicking wrote:
>
> The current API doesn't intend to expose search, just like most basic
> file systems (and the google FileSystemAPI) doesn't expose search. To
> do search in most file systems you enumerate all files and then either
> look at the file name, or at the file contents, or a combination of
> the two, to determine which files you are interested in.
>
> That's how you would use the proposed API too.

It is curious. What is argument "type" passed to getDeviceStorage() for? Is not it supposed to return an instance that can be used to enumerate all files of given type? For example, "image" or "image/png" ?

[Sinker]

On Saturday, March 24, 2012 8:13:57 AM UTC+8, Jonas Sicking wrote:
>
> The current API doesn't intend to expose search, just like most basic
> file systems (and the google FileSystemAPI) doesn't expose search. To
> do search in most file systems you enumerate all files and then either
> look at the file name, or at the file contents, or a combination of
> the two, to determine which files you are interested in.
>
> That's how you would use the proposed API too.

[Jonas Sicking]

On Sat, Mar 24, 2012 at 7:38 PM, Sinker <think...@gmail.com> wrote:

> On Saturday, March 24, 2012 8:13:57 AM UTC+8, Jonas Sicking wrote:
>>
>> The current API doesn't intend to expose search, just like most basic
>> file systems (and the google FileSystemAPI) doesn't expose search. To
>> do search in most file systems you enumerate all files and then either
>> look at the file name, or at the file contents, or a combination of
>> the two, to determine which files you are interested in.
>>
>> That's how you would use the proposed API too.

> It is curious.  What is argument "type" passed to getDeviceStorage() for?  Is not it supposed to return an instance that can be used to enumerate all files of given type? For example, "image" or "image/png" ?

The "type" argument is basically just a root directory name. Think of
it like this for b2g:

We would basically have a directory like
~/userFiles/

when someone calls getDeviceStorage("pictures") and creates a file
named "foo.jpg" in that storage area we would create a file with name
~/userFiles/pictures/foo.jpg

If someone calls getDeviceStorage("documents") and creates a file
named "hello.doc" in that storage area we would create a file with
name
~/userFiles/documents/hello.doc

If someone calls getDeviceStorage("mayaFiles") and creates a file
named "spaceship.mp" in that storage area we would create a file with
name
~/userFiles/mayaFiles/spaceship.mp

However on desktop firefox on OSX we might map the "pictures" type not
to "~/userFiles/pictures", but instead to "~/Pictures" since that is
where many OSX programs save their picture files.

I tried to clarify this in the spec draft, please feel free to add
further information that you think could help others.

/ Jonas

[Jason Miller]

Ah, I was wondering if this was the case. So the "type" is not a mimetype,
but a storage namespace/grouping.
That makes more sense (I am not thinking about utility here, only
implementation) when it's getting built on top of a hierarchical filesystem.

I'm assuming the type string would be subject to permissions - wouldn't
want unfettered access to my pictures because a third-party dev knew to
declare their storage instance with that particular type-string.

- Jason


Jason Miller
519.872.0797 // developIT <http://developit.ca/> // Jason Miller
Design<http://jasonmillerdesign.com/>
*Developer of amoebaOS <https://amoebaos.com/>,
Shutterborg<http://shutterb.org/> &
more

*

[Jonas Sicking]

On Sun, Mar 25, 2012 at 11:59 AM, Jason Miller <ja...@developit.ca> wrote:

> Ah, I was wondering if this was the case. So the "type" is not a mimetype,
> but a storage namespace/grouping.
> That makes more sense (I am not thinking about utility here, only
> implementation) when it's getting built on top of a hierarchical filesystem.
>
> I'm assuming the type string would be subject to permissions - wouldn't
> want unfettered access to my pictures because a third-party dev knew to
> declare their storage instance with that particular type-string.
>

Are there other types of data than pictures that you are more ok with
sharing? I think I would be equally concerned about all the shared data
stored on my device.

The idea is definitely to have a permissions system which guards the files
from being accessed without user consent. Please read the security section
of the wiki page.

/ Jonas

[Chris Jones]

----- Original Message -----
> From: "Jonas Sicking" <jo...@sicking.cc>
> To: "Chris Jones" <cjo...@mozilla.com>

> Cc: "Thinker Li" <think...@gmail.com>, "Jonas Sicking" <sic...@mozilla.com>, mozilla-d...@lists.mozilla.org
> Sent: Monday, March 19, 2012 2:59:23 PM
> Subject: Re: "Device storage" --- storing devices?
>

> >  - files of certain MIME types are handled specially (or, MIME
> >  types are the directory)
>
> How so? That's not currently enforced by the API, simply encouraged
> by
> it. But if you look at how user-data is handled by modern OSs they
> are
> moving in the same direction by creating folders for common file
> types.
>
> And the intent is that it won't be focused on MIME types as they map
> pretty poorly to what we humans think of as types, i.e. there are 9
> different MIME types for Word documents, with new ones added as they
> revise the format. And there is no MIME type for "fonts" or
> "documents" and only sort-of a MIME type for "images".
>

Reading this and the later comments, I think we're again drifting away from the original motivation for this API, and I don't entirely follow why.

We had originally discussed letting apps store pictures, and b2g being able to automatically store them in DCF format so that existing tools can recognize them when a b2g phone is plugged into a host computer. This new proposal restricts us to exposing files to hosts through MTP, which may be OK (it's vastly superior), but that's an economic issue we'll need to investigate.

More concerningly, it breaks the *host* uploading files to the *device*. If I upload music files to my b2g phone, how is the music player supposed to find them? I have no idea what directory the app is looking for. In addition, that means if I upload a bunch of music files, but then switch to a different music-player app, it might no longer be looking for the same magic string that the old one was, and it won't be able to play my music.

So, color me concerned about this API.

Cheers,
Chris

[Jonas Sicking]

On Thu, Mar 29, 2012 at 9:27 PM, Chris Jones <cjo...@mozilla.com> wrote:
> ----- Original Message -----
>> From: "Jonas Sicking" <jo...@sicking.cc>
>> To: "Chris Jones" <cjo...@mozilla.com>
>> Cc: "Thinker Li" <think...@gmail.com>, "Jonas Sicking" <sic...@mozilla.com>, mozilla-d...@lists.mozilla.org
>> Sent: Monday, March 19, 2012 2:59:23 PM
>> Subject: Re: "Device storage" --- storing devices?
>>
>> >  - files of certain MIME types are handled specially (or, MIME
>> >  types are the directory)
>>
>> How so? That's not currently enforced by the API, simply encouraged
>> by
>> it. But if you look at how user-data is handled by modern OSs they
>> are
>> moving in the same direction by creating folders for common file
>> types.
>>
>> And the intent is that it won't be focused on MIME types as they map
>> pretty poorly to what we humans think of as types, i.e. there are 9
>> different MIME types for Word documents, with new ones added as they
>> revise the format. And there is no MIME type for "fonts" or
>> "documents" and only sort-of a MIME type for "images".
>>
>
> Reading this and the later comments, I think we're again drifting away from the original motivation for this API, and I don't entirely follow why.
>
> We had originally discussed letting apps store pictures, and b2g being able to automatically store them in DCF format so that existing tools can recognize them when a b2g phone is plugged into a host computer.  This new proposal restricts us to exposing files to hosts through MTP, which may be OK (it's vastly superior), but that's an economic issue we'll need to investigate.

Why are we restricted to MPT? Sorry, I don't know enough about MPT and
DCF so I need might need pointers.

> More concerningly, it breaks the *host* uploading files to the *device*.  If I upload music files to my b2g phone, how is the music player supposed to find them?  I have no idea what directory the app is looking for.  In addition, that means if I upload a bunch of music files, but then switch to a different music-player app, it might no longer be looking for the same magic string that the old one was, and it won't be able to play my music.

This is indeed not handled, and has never been handled by the API.
It's exactly the same issue as if application A stores pictures in the
DeviceStorage, then application B might need to update its data
structures.

In general, if applications can, they should simply enumerate all
files. That's what for example my photo-library app does on my phone
I'm fairly sure.

However this won't work for something like a music player app which
needs to read out meta-data from all stored music files and
index/store this data in a app-specific IndexedDB database.

In order to solve this use case I think we need to have some
notification which applications can register for which lets them know
about any additions/removals/modifications done to any files in a
given subdirectory (i.e. in a given "type" as passed to
getDeviceStorage). This might be a similar notification mechanism as
we'd use to deliver notifications to for example the phone application
about incoming phone calls if the phone application isn't currently
running.

An alternative solution would be to allow applications to get a list
of modifications done since last startup, but I'm not sure if that's
implementable on desktop Firefox.

If you have other proposals for how to solve this don't hesitate to speak up.

/ Jonas

[Chris Jones]

DCF is a magical file-system layout that allows hosts to recognize image files by the format. If you've ever looked at the raw mounted USB storage on your camera/phones, that's the DCIM/[blah]/[blerg] stuff. Trying to emulate this somehow would be extremely hard. Not in scope for initial b2g.

MTP is a more general protocol that works on the "object" level, allowing the host to query for objects of various types instead of forcing it to mount the raw fs and poke around for magical directories. With MTP, we could share image files, say, stored in arbitrary locations by indexing them and using the index to answer queries. (/waving hands a bit.)

> > More concerningly, it breaks the *host* uploading files to the
> > *device*.  If I upload music files to my b2g phone, how is the
> > music player supposed to find them?  I have no idea what directory
> > the app is looking for.  In addition, that means if I upload a
> > bunch of music files, but then switch to a different music-player
> > app, it might no longer be looking for the same magic string that
> > the old one was, and it won't be able to play my music.
>
> This is indeed not handled, and has never been handled by the API.
> It's exactly the same issue as if application A stores pictures in
> the
> DeviceStorage, then application B might need to update its data
> structures.
>

That's not what we initially discussed, is my point :). At least as I understood it.

I don't understand what problem you're pointing at here.

> In general, if applications can, they should simply enumerate all
> files. That's what for example my photo-library app does on my phone
> I'm fairly sure.
>

This paragraph literally made my jaw drop, so I think I'm misunderstanding what you're proposing. I'll try to catch you on IRC tomorrow.

> However this won't work for something like a music player app which
> needs to read out meta-data from all stored music files and
> index/store this data in a app-specific IndexedDB database.
>

Exposing metadata would be cool for a v2 API, IMHO.

> In order to solve this use case I think we need to have some
> notification which applications can register for which lets them know
> about any additions/removals/modifications done to any files in a
> given subdirectory (i.e. in a given "type" as passed to
> getDeviceStorage). This might be a similar notification mechanism as
> we'd use to deliver notifications to for example the phone
> application
> about incoming phone calls if the phone application isn't currently
> running.
>

I thought we'd already discussed wanting to do this. But my memory may be fuzzy.

At any rate, I agree.

> An alternative solution would be to allow applications to get a list
> of modifications done since last startup, but I'm not sure if that's
> implementable on desktop Firefox.
>

It is, although not efficiently in general.

I don't feel like your reply answered my questions, though. What it sounds like is being proposed is a file system API, but with the directory hierarchy limited to one level, and the directory hierarchy not enumerable. I don't think that addresses the initial use cases.

Cheers,
Chris

[Doug Turner]

> I don't feel like your reply answered my questions, though. What it sounds like is being proposed is a file system API, but with the directory hierarchy limited to one level, and the directory hierarchy not enumerable. I don't think that addresses the initial use cases.

I tend to agree. We want to be able to fully support directory listings. The simplest use-case for any file system API is the ability to generate and navigate a hierarchical file system. The current spec'ed api requires that you enumerable every file on the fs to determine what is directly under '/'.


Assuming that the webapp had permissions, I want this to enumerate the files and directories in /home/

var storage = navigator.getDeviceStorage("/");
var cursor = storage.enumerate("home");


I also want well-defined names to work:

var storage = navigator.getDeviceStorage("pictures");
var cursor = storage.enumerate();


If you really want a deep enumeration, you can do this in your web app. If lots of app developer scream for this feature, we can add a flag to the enumerate() method.


Jonas -
I understand it is weird to return a DOMFile which is a directory. But, maybe we should use the type to specify that this is a special file?

[Doug Turner]

> I don't feel like your reply answered my questions, though. What it sounds like is being proposed is a file system API, but with the directory hierarchy limited to one level, and the directory hierarchy not enumerable. I don't think that addresses the initial use cases.

[Chris Jones]

-1, fully generic file system API is moving in the wrong direction.

Cheers,
Chris

----- Original Message -----
> From: "Doug Turner" <doug....@gmail.com>
> To: "mozilla dev webapi" <mozilla.d...@googlegroups.com>
> Cc: "Jonas Sicking" <sic...@mozilla.com>, mozilla-d...@lists.mozilla.org, "Jonas Sicking" <jo...@sicking.cc>,
> "Thinker Li" <think...@gmail.com>
> Sent: Saturday, March 31, 2012 1:09:08 AM
> Subject: Re: "Device storage" --- storing devices?
>
>

> _______________________________________________
> dev-webapi mailing list
> dev-w...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-webapi
>

[Doug Turner]

On Sunday, April 1, 2012 10:56:47 AM UTC-7, Chris Jones wrote:
> -1, fully generic file system API is moving in the wrong direction.

You mentioned earlier in the thread that you scoped out something already - something that you think is in the right direction. Do you have a link?

[Doug Turner]

On Sunday, April 1, 2012 10:56:47 AM UTC-7, Chris Jones wrote:

> -1, fully generic file system API is moving in the wrong direction.