Gallery API

[Ben Francis]

Hi there,

In the last WebAPI meeting we discussed options for the storage of media
like photographs and reservations were made about implementing the W3C's File
<goog_1445904292>API: Directories and
System<http://www.w3.org/TR/file-system-api/>[1]
specification.

Instead it was suggested that maybe a multimedia storage API could be
developed, specifically for storing images, audio and video.

Having looked into this a little further I came across a W3C editors draft
called The Gallery API <http://dev.w3.org/2009/dap/gallery/>[2]. I was
wondering whether anyone is familiar with this work?

Just as a reminder, the use cases which triggered this discussion are:

1. Locally storing photographs in such a way that they can be accessed
from multiple web apps
2. Mounting the stored photographs as a file system, where the user
agent device (e.g. a smartphone) acts as a USB mass storage device plugged
into a USB host (e.g. a desktop PC), or a memory card is taken from the
device and placed in a memory card reader to be plugged into a host device
(e.g. a desktop PC)
3. Accessing media stored on external storage (e.g. where the user agent
device is a desktop PC and a USB flash drive is plugged in)

Would a "Gallery API" specifically for certain types of data be better or
worse than a general purpose file API along the lines of the "File API:
Directories and System" spec?

Alternatively, should some other method (such as
PTP<http://en.wikipedia.org/wiki/Picture_Transfer_Protocol>[3],
PTP/IP or something else more web-like) be used for transferring media
resources between devices?

Ben

1. File API: Directories and System http://www.w3.org/TR/file-system-api/
2. The Gallery API http://dev.w3.org/2009/dap/gallery/
3. PTP http://en.wikipedia.org/wiki/Picture_Transfer_Protocol

--
Ben Francis
http://tola.me.uk

[Anssi Kostiainen]

Hi Ben,

On 24.11.2011, at 20.51, ext Ben Francis wrote:

> Having looked into this a little further I came across a W3C editors draft
> called The Gallery API <http://dev.w3.org/2009/dap/gallery/>[2]. I was
> wondering whether anyone is familiar with this work?

The W3C DAP working group is shelving the Gallery API. For more details, see:

http://lists.w3.org/Archives/Public/public-device-apis/2011Nov/0035.html

If you (plan to) implement something similar I'd encourage you to drop an email to the public-device-apis mailing list:

http://lists.w3.org/Archives/Public/public-device-apis/

-Anssi

[JOSE MANUEL CANTERA FONSECA]

Hi Ben,

In WAC 2.0 we defined a spec for getting access to the Filesystem [1],
supporting the concept of 'virtual roots'. Actually we supported the
following:


* images: the images location
* videos: the videos location
* music: the sounds location
* documents: the documents location
* downloads: the location for downloaded items
* wgt-package: the widget package location (read only location)
* wgt-private: the widget private storage
* wgt-private-tmp: the widget private volatile storage area
* removable: the removable storage area location.



I think for B2G we can follow a similar approach, allowing to get access
to various virtual roots and from there we can have a simple API to get
access to the content and to create new content there. For doing so, We
can base this 'B2G Gallery API' on FileReader and FileWriter and with the
minimal additions to navigate through folders. So for the Gallery API my
proposal would be:

* Forget about the W3C DAP work and Directories and Filesystem
* Reuse the concept of 'virtual roots' from WAC 2.0
* Create a B2G Gallery API using the WAC 2.0 Filesystem API as a basis,
but, making it simple and based on FileReader and FileWriter interfaces.

That's my view on this particular problem. I'm happy to contribute to the
design of the API in accordance with the principles above.

Many thanks

Best

[1] http://specs.wacapps.net/2.0/jun2011/deviceapis/filesystem.html




El 24/11/11 19:51, "Ben Francis" <b...@krellian.com> escribió:

>Hi there,
>
>In the last WebAPI meeting we discussed options for the storage of media
>like photographs and reservations were made about implementing the W3C's
>File
><goog_1445904292>API: Directories and
>System<http://www.w3.org/TR/file-system-api/>[1]
>specification.
>
>Instead it was suggested that maybe a multimedia storage API could be
>developed, specifically for storing images, audio and video.
>

>Having looked into this a little further I came across a W3C editors draft
>called The Gallery API <http://dev.w3.org/2009/dap/gallery/>[2]. I was
>wondering whether anyone is familiar with this work?
>

>_______________________________________________
>dev-webapi mailing list
>dev-w...@lists.mozilla.org
>https://lists.mozilla.org/listinfo/dev-webapi


Este mensaje se dirige exclusivamente a su destinatario. Puede consultar nuestra política de envío y recepción de correo electrónico en el enlace situado más abajo.
This message is intended exclusively for its addressee. We only send and receive email on the basis of the terms set out at.
http://www.tid.es/ES/PAGINAS/disclaimer.aspx

[Jonas Sicking]

On Thu, Nov 24, 2011 at 10:51 AM, Ben Francis <b...@krellian.com> wrote:
> Hi there,
>
> In the last WebAPI meeting we discussed options for the storage of media
> like photographs and reservations were made about implementing the W3C's File
> <goog_1445904292>API: Directories and
> System<http://www.w3.org/TR/file-system-api/>[1]
> specification.
>
> Instead it was suggested that maybe a multimedia storage API could be
> developed, specifically for storing images, audio and video.
>
> Having looked into this a little further I came across a W3C editors draft
> called The Gallery API <http://dev.w3.org/2009/dap/gallery/>[2]. I was
> wondering whether anyone is familiar with this work?
>
> Just as a reminder, the use cases which triggered this discussion are:
>
>   1. Locally storing photographs in such a way that they can be accessed
>   from multiple web apps
>   2. Mounting the stored photographs as a file system, where the user
>   agent device (e.g. a smartphone) acts as a USB mass storage device plugged
>   into a USB host (e.g. a desktop PC), or a memory card is taken from the
>   device and placed in a memory card reader to be plugged into a host device
>   (e.g. a desktop PC)
>   3. Accessing media stored on external storage (e.g. where the user agent
>   device is a desktop PC and a USB flash drive is plugged in)

Sweet! This matches exactly the use-cases I have had in mind (not
entirely surprisingly since much of my thinking was set off by you
bringing it up on the above mentioned call :) ).

On the API-side, I think we can cover all above use cases using two APIs:

1. An API for reading and writing media files to a central storage
area on the device.
For image/video files this is pretty easy. The set of files store is
basically just a straight list of File objects. To save a file the
page would provide a Blob object, we'd slap a unique name and the
current time on it and store it. (Note that the current File
implementation in Gecko doesn't allow accessing the lastModifiedDate,
but that's easily fixed). When reading, the page can use asynchronous
APIs to retrieve a list of File objects.

Remaining metadata, such as EXIF, video length, etc, can be retrieved
from the Blob itself.

For things like music libraries we *might* be able to get away with
something even as simple as this. Most music files comes with a fair
amount of meta-data, such as artist, song name, album name, album
cover-picture, etc. Technically this data can usually be read from the
music file itself, but I'm not sure if that will produce good enough
performance. So maybe we'll need something more advanced for the music
library. In any case I suspect we'll want to keep the music library
separate from for example media files that the user creates since they
are synched differently (you don't want to move your music files to
your desktop computer, you want to copy it there).

2. An API for retrieving a list of File objects that exist on an
external device, such as a memory card, or USB mass storage.
This API can be used to read data from a phone which is connected
through USB, as well as from memory cards which have been moved from a
camera and inserted into the device (the latter is especially common
for laptops or desktops).
Additionally this API likely needs a way to delete selected files from
the memory card/USB storage.
Each File object will contain their full path, so if the pathname
happens to be meaningfull the webapp can get to it, but for the common
case when it's not, the webapp doesn't need to care about it.


Implementation-wise, the first API would likely write to some
dedicated directory structure which follows the DCIM [1] standard.
We'd then expose this directory whenever the device was plugged in
through USB. This way a web-app running on the desktop computer can
use the second API to read in that data.


Some of the advantages with this approach is:
* We get a central storage on the B2G device for pictures. So if you
have multiple camera apps installed (the default camera app +
hipstamatic + instagram) they all store their pictures in the same
location and so will all be synced whenever you sync your device.
* The API for reading files from memory cards will be super easy to
use and very easy to implement. From the web apps point of view it's
simply a list of all the files they are interested in. No need to
write recursive directory traversing functions, and no need to worry
about where the files are stored on the card. In most cases the
directory and file names are meaningless anyway.
* Since each media file is given a unique name, apps that insert files
can keep managing just the files it has created if it so desires.
* We technically could implement the first API described above using
IndexedDB if that makes for better performance for things like
enumerating. But that's something we can worry about later once we
have time.

[1] http://en.wikipedia.org/wiki/Design_rule_for_Camera_File_system

/ Jonas

[Robert Kaiser]

Jonas Sicking schrieb:

> The set of files store is
> basically just a straight list of File objects. To save a file the
> page would provide a Blob object, we'd slap a unique name and the
> current time on it and store it.

Note that some slice of people actually want files with specific names
in a "folder"-style hierarchy.
I often get really frustrated with mobile photo viewers that present the
images I move on their storage (for showing them to others anywhere I
go) in one single date-sorted stream, as it would be much easier to find
a specific photo or collection if my folders would be represented there.
There's usually two kinds of people you encounter with all those
store-and-view mechansims: ones who prefer flat storages and find
hierarchies complicated and ones who get really upset about anything
where they can't use an orderly hierarchy. I think we need to have the
API possibilities for both - whatever we implement in our
default-provided viewer app(s).

Robert Kaiser

[Jonas Sicking]

On Sun, Jan 8, 2012 at 8:54 AM, Robert Kaiser <ka...@kairo.at> wrote:
> Jonas Sicking schrieb:

>
>> The set of files store is
>> basically just a straight list of File objects. To save a file the
>> page would provide a Blob object, we'd slap a unique name and the
>> current time on it and store it.
>
>

> Note that some slice of people actually want files with specific names in a
> "folder"-style hierarchy.
> I often get really frustrated with mobile photo viewers that present the
> images I move on their storage (for showing them to others anywhere I go) in
> one single date-sorted stream, as it would be much easier to find a specific
> photo or collection if my folders would be represented there.
> There's usually two kinds of people you encounter with all those
> store-and-view mechansims: ones who prefer flat storages and find
> hierarchies complicated and ones who get really upset about anything where
> they can't use an orderly hierarchy. I think we need to have the API
> possibilities for both - whatever we implement in our default-provided
> viewer app(s).

Since we expose the path of each file it's easy to build a UI which
displays a full tree view, or lets you choose a given folder to view
or anything of the like.

/ Jonas

[Robert Kaiser]

Jonas Sicking schrieb:

> Since we expose the path of each file it's easy to build a UI which
> displays a full tree view, or lets you choose a given folder to view
> or anything of the like.

OK, if this is regular files that have a path, that sounds good. I
wasn't sure if the API just has some kind of objects that represent the
files in some way or actual files with their usual properties. If the
latter, that should be fine for designing apps either way, which is good.

Robert Kaiser

[Chris Jones]

----- Original Message -----
> From: "Jonas Sicking" <jo...@sicking.cc>
> To: "Ben Francis" <b...@krellian.com>
> Cc: dev-w...@lists.mozilla.org
> Sent: Saturday, January 7, 2012 8:48:02 PM
> Subject: Re: Gallery API
>
> 1. An API for reading and writing media files to a central storage
> area on the device.

I assume this would be the case, but reading files stored by a different domain/origin would require a permission bit, right?

> 2. An API for retrieving a list of File objects that exist on an
> external device, such as a memory card, or USB mass storage.

What's the benefit of a dedicated interface for this, vs. general filesystem access? (I also assume this would be a privileged API?)

This proposal sounds like a good evolutionary path because it seems simple implementation-wise. But, if we had full intents and WebUSB implementations, would we still want these dedicated APIs? See [1] for some discussion. One can argue that this approach is paving a cowpath that's already known to exist, and we can later implement the dedicated interfaces using lower-level mechanisms.

Also, a problem with an intents+WebUSB implementation is that the user still has to explicitly select the media files from each app to be shared with the USB broker. With the APIs here, there's no user interaction needed, which is good. Another approach for the intents+WebUSB solution is to spec out a privileged "silent intents" model, with no user interaction needed (like android's). This would need to be a separate spec since the current intents security model is built around the user interaction.

Cheers,
Chris

[Chris Jones]

----- Original Message -----
> From: "Chris Jones" <cjo...@mozilla.com>
> To: "Jonas Sicking" <jo...@sicking.cc>
> Cc: dev-w...@lists.mozilla.org, "Ben Francis" <b...@krellian.com>
> Sent: Monday, January 9, 2012 12:56:18 PM
> Subject: Re: Gallery API
>
> ----- Original Message -----
> > From: "Jonas Sicking" <jo...@sicking.cc>
> > To: "Ben Francis" <b...@krellian.com>
> > Cc: dev-w...@lists.mozilla.org
> > Sent: Saturday, January 7, 2012 8:48:02 PM
> > Subject: Re: Gallery API
> >
> > 1. An API for reading and writing media files to a central storage
> > area on the device.
>
> I assume this would be the case, but reading files stored by a
> different domain/origin would require a permission bit, right?
>

Another issue: if I load media files into this storage area from my host computer, which domains are allowed to access them?

Maybe this entire API should require a permission bit.

Cheers,
Chris

[Jonas Sicking]

On Mon, Jan 9, 2012 at 12:56 PM, Chris Jones <cjo...@mozilla.com> wrote:
>
> ----- Original Message -----
> > From: "Jonas Sicking" <jo...@sicking.cc>
> > To: "Ben Francis" <b...@krellian.com>
> > Cc: dev-w...@lists.mozilla.org
> > Sent: Saturday, January 7, 2012 8:48:02 PM
> > Subject: Re: Gallery API
> >
> > 1. An API for reading and writing media files to a central storage
> > area on the device.
>
> I assume this would be the case, but reading files stored by a different domain/origin would require a permission bit, right?

Definitely. We'll likely have two levels of access here, read access
and read/write access.

> > 2. An API for retrieving a list of File objects that exist on an
> > external device, such as a memory card, or USB mass storage.
>
> What's the benefit of a dedicated interface for this, vs. general filesystem access?  (I also assume this would be a privileged API?)
>
> This proposal sounds like a good evolutionary path because it seems simple implementation-wise.  But, if we had full intents and WebUSB implementations, would we still want these dedicated APIs?  See [1] for some discussion.  One can argue that this approach is paving a cowpath that's already known to exist, and we can later implement the dedicated interfaces using lower-level mechanisms.

I don't quite see how intents are playing in here (in general I think
people are using intents to mean many different things).

The main advantage I see here is actually ease of use for web
developers. With something like the FileSystem API you need to
traverse the directory tree yourself, which is a recursive
asynchronous task. Most likely you'll end up with lots of nested
asynchronous callbacks.

If we just hand them a list of files (which has the full path
available as a property) all you need to do is to write a simple
for-loop to look for the files you are interested in and render
previews of them in the page.

> Also, a problem with an intents+WebUSB implementation is that the user still has to explicitly select the media files from each app to be shared with the USB broker.  With the APIs here, there's no user interaction needed, which is good.  Another approach for the intents+WebUSB solution is to spec out a privileged "silent intents" model, with no user interaction needed (like android's).  This would need to be a separate spec since the current intents security model is built around the user interaction.

I'm not following you here.

/ Jonas

[Jonas Sicking]

On Mon, Jan 9, 2012 at 1:56 PM, Chris Jones <cjo...@mozilla.com> wrote:
> ----- Original Message -----

>> From: "Chris Jones" <cjo...@mozilla.com>
>> To: "Jonas Sicking" <jo...@sicking.cc>
>> Cc: dev-w...@lists.mozilla.org, "Ben Francis" <b...@krellian.com>

>> Sent: Monday, January 9, 2012 12:56:18 PM
>> Subject: Re: Gallery API
>>
>> ----- Original Message -----
>> > From: "Jonas Sicking" <jo...@sicking.cc>
>> > To: "Ben Francis" <b...@krellian.com>
>> > Cc: dev-w...@lists.mozilla.org
>> > Sent: Saturday, January 7, 2012 8:48:02 PM
>> > Subject: Re: Gallery API
>> >
>> > 1. An API for reading and writing media files to a central storage
>> > area on the device.
>>
>> I assume this would be the case, but reading files stored by a
>> different domain/origin would require a permission bit, right?
>>
>

> Another issue: if I load media files into this storage area from my host computer, which domains are allowed to access them?
>
> Maybe this entire API should require a permission bit.

Yes. I think it should be like the Contacts API. I.e. the device acts
as a generic "contacts store" without caring where any individual
contact comes from. Any app (who has access) can enumerate all
contacts and interact with just the ones the app is interested in.

Similarly, the media stored on the device is just a generic media
store. It doesn't matter if you take a picture using the built-in
camera app, a downloaded instagram app, or draw it using a
FingerPaint-for-HTML5 app. All pictures end up in the same store. When
you go to your facebook-app and choose to upload a picture, the
facebook app will see all store pictures together.

This doesn't mean that FingerPaint's load/save UI has to render all
pictures stored on the phone. It can easily remember which pictures it
has stored in the media store and only render those pictures.

/ Jonas

[Chris Jones]

----- Original Message -----
> From: "Jonas Sicking" <jo...@sicking.cc>
> To: "Chris Jones" <cjo...@mozilla.com>
> Cc: dev-w...@lists.mozilla.org, "Ben Francis" <b...@krellian.com>
> Sent: Monday, January 9, 2012 4:44:40 PM
> Subject: Re: Gallery API
>

> On Mon, Jan 9, 2012 at 12:56 PM, Chris Jones <cjo...@mozilla.com>

> wrote:
> >
> > ----- Original Message -----
> > > From: "Jonas Sicking" <jo...@sicking.cc>
> > > To: "Ben Francis" <b...@krellian.com>
> > > Cc: dev-w...@lists.mozilla.org
> > > Sent: Saturday, January 7, 2012 8:48:02 PM
> > > Subject: Re: Gallery API
> > >

> > This proposal sounds like a good evolutionary path because it seems
> > simple implementation-wise.  But, if we had full intents and
> > WebUSB implementations, would we still want these dedicated APIs?
> >  See [1] for some discussion.  One can argue that this approach is
> > paving a cowpath that's already known to exist, and we can later
> > implement the dedicated interfaces using lower-level mechanisms.
>
> I don't quite see how intents are playing in here (in general I think
> people are using intents to mean many different things).
>

With the intents proposal, each app would store its own images in IDB or something like that. By definition, there's no issue with cross-origin access because each app owns its own datastore. When my host computer, say, wanted to download all my pictures, the app implementing MTP, say, over WebUSB (let's call it the "USB broker") would broadcast an intent like "GET image/*". Each app that had its own image store would handle that intent and send back a list of the images in its store. Then the USB broker would send these images back to host, as the host requests them.

This is a more general mechanism because both IDB and intents will already exist.

> The main advantage I see here is actually ease of use for web
> developers. With something like the FileSystem API you need to
> traverse the directory tree yourself, which is a recursive
> asynchronous task. Most likely you'll end up with lots of nested
> asynchronous callbacks.
>

I thought we weren't implementing FileSystem? But yes, file systems as currently implemented aren't a great fit for The Web, I agree.

> > Also, a problem with an intents+WebUSB implementation is that the
> > user still has to explicitly select the media files from each app
> > to be shared with the USB broker.  With the APIs here, there's no
> > user interaction needed, which is good.  Another approach for the
> > intents+WebUSB solution is to spec out a privileged "silent
> > intents" model, with no user interaction needed (like android's).
> >  This would need to be a separate spec since the current intents
> > security model is built around the user interaction.
>
> I'm not following you here.
>

In the example above, the "GET image/*" broadcast would need to be mediated by browser UI. If my flickr app, say, sent a list of files in response to the GET intent, then the user would have to explicitly choose which ones the USB broker app got access to. (In the current intents model.) That's a good thing in general, but it makes writing e.g. an auto-sync app more difficult, since it can't operate silently and in the background.

One way to address that is something like this privileged media storage API. Another way is with a privileged "silent intents" API, that doesn't require mediation through UI. We'll want something like that for Gecko<-->privileged app communication anyway, I suspect. Like asking a window manager to launch a new application.

Cheers,
Chris