Proposal to change the way /shared works in Gaia

[Antonio Manuel Amaya Calvo]

Hey all.

Even if this is somewhat related to the discussion taking place on
https://groups.google.com/forum/?fromgroups#!topic/mozilla.dev.gaia/Ls9U-CScxWA
I'm opening a new thread so as to not hijack/derail the other thread.

Currently the way it works when you include a resource (be it a CSS, JS,
img or whatever) from /shared in gaia is that, at build time, the
included resources are copied into the zip of every app that uses it.
And this means that resources end on the device as many times as
applications use them. I think that this is so, in part, because the
app:// protocol forbids loading resources from other apps unless you
have the webapps-manage permission.

My proposal is making a small change on the way the app protocol works.
The change would be to include a isLibrary boolean field on the
manifest.webapp, with the following semantic:

if isLibrary is true:
* Homescreen won't show the app
* The app:// protocol handler will allow loading resources (by way
of linking, or XHR, or whatever) from any other origin (could be
restricted to app:// origins)

What would that mean, from a practical point of view:

1 The shared directory would become a library app (say
app://sharedresources.gaiamobile.org).
2 At build time, when a gaia app includes/links to /shared/whatever,
replace it with app://sharedresources.gaiamobile.org/whatever (this is
just so we don't have to actually touch all the current apps to convert
the local resources into global ones, other approach would be to do a
one time sweep of all the apps to replace/correct the links).
3. At run time, when any app accesses
app://sharedresources.gaiamobile.org/whatever the system would just
allow that load.

So no more replicating resources n-times. Oh, and this means that third
party apps will also be able to use resources from /shared without
having to pull them out of gaia and copying them into their own apps.
They will just link to them, as they would on any other place on the web.

This means, obviously, doing some changes at the platform level:

* Allowing the app:// protocol to load resources from any app that
declares itself to be a library (manifest.isLibrary == true).
* Changing the default CSP for certified and privileged apps to allow
executing JSs from self and app://sharedresources.gaiamobile.org.

I could take charge of making the platform changes myself.

So what do you think?

Antonio


________________________________

Este mensaje se dirige exclusivamente a su destinatario. Puede consultar nuestra política de envío y recepción de correo electrónico en el enlace situado más abajo.
This message is intended exclusively for its addressee. We only send and receive email on the basis of the terms set out at:
http://www.tid.es/ES/PAGINAS/disclaimer.aspx

[Matthew R. MacPherson]

As a third-party developer of a packaged app for Gaia, I'm all for this. Having to include all of Gaia slows down my tests on Travis, as I have to pull in two copies of Gaia (one already in B2G binary; one in my app used to have up-to-date versions of Gaia's CSS) -- nixing that alone would be great.

Perhaps at the very least this would be great for _packaged_ apps?

- tofumatt | http://tofumatt.com

> _______________________________________________
> dev-gaia mailing list
> dev-...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-gaia

[Fred Lin]

That way might be simpler for reuse /shared resource, though, that hack will broken the web.
Besides change the app:// protocol, We have to introduce the app/module-dependency manager like npm, to insure /shared app is properly installed before start the dependent webapp.


I think its good to think about how we use /shared resource, and maybe we could do better in build time.
Currently we have to declare used /shared resource in index.html and they'll be scanned and copied into the zip without optimization/minification. Minify JS/Building Blocks will help us cut down some launch latency.


regards
--
Fred Lin

----- Origin -----
Sender: "Antonio Manuel Amaya Calvo" <am...@tid.es>

[Tim Chien]

Agree, this sounds like what Microsoft do for packaged Javascript
Metro apps. For example, this is how to reference a WinJS library in
the HTML:
<script src="//Microsoft.WinJS.0.6/js/base.js"></script>

Don't offer a solution that only works on the Mozilla packaged apps.
Think about what the feature could offer to the real Open Web.

--
Tim Guan-tin Chien, Engineering Manager and Front-end Lead, Firefox
OS, Mozilla Corp. (Taiwan)

[ANTONIO MANUEL AMAYA CALVO]

On 14/06/2013, at 17:10, "Tim Chien" <timd...@mozilla.com> wrote:

> Agree, this sounds like what Microsoft do for packaged Javascript
> Metro apps. For example, this is how to reference a WinJS library in
> the HTML:
> <script src="//Microsoft.WinJS.0.6/js/base.js"></script>
>
> Don't offer a solution that only works on the Mozilla packaged apps.
> Think about what the feature could offer to the real Open Web.

Oh, but I do. Just allowing to load from the shared app library from the web (from http on other words) would give developers three options to use a 'system' CSS or lib (like the IndexedDB wrapper one for example):

1. Copy the files he wants to use on his own server and link just to his server. He controls all, but doesn't seem very webbie to me.
2. Link to an external server where the file reside. This is something he can currently do by linking to the raw github file, for example. But it's strange having to link to an external copy of something that's already on the device.

What I'm proposing, if we allow loading the shared files from http (and to be honest I can't think of a reason not to) doesn't remove those two options. It just add one more that I think is more attractive:

3. Just link (not copy, not include, and I can't think of anything more webby than linking) to the adequate URL. You can even add checks to see if the file does or does not actually exist and fall back to 2 or 1 if you want.

And for packaged apps, well for privileged ones using 2 isn't an option (at least nor for JavaScript). So without this you have to copy the system resources inside your app if you want to reuse code. Again, allowing to link to a shared resource seems more natural to someone that comes from the HTTP web to the APP Web world than just telling him "no, you can't link to this shared resource or library or whatever, if you want to use it just feel free to copy it".

Best,

Antonio

[Dale Harvey]

Linking to external resources is just a plain bad idea though, you shouldnt
do it on the web, douglas crockford put alert()s into json2.js that was
very widely hotlinked just to prove the point

[Kevin Grandon]

I would honestly prefer a submodule/repo approach. I have a POC with calendar app here: https://github.com/KevinGrandon/gaia/compare/master...gaia_submodule

Essentially you add a submodule for each library, and the files are synced when you perform a make. (The only difference in process is that instead of linking or copying a resource, you add the git URL as submodule). The nice thing is that it's a simple change and then both internal and third-party developers could utilize the same workflow.

I think this is nicer because we can actually freeze at a certain commit. Due to our uplift process it makes updating shared code extremely difficult late in the game. We need a process which takes that into consideration.

Last time I brought this up folks seemed to prefer repo, but I don't know too much about repo to do a solid POC right now.

Best,
Kevin

[Ben Kelly]

On Jun 14, 2013, at 11:55 AM, ANTONIO MANUEL AMAYA CALVO <am...@tid.es> wrote:
> What I'm proposing, if we allow loading the shared files from http (and to be honest I can't think of a reason not to) doesn't remove those two options. It just add one more that I think is more attractive:
>
> 3. Just link (not copy, not include, and I can't think of anything more webby than linking) to the adequate URL. You can even add checks to see if the file does or does not actually exist and fall back to 2 or 1 if you want.

Do you have any thoughts on how to handle changes to shared resources on different versions of the OS? In particular, API incompatible changes.

It seems apps would need to link to particular versions of the shared resource. Would this just be encoded in the URL?

Thanks!

Ben

[Andrew Sutherland]

On 06/14/2013 12:06 PM, Kevin Grandon wrote:
> I would honestly prefer a submodule/repo approach. I have a POC with calendar app here: https://github.com/KevinGrandon/gaia/compare/master...gaia_submodule

I'd like to strongly second this request. Either submodules or repo.
While this would be great to avoid regressions from shared resources, it
would make it much easier to reduce the barrier to entry of hacking on
the e-mail app.

Now that 1.1 is done-ish and we have some breathing space, this seems
like a good time to pursue a cleanup like this.

Vivien, Tim, you are the module co-owners of Gaia; is this something you
would approve if we pursued this more aggressively? The gaia
productivity team is meeting early next week in SF to focus on making
the e-mail app something that does not require major activation energy
to hack on; this is something we can mini-sprint on as an important
first step.

Andrew

PS: There was an outstanding question from Tim about the capabilities of
repo; I've addressed that in the origin thread.

[Tim Chien]

This is not entirely related to Antonio proposed, let's try not to
hijack his thread here.

Antonio tried to offer a solution where we will not pack duplicate
files into valuable phone memory; I agree the problem exist but I wish
we could refine the idea more, before we further diverge how packaged
app works v.s. how real web works. The word "library" scared me a lot
too -- if those packages will be called packaged libraries, we would
eventually re-implement |apt-get| to deal the dependency right?

Expect my other response on the submodule v.s. repo thread.

[Antonio Manuel Amaya Calvo]

Hey all.

Answers inline :)

Er, sorry for using the l-word then ;). Not really. Using isLibrary
(maybe "isApp: false" would be more descriptive?) is just a trick to
allow the homescreen to not show apps that are just resource buckets
without having to hardcode on the homescreen app the name of the shared
file.

I still don't know how this is making more divergent the way packaged
apps work versus the rest of the web works. On the rest of the web,
currently, you can link to a shared resource on your same origin/app or
on any other origin/app you can think off. Leaving aside if it's a good
idea or practice, the fact is that you can do a
<script src="http://code.jquery.com/jquery-1.10.1.min.js"></script> on
your real web app and it will work.

On other words, you can reuse other HTTP resources from a HTTP app just
by linking. You cannot do the same, at all, on a packaged app (that is,
you cannot reuse an app:// resource from another app:// origin). What
I'm proposing is just relaxing that restriction a little bit (we cannot
relax it completely for security reasons). So, if anything, this change
takes the packaged apps a step closer to what a real web developer would
expect.

Someone asked before about versions. For Gaia apps this problem doesn't
actually exist, because everything is packed together at build time, so
a Gaia app will work the same if the shared code is shared-by-copying as
it is now, or if it's shared-by-linking as I propose. For the rest of
the apps, well, there are a bunch of possible solutions. For example,
just prefix the shared data URLs with the version:

app://sharedresources.gaiamobile.org/v1/js/async-storage.js

Or creating a different package for each version:

app://sharedresources-v1.gaiamobile.org/js/async-storage.js

Again, both solutions are transparent for Gaia. The advantage for the
second one is that it's easier to selectively add (or not) some old
versions to a build.

In any case, and concerning versions, I think that the best solution
would be just to add a

app://sharedresources.gaiamobile.org/version_info.json

and let the third party apps decide if they can work or not with
whatever version is included with the OS. Note that in most cases if
your app is designed for a more recent version of the shared resources
(when there has been changes in APIs and such) then it will probably not
work on a OS version that has an older sharedresouce, because probably
the needed underlying changes on Gecko won't exist either.

Best,

Antonio

>
> Expect my other response on the submodule v.s. repo thread.
>
> --
> Tim Guan-tin Chien, Engineering Manager and Front-end Lead, Firefox
> OS, Mozilla Corp. (Taiwan)

[Jan Jongboom]

Regardless of the submodule/shared discussion. A field indicating that an app is not an app would be a great addition. For example: applications that only make sense if being called from an activity have no need of being on the homescreen, yet at the moment they'll have to be there. Or an app that backups data, I'd rather have an entry in Settings then an extra icon on the homescreen.

[Salvador de la Puente González]

Hi people

Answering inline:

May you feel more comfortable dealing with a local CDN? A CDN is a very
webby concept.

With a "CDN application" (i.e. a library) you can use this schema:
app://localcdn.gaiamobile.org/category/resource/version/name.js --> for
a specific version
app://localcdn.gaiamobile.org/category/resource/name.js --> for the
latest version

But you already need some kind of trick to mean this is not an
application but shared resource and avoid listing it.

Best!

PS: Please, avoid remember me what N stands for, I know being local it
can not be a Network. I'm trying to deal with the abstraction perceived
by the user, not implementation details.

[Axel Hecht]

On 6/17/13 10:54 AM, Jan Jongboom wrote:
> Regardless of the submodule/shared discussion. A field indicating that an app is not an app would be a great addition. For example: applications that only make sense if being called from an activity have no need of being on the homescreen, yet at the moment they'll have to be there. Or an app that backups data, I'd rather have an entry in Settings then an extra icon on the homescreen.

Any proposal on how the UX for deinstallation should work? I guess the
only way we have right now is through the homescreen customization?

Axel

[Jan Jongboom]

I'd guess either a list of installed applications that are no real applications in Settings or in Marketplace.

[Antonio Manuel Amaya Calvo]

You can uninstall apps from the settings app already. On Settings -> App
Permissions -> (chose any app) there's a Uninstall app button.

On 17/06/2013 12:29, Axel Hecht wrote:
> On 6/17/13 10:54 AM, Jan Jongboom wrote:

>> Regardless of the submodule/shared discussion. A field indicating
>> that an app is not an app would be a great addition. For example:
>> applications that only make sense if being called from an activity
>> have no need of being on the homescreen, yet at the moment they'll
>> have to be there. Or an app that backups data, I'd rather have an
>> entry in Settings then an extra icon on the homescreen.

> Any proposal on how the UX for deinstallation should work? I guess the
> only way we have right now is through the homescreen customization?
>
> Axel

[JOSE MANUEL CANTERA FONSECA]

El 14/06/13 17:06, "Tim Chien" <timd...@mozilla.com> escribió:

>Agree, this sounds like what Microsoft do for packaged Javascript
>Metro apps. For example, this is how to reference a WinJS library in
>the HTML:
><script src="//Microsoft.WinJS.0.6/js/base.js"></script>
>
>Don't offer a solution that only works on the Mozilla packaged apps.
>Think about what the feature could offer to the real Open Web.

We need to think about the possibilities of standardization of getting
access to local resources present on the target device. For doing that we
might need to define new formalisms to declare access to such resources.
For instance

<script src="localUri('async_storage.js')"></script>

In the case of JS we could even think in terms of modules "ala node.js" .

<link rel="local stylesheet" href="localUri('buttons.css')">

In the case of CSS we could also think about building blocks and not the
particular CSS files.

All of them combined with @media directives can create a powerful and
standardizable path for this problem. But modifying the app protocol or
creating the concept of 'special lib app' is something I believe is not
going to fly.

>> ________________________________
>>
>> Este mensaje se dirige exclusivamente a su destinatario. Puede
>>consultar nuestra política de envío y recepción de correo electrónico en
>>el enlace situado más abajo.
>> This message is intended exclusively for its addressee. We only send
>>and receive email on the basis of the terms set out at:
>> http://www.tid.es/ES/PAGINAS/disclaimer.aspx
>> _______________________________________________
>> dev-gaia mailing list
>> dev-...@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-gaia
>> _______________________________________________
>> dev-gaia mailing list
>> dev-...@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-gaia
>
>
>

>--
>Tim Guan-tin Chien, Engineering Manager and Front-end Lead, Firefox
>OS, Mozilla Corp. (Taiwan)

[ANTONIO MANUEL AMAYA CALVO]

On 17/06/2013, at 16:35, "JOSE MANUEL CANTERA FONSECA" <jm...@tid.es> wrote:

> El 14/06/13 17:06, "Tim Chien" <timd...@mozilla.com> escribió:
>
>> Agree, this sounds like what Microsoft do for packaged Javascript
>> Metro apps. For example, this is how to reference a WinJS library in
>> the HTML:
>> <script src="//Microsoft.WinJS.0.6/js/base.js"></script>
>>
>> Don't offer a solution that only works on the Mozilla packaged apps.
>> Think about what the feature could offer to the real Open Web.
>
> We need to think about the possibilities of standardization of getting
> access to local resources present on the target device. For doing that we
> might need to define new formalisms to declare access to such resources.
> For instance
>
> <script src="localUri('async_storage.js')"></script>

Why do we need to invent a new syntax to do something that's basically accessing a resource? I thought that's what URIs were for.

>
> In the case of JS we could even think in terms of modules "ala node.js" .
>
> <link rel="local stylesheet" href="localUri('buttons.css')">
>
> In the case of CSS we could also think about building blocks and not the
> particular CSS files.
>
> All of them combined with @media directives can create a powerful and
> standardizable path for this problem. But modifying the app protocol or
> creating the concept of 'special lib app' is something I believe is not
> going to fly.

We're already using the app protocol to access resources from other apps. That's what the homescreen does to put the homescreen icons up. The access that homescreen has (through the webapps-manage permission) is a global, origin controlled access. That is, a given app origin (the homescreen in this case) has access to all other app origins on the device. The change I was proposing was just adding a other option, a global, destiny controlled access. That is, allowing an app to say "it's ok to load resources from me". Which would allow, as a first step, putting all shared resources on a single place and, well, share them.

Best regards,

Antonio