Re: Aligning on an App Model for the future

[Peter Dolanjski]

As an update, I've created a wiki for the summarized app types here:
https://wiki.mozilla.org/Gaia/App_Definitions#App_Types_Proposal

Please let me know if you disagree with anything as this will be used as the basis for the app types that we will support going forward.

Thanks,

Peter Dolanjski

Product Manager, Firefox OS

Mozilla

On Thu, Apr 16, 2015 at 3:27 PM, Peter Dolanjski <pdola...@mozilla.com> wrote:

Hello All,

There have been a number of discussion threads around the details pertaining to the various forms of apps that are needed as we move forward with new architecture (based on permissions, presence of a manifest, etc.).  A few of us (product, engineering, partner engineering, Marketplace) got together to try to assemble a simple description of what each distinct category is for and when they would be used.  I plan to publish this on one of the v3 wikis, but before I do I just wanted to open it up for comments/suggestions.

You can see the breakdown here: v3 App Model

Feel free to comment right in the doc.

In addition, I'll summarize it here in case you prefer some email dialogue on it:

Web Site
Description:

Vanilla web site with no app manifest.

Distribution Channel(s):

1. Any web server
2. Indexed by Marketplace (hosted elsewhere)

Why would a Developer choose this option?

- Just a web site that should be used inside the browser. (no point in enumerating all reasons for building a web site)

Why would a Developer *not* choose this option?

- Want to provide a more "native" experience
- Need access to sensitive APIs

Signing Required?

No

Mozilla Review?

Yes, for Marketplace indexed content only

Impact of Changes to Existing Apps

none

Proposed User Experience on v3

Discovered through browsing (any web server), searching or via Marketplace with “open” button. Can optionally pin a page (not “site”) to the homescreen which will open in a browser window.

Web App
Description:

Standards-based web app with W3C web app manifest.

Distribution Channel(s):

1. Any web server
2. Indexed by Marketplace (hosted elsewhere)
3. Hosted by Marketplace (TBD)

Why would a Developer choose this option?

- Want content to run standalone outside the browser on multiple platforms.

- No need to package assets

- If no sensitive APIs required, no extra signing process

Why would a Developer *not* choose this option?

- Developers coming from native consider hosting a service provided by the Marketplace

Signing Required?

No

Mozilla Review?

Yes, for Marketplace indexed/hosted content only

Impact of Changes to Existing Apps

Existing hosted apps should use the new manifest format going forward, but new FxOS versions should support legacy app versions.

Proposed User Experience on v3

Discovered through browsing (any web server), searching or via Marketplace with “open” button. Can use instantly in the browser and optionally pin to the homescreen to use as a standalone app (in an app window with fullscreen, standalone or minimal-ui display mode). Details TBD

Firefox App
Description:

Signed web app (when privileged API access needed), at least initially Firefox* specific. W3C web app manifest with moz-prefixed extensions, inside a hosted streamable package.

Distribution Channel(s):

1. Any web server
2. Indexed by Marketplace (hosted elsewhere)
3. Hosted by Marketplace

Why would a Developer choose this option?

- Easier to handle offline support until ServiceWorkers lands and gains support

- Need access to Firefox* specific privileged APIs

Why would a Developer *not* choose this option?

- Requires signing by Marketplace (for privileged apps)

Signing Required?

Yes (for privileged apps, TBD)

Mozilla Review?

Yes, for Marketplace indexed/hosted content only

Impact of Changes to Existing Apps

Existing packaged apps need to be converted to new format.  Developers need to adhere to new format going forward.

Proposed User Experience on v3

Discovered through browsing, searching or via Marketplace via “open” button. Can use instantly in the browser and optionally pin to the homescreen to use as a standalone app (in an app window with fullscreen, standalone or minimal-ui display mode). Some permissions may require app installation (TBD).

Legacy (Open Web App)
Description:

“web”, “privileged” and “certified” apps with an Open Web App Manifest. Packaged in a signed zip file (except “web” type).

Distribution Channel(s):

1. Marketplace
2. Any web server (“web” type only)

Why would a Developer choose this option?

- Supporting an old app

Why would a Developer *not* choose this option?

- Requires review by Marketplace

Signing Required?

Yes (except web type)

Mozilla Review?

Yes, for Marketplace indexed/hosted content only

Impact of Changes to Existing Apps

none

Proposed User Experience on v3

Hosted apps should work the same as web apps in v3 adhering to the display mode. Packaged apps likely won’t be supported but we’ll need to automatically convert them to work with the new UX.

Thanks,

Peter Dolanjski

Product Manager, Firefox OS

Mozilla

[Anders Rundgren]

On Friday, April 24, 2015 at 5:49:25 PM UTC+2, Peter Dolanjski wrote:
> As an update, I've created a wiki for the summarized app types here:
> https://wiki.mozilla.org/Gaia/App_Definitions#App_Types_Proposal
>
> Please let me know if you disagree with anything as this will be used as the basis for the app types that we will support going forward.

I really like this summary!

For those who like me do a clear distinction between the Open Web and other forms of more or less proprietary web-technology, it would be nice with a row containing standards status which is not limited to the existence of a W3C draft but to actual interoperable competing implementations.

Although Mozilla want to avoid the extension issue, the reality is that Chrome's (inferior but anyway "better-than-nothing") extension scheme is gaining traction:
http://www.cnet.com/news/google-paves-over-hole-left-by-chrome-plug-in-ban/
http://blog.chromium.org/2013/10/connecting-chrome-apps-and-extensions.html
There are TONS of folks out there who want a from the web-side seen, portable Web app extension method! Yes, this would effectively be another Web app in the table.

Anders

[Natalia Martinez-Winter]

thanks Peter, that's very clear.
what about apps created through webmaker ?
where would they fit ?

On 24/04/15 17:49, Peter Dolanjski wrote:

As an update, I've created a wiki for the summarized app types here:
https://wiki.mozilla.org/Gaia/App_Definitions#App_Types_Proposal

Please let me know if you disagree with anything as this will be used as the basis for the app types that we will support going forward.

Thanks,

_______________________________________________
dev-b2g mailing list
dev...@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-b2g

--

Natalia Martinez-Winter Sr Marketing Manager Firefox OS nat...@mozilla.com +33 6 888 11 959 natalia...@skype.com twitter : @martinezwinter

[David Ascher]

Webmaker will create apps under the Web App or Web Site categories.  Nothing special needed.

[Jonas Sicking]

Hi Peter,

I think we can and should simplify this actually.

The main thing that I think we should do is abandon the term "app".
The term "app" is often associated with things like "installing" and
"app store", both of which I'd like to move away from.

To be clear, I think we should still have our marketplace. But not as
the sole provider of "install buttons" as it is today. But rather as a
directory of content works really well on FirefoxOS, similar to the
dmoz.org of days past.

But I think we generally should just consider web content as web
content. I.e. as normal websites that you can browse to. Having a W3C
or a FirefoxOS manifest shouldn't really make a big difference in what
that content can do, what permission it gets or what UI it has.

I think this should be the basis for our design. That content is just
content. Especially from a user perspective having just a single type
of content is the simplest thing.

There are a couple of complexities though.

The first one is what we currently call "privileged apps". I.e. pages
that get access to additional APIs, like TCPSocket and DeviceStorage.

>From a user perspective I think we should still simply treat this as
"web content". I.e. it's something that you can browse to in the
browser and without requiring any installation.

However from a security point of view we'll still need this content to
go through a marketplace review and get signed.

I think we should refer to this content as "signed content". I.e. web
pages that have been authorized by the marketplace team to get access
to additional APIs, and which is signed by the marketplace. More
details about the current proposal for how to deliver that content and
that signature is available in [1], though there are a couple of
alternative proposals that we're still investigating.

But from a user point of view, "signed content" still acts and behaves
just like normal web content. So mainly "signed content" is an
implementation complexity which hopefully won't affect the user or UX.

But I do think that this signing mechanism is something we need to
call out in your document because it does affect web developers.
Specifically it affects developers which need access to "sensitive
APIs" as described by [1].

The second complexity is "pinning". I think we should allow users to
pin websites to the homescreen. This should be possible for *any*
website, whether it has a W3C manifest, a FirefoxOS manifest, the
Apple-invented <meta> tags, or none of the above.

When the user pins a website I think we should grant it some
additional permissions automatically. Mainly we should grant it the
ability to store more data, run in the background using the
BackgroundSync and Alarm APIs and maybe create notifications. It might
also give that content special UX treatments, such as the ability to
show up in webactivity picker, or to run without a URL bar.

But all those capabilities should come with getting pinned. It should
not be related to if the website has any types of manifests or not.

Additionally, ideally the capabilities, like storage, BackgroundSync
API, Alarm API and notification API, is something that we make
available to all websites. However if the user hasn't pinned the
website the user would get prompted. So really the pinning is mainly
about UX. I.e. the prompting UX disappear, and the website can choose
to hide URL bar or appear in the webactivity picker.

Given that web developers care a lot about UX, I do think that the
pinning aspect is also worth mentioning in your document.

[1] https://wiki.mozilla.org/FirefoxOS/New_security_model

/ Jonas


On Thu, Apr 16, 2015 at 12:27 PM, Peter Dolanjski

[Eric Shepherd]

Jonas Sicking

May 4, 2015 at 5:06 PM

_______________________________________________
dev-b2g mailing list
dev...@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-b2g

Peter Dolanjski

April 16, 2015 at 3:27 PM

---

I agree with everything Jonas said there. Let's eliminate the fictional differentiation between web sites and web apps. Not only will this simplify terminology, but it will simplify explanations of how to do things at a fundamental level. MDN is currently increasingly peppered with "You can use this in your Web site or app to..."; eventually it gets cumbersome.

But obviously the best part is the simplification of getting basic privileges to more content. The idea of doing this based on pinning is clever; people pin stuff they use a lot, which implies a degree of trust.

I also agree that there are several APIs that need to be opened up to Web content that don't really need to be as locked away as they currently are.

I'd love to see more conversation going on this.

--

Eric Shepherd
Senior Technical Writer
Mozilla
Blog: http://www.bitstampede.com/
Twitter: http://twitter.com/sheppy

[Peter Dolanjski]

Thanks Jonas/Eric.

Excellent points.  Let me engage some folks on our UX team to get their input and we'll summarize that on this thread.

A big thing that we need to grapple with are user expectations and perceptions with respect to apps.  Obviously we want to advance the technology and change the current perception that the Web is incapable of some of the experiences that apps provide, but we also need to deal with the reality in the market that no "apps" = a product death sentence. 

We witnessed that when a shop keeper in New Delhi sells a device, the sales pitch is roughly "this device has all the apps you need so you can't go wrong buying this Android smartphone" vs. "this device doesn't have any apps, so you definitely don't want this unless you're just making phone calls". 

My point is, there is an experience and positioning line we'll need to straddle to get this right.

Peter

[Paul Theriault]

While I’m all for simplification, installation is a useful metaphor for signalling user intent. Removing installation leaves us limited to prompts only. Prompts are pretty limited in terms of a security mitigation and we will have a hard time authorising access to the privileged APIs, especially (but not only) those which are currently granted without user consent (implicit permissions). And we are not talking at all here about extensions/add-ons which afaik is a key part of ignite. I think there is a still case for install, especially for “add-on” content. 

So I imagine something more like this:

Content (linkable, no install required)

Web Content

Web Apps

Signed Web Apps (for APIs with prompt only)

Add-ons (not linkable? installed via install step)

Signed Web Apps (apps that need access to more sensitive or implicit APIs, or things we generate with customiser etc)

- Paul

[Augustin Trancart]

On 04/05/2015 23:06, Jonas Sicking wrote:
> The second complexity is "pinning". I think we should allow users to
> pin websites to the homescreen. This should be possible for *any*
> website, whether it has a W3C manifest, a FirefoxOS manifest, the
> Apple-invented <meta> tags, or none of the above.
>
> When the user pins a website I think we should grant it some
> additional permissions automatically.

So we will replacing the word "installing" with the word "pinning"?
In this case, the challenge would be to make sure the user is aware of
the consequences of pinning on permissions. Indeed, a user might want to
be more restrictive than the Firefox Marketplace, and might not want to
give additional permissions even to signed content. In this case, he/she
should at least know what pinning grants.
IMO, the advantage of the word "installing" is that it makes clearer
that you give some level of trust to the
application/website/webapp/whatever you're installing. But I agree it is
quite a strong word for just bookmarking stuff in the homescreeen.
Maybe neither of these 2 words is good and we need to find something else...

[Jonas Sicking]

On Tue, May 5, 2015 at 7:14 PM, Paul Theriault <pther...@mozilla.com> wrote:
> While I'm all for simplification, installation is a useful metaphor for
> signalling user intent. Removing installation leaves us limited to prompts
> only. Prompts are pretty limited in terms of a security mitigation and we
> will have a hard time authorising access to the privileged APIs, especially
> (but not only) those which are currently granted without user consent
> (implicit permissions). And we are not talking at all here about
> extensions/add-ons which afaik is a key part of ignite. I think there is a
> still case for install, especially for "add-on" content.
>
> So I imagine something more like this:
>
> Content (linkable, no install required)
> Web Content
> Web Apps
> Signed Web Apps (for APIs with prompt only)
>
> Add-ons (not linkable? installed via install step)
> Signed Web Apps (apps that need access to more sensitive or implicit APIs,
> or things we generate with customiser etc)

Yeah. Sorry. I simplified a bit too much.

I do indeed think that we still need installations for things that
change behavior of other content (be that gaia content or content from
other websites). Navigating to a new keyboard makes as little sense as
navigating to a greasemonkey script or navigating to an ad blocker
addon.

So I do indeed think that we need installations for addons. And that
we should use addons for things like new keyboards and new
homescreens. We should also enable creating addons which implement
security policies, privacy policies network policies or even SMS/phone
policies.

I'll write more about addons separately.

/ Jonas

[Benjamin Francis]

On 4 May 2015 at 22:06, Jonas Sicking <jo...@sicking.cc> wrote:

The main thing that I think we should do is abandon the term "app".
The term "app" is often associated with things like "installing" and
"app store", both of which I'd like to move away from.

...

But I think we generally should just consider web content as web
content. I.e. as normal websites that you can browse to. Having a W3C
or a FirefoxOS manifest shouldn't really make a big difference in what
that content can do, what permission it gets or what UI it has.

 ...

The second complexity is "pinning". I think we should allow users to
pin websites to the homescreen. This should be possible for *any*
website, whether it has a W3C manifest, a FirefoxOS manifest, the
Apple-invented <meta> tags, or none of the above.

We're now working on implementing this as part of Pinning the Web [1]. We're removing the distinction between web apps and web sites, between app windows and browser windows, and making it so that any web site can be pinned. We don't call it an "app" or a "site", we just refer to it by its name (e.g. "Pin Facebook"). We intend to put this through user testing.

When the user pins a website I think we should grant it some

additional permissions automatically. Mainly we should grant it the
ability to store more data, run in the background using the
BackgroundSync and Alarm APIs and maybe create notifications. 

It might
also give that content special UX treatments, such as the ability to
show up in webactivity picker, or to run without a URL bar.

But all those capabilities should come with getting pinned. It should
not be related to if the website has any types of manifests or not.

...

Additionally, ideally the capabilities, like storage, BackgroundSync
API, Alarm API and notification API, is something that we make
available to all websites. However if the user hasn't pinned the
website the user would get prompted. So really the pinning is mainly
about UX. I.e. the prompting UX disappear, and the website can choose
to hide URL bar or appear in the webactivity picker.

In our implementation we're largely replacing the app registry with a pinned sites store (which replaces the bookmarks store) in Gaia. To begin with this is a DataStore but eventually I would like it if this could become part of a Places web service, working offline via cross-origin Service Workers.

One part I'm not sure about is if we're saying we want to grant permissions based on pinning, how will Gecko know when a site is pinned? Currently an app is registered in the app registry when it's installed via the Apps API using its app manifest so that Gecko knows about it. What should be the equivalent in this new content model?

Ben

1. https://wiki.mozilla.org/FirefoxOS/Pinning_the_Web

[Paul Theriault]

> One part I'm not sure about is if we're saying we want to grant permissions based on pinning, how will Gecko know when a site is pinned? Currently an app is registered in the app registry when it's installed via the Apps API using its app manifest so that Gecko knows about it. What should be the equivalent in this new content model?

We discussed this in another thread, but Jonas convinced me that pinning will not have any security side-effects, and I’ve updated the permission model to reflect this [1].

I assume “installation” will remain for add-ons and will remain as gate to implicit permissions.

[1] https://wiki.mozilla.org/Security/B2G/PermissionReview/New_permission_model#Web_Permissions

[Benjamin Francis]

On 11 June 2015 at 06:17, Paul Theriault <pther...@mozilla.com> wrote:

We discussed this in another thread, but Jonas convinced me that pinning will not have any security side-effects, and I’ve updated the permission model to reflect this [1].

To clarify, have you decided that granting extra storage, the ability to run in the background and the ability to appear in the web activity picker will have nothing to do with whether or not the site is pinned, so Gecko doesn't need to know? Does that mean that the user will always be prompted for these permissions instead?
 

I assume “installation” will remain for add-ons and will remain as gate to implicit permissions.

That's my understanding too, that add-ons will be "installed".

Ben

[Paul Theriault]

On 11 Jun 2015, at 10:39 pm, Benjamin Francis <bfra...@mozilla.com> wrote:

On 11 June 2015 at 06:17, Paul Theriault <pther...@mozilla.com> wrote:

We discussed this in another thread, but Jonas convinced me that pinning will not have any security side-effects, and I’ve updated the permission model to reflect this [1].

To clarify, have you decided that granting extra storage, the ability to run in the background and the ability to appear in the web activity picker will have nothing to do with whether or not the site is pinned, so Gecko doesn't need to know?

Thats correct. Though you are mixing things up a little here by throwing in Activities. That’s not a permission, that’s just something that is registered at install time. (In the same way that system message listeners are declared). Without an install step, we will need a new way for apps to register anything that was previously registered in the manifest that we want to continue to support.

Does that mean that the user will always be prompted for these permissions instead?

Yes, for now. But it doesn’t have to be, I would like to see improved UX so we don’t need to prompt up front for things that don’t really need user permission. For example, currently the only two manifest-declared web permissions are Alarms[1] & Storage. Neither of these would be terribly bad to grant to a website, so long as the user had a way to a) see that the permission was being used and b) had a way to respond if the website abuses the permission. I imagining something like a notification, when these APIs are used. In the case of alarms, we might have a persistent Alarm notification set, so the user is aware that there are pending alarms (and can remove them if they don’t want to be woken up!). I also imagine something an “about:permissions” style UX that allows the user to block annoying sites, and set defaults (a user might want to decide that alarms should be denied by default, and only allowed by exception).

[1] I remember elsewhere Jonas suggested to merge Alarms into something like a “run in the background” permission. Imho we should keep alarms separate since its easy to explain and something the user can be expected to manage. This implications of “run in the background” aren’t very clear to me, especially when our app model already does’t suspend apps that are not foreground.

[Benjamin Francis]

On 12 June 2015 at 04:46, Paul Theriault <pther...@mozilla.com> wrote:

Thats correct. Though you are mixing things up a little here by throwing in Activities. That’s not a permission, that’s just something that is registered at install time. (In the same way that system message listeners are declared). Without an install step, we will need a new way for apps to register anything that was previously registered in the manifest that we want to continue to support.

Yeah that's really what I'm getting at. Do we still need window.navigator.mozApps.install() ? If not then how do we do all the other things that we currently register at install time? Do we need new (internal?) APIs for those things?

* activities

* appcache_path (I guess we will deprecate this, but we might want to replace it with service worker integration)

* customizations (I guess we can keep this for the manifests of addons)

* datastores-owned/datastores-access (I hope we can deprecate DataStore eventually but we need it for now)

* fullscreen (I think we can replace this with display_mode and handle it in Gaia for pinned sites only)

* messages (as you mentioned)

* orientation (I think we can support this in Gaia for pinned sites)

Ben

[Jonas Sicking]

On Fri, Jun 12, 2015 at 10:41 AM, Benjamin Francis <bfra...@mozilla.com> wrote:
> On 12 June 2015 at 04:46, Paul Theriault <pther...@mozilla.com> wrote:
>>
>> Thats correct. Though you are mixing things up a little here by throwing
>> in Activities. That’s not a permission, that’s just something that is
>> registered at install time. (In the same way that system message listeners
>> are declared). Without an install step, we will need a new way for apps to
>> register anything that was previously registered in the manifest that we
>> want to continue to support.
>
>
> Yeah that's really what I'm getting at. Do we still need
> window.navigator.mozApps.install() ? If not then how do we do all the other
> things that we currently register at install time? Do we need new
> (internal?) APIs for those things?

We will definitely need a FirefoxOS-specific API which tracks the list
of URLs that have been bookmarked, and which maybe caches manifest
information as needed. Likely this API would also need to send
notifications when pins are added/removed.

One thing that is tricky is that in the new manifest spec the manifest
URL is meaningless. In order to update an icon we need to re-download
the HTML file and see what manifest URL it points to.

This is especially problematic if the user "pinned the website" rather
than "pinned the page", i.e. if we bookmarked the start_url rather
than the current page URL. What do we do if the page on the start_url
doesn't link to a manifest? Or links to a different manifest? Does the
spec address this?

I definitely am hoping that the entire navigator.mozApps API goes
away. At least I don't see any needs for it for FirefoxOS.

> * activities

Yeah, the system app needs to use the notifications mentioned above to
track when a website gets pinned and that website uses a manifest
which contain activity data.

> * appcache_path (I guess we will deprecate this, but we might want to
> replace it with service worker integration)

I don't think this is needed. The user has visited the website anyway,
and so the appcache should have already been installed.

> * customizations (I guess we can keep this for the manifests of addons)

I'm not sure what you mean by "customizations".

But yes, I'm sure we'll need APIs for managing the list of installed
addons exposed to various gaia apps, like settings and system app. And
we'll need to expose something to the amo website as well.

> * datastores-owned/datastores-access (I hope we can deprecate DataStore
> eventually but we need it for now)

I'm hoping that we can deprecate DataStore as well. In the meantime I
don't think we should expose it to privileged content. Hence I think
we can do whatever we need to here in order to keep allowing Gaia to
work. No need to be overly concerned about the APIs.

> * fullscreen (I think we can replace this with display_mode and handle it in
> Gaia for pinned sites only)
> * messages (as you mentioned)
> * orientation (I think we can support this in Gaia for pinned sites)

I'm not sure what you mean by this. But if needed we can automatically
grant websites some permissions when they are pinned. Not sure if
we'll do this in gecko or in gaia. I don't think that matters really,
we should do whatever's easiest to implement.

/ Jonas

[Benjamin Francis]

On 16 June 2015 at 00:59, Jonas Sicking <jo...@sicking.cc> wrote:

We will definitely need a FirefoxOS-specific API which tracks the list
of URLs that have been bookmarked, and which maybe caches manifest
information as needed. Likely this API would also need to send
notifications when pins are added/removed.

Any idea who might work on this API? What are the next steps?

One thing that is tricky is that in the new manifest spec the manifest
URL is meaningless. In order to update an icon we need to re-download
the HTML file and see what manifest URL it points to.

This is especially problematic if the user "pinned the website" rather
than "pinned the page", i.e. if we bookmarked the start_url rather
than the current page URL. What do we do if the page on the start_url
doesn't link to a manifest? Or links to a different manifest? Does the
spec address this?

Yeah this is annoying. But thinking about this, do we actually need to fetch the page on every update? The spec says you can just fetch the manifest URL [1]. We definitely need to check the start_url in the manifest is same origin with the document the manifest is referenced from at "pin" time, but after it's pinned do we need to keep checking that on every update? We could just keep a record of the URL the site was pinned from to do these checks on update, or make sure the origin of the start_url never changes.

We do need an API for Gaia to tell Gecko to fetch a manifest with a particular browsing context (I proposed a getManifest method on the Browser API in bug 1169633).

 

I definitely am hoping that the entire navigator.mozApps API goes
away. At least I don't see any needs for it for FirefoxOS.

> * activities

Yeah, the system app needs to use the notifications mentioned above to
track when a website gets pinned and that website uses a manifest
which contain activity data.

OK, so web activity registration should happen in Gaia. We can probably track that in IndexedDB if we get all the necessary events from Gecko (I'd need to check).

 

> * appcache_path (I guess we will deprecate this, but we might want to
> replace it with service worker integration)

I don't think this is needed. The user has visited the website anyway,
and so the appcache should have already been installed.

OK.
 

> * customizations (I guess we can keep this for the manifests of addons)

I'm not sure what you mean by "customizations".

It's a new property in the manifest used by addons. I didn't know about it until I looked.

But yes, I'm sure we'll need APIs for managing the list of installed
addons exposed to various gaia apps, like settings and system app. And
we'll need to expose something to the amo website as well.

> * datastores-owned/datastores-access (I hope we can deprecate DataStore
> eventually but we need it for now)

I'm hoping that we can deprecate DataStore as well. In the meantime I
don't think we should expose it to privileged content. Hence I think
we can do whatever we need to here in order to keep allowing Gaia to
work. No need to be overly concerned about the APIs.

We need a solution for the new privileged homescreen app accessing the pinned sites/places Data Stores. I'd suggest that either we need to allow privileged apps with the "homescreen" role to access Data Stores, or come up with a Data Store replacement pretty sharpish.

 

> * fullscreen (I think we can replace this with display_mode and handle it in
> Gaia for pinned sites only)
> * messages (as you mentioned)
> * orientation (I think we can support this in Gaia for pinned sites)

I'm not sure what you mean by this. But if needed we can automatically
grant websites some permissions when they are pinned. Not sure if
we'll do this in gecko or in gaia. I don't think that matters really,
we should do whatever's easiest to implement.

I think currently the display mode is saved in Gecko and passed to Gaia through the launch event. We can do this entirely in Gaia instead if needed.

1. https://w3c.github.io/manifest/#updating

[Jonas Sicking]

On Tue, Jun 16, 2015 at 12:02 PM, Benjamin Francis <bfra...@mozilla.com> wrote:
> On 16 June 2015 at 00:59, Jonas Sicking <jo...@sicking.cc> wrote:
>>
>> We will definitely need a FirefoxOS-specific API which tracks the list
>> of URLs that have been bookmarked, and which maybe caches manifest
>> information as needed. Likely this API would also need to send
>> notifications when pins are added/removed.
>
> Any idea who might work on this API? What are the next steps?

I think anyone with gecko experience could.

I think we generally should stop worrying about the exact nature of
"sensitive APIs", i.e. APIs that are sensitive enough we can't expose
them to unsigned content. And this falls into that category. APIs that
require signature stand no chance of getting standardized within the
next few years, and I think that's ok.

I am definitely one of the primary people that has been overly
concerned about getting these types of APIs exactly right, but I think
that has been a mistake on my part.

But could we perhaps simply use a DataStore here for now? And have the
system app responsible for checking for and downloading updated
manifests. The system app could even download the necessary icon files
and stick them as Blobs in the DataStore.

>> One thing that is tricky is that in the new manifest spec the manifest
>> URL is meaningless. In order to update an icon we need to re-download
>> the HTML file and see what manifest URL it points to.
>>
>> This is especially problematic if the user "pinned the website" rather
>> than "pinned the page", i.e. if we bookmarked the start_url rather
>> than the current page URL. What do we do if the page on the start_url
>> doesn't link to a manifest? Or links to a different manifest? Does the
>> spec address this?
>
> Yeah this is annoying. But thinking about this, do we actually need to fetch
> the page on every update? The spec says you can just fetch the manifest URL
> [1].

That's a poor suggestion by the spec. By encouraging the manifest to
be hosted on a CDN, we should expect that developers will change the
manifest URL any time the manifest content changes.

While the spec might allow refetching the manifest only, it will
definitely cause users to use an outdated manifest.

> We definitely need to check the start_url in the manifest is same
> origin with the document the manifest is referenced from at "pin" time, but
> after it's pinned do we need to keep checking that on every update? We could
> just keep a record of the URL the site was pinned from to do these checks on
> update, or make sure the origin of the start_url never changes.

I suspect that if the user chooses to bookmark the start_url, then we
should simply refetch the HTML page at the start_url every time we
want to look for an updated icon.

I don't think this refetching is a big deal though.

> We do need an API for Gaia to tell Gecko to fetch a manifest with a
> particular browsing context (I proposed a getManifest method on the Browser
> API in bug 1169633).

Given that we are getting rid of different cookie jars for different
"apps", I don't think using the right browsing context is particularly
important any more.

I'm still totally fine with having gecko do the fetching if that's
simpler. Another alternative is to have the system app do it.

>> But yes, I'm sure we'll need APIs for managing the list of installed
>> addons exposed to various gaia apps, like settings and system app. And
>> we'll need to expose something to the amo website as well.
>>
>> > * datastores-owned/datastores-access (I hope we can deprecate DataStore
>> > eventually but we need it for now)
>>
>> I'm hoping that we can deprecate DataStore as well. In the meantime I
>> don't think we should expose it to privileged content. Hence I think
>> we can do whatever we need to here in order to keep allowing Gaia to
>> work. No need to be overly concerned about the APIs.
>
> We need a solution for the new privileged homescreen app accessing the
> pinned sites/places Data Stores. I'd suggest that either we need to allow
> privileged apps with the "homescreen" role to access Data Stores, or come up
> with a Data Store replacement pretty sharpish.

I suggest we use a DataStore for now.

/ Jonas

[Benjamin Francis]

On 16 June 2015 at 00:59, Jonas Sicking <jo...@sicking.cc> wrote:

We will definitely need a FirefoxOS-specific API which tracks the list
of URLs that have been bookmarked, and which maybe caches manifest
information as needed. Likely this API would also need to send
notifications when pins are added/removed.

I think this API is now the biggest unknown platform dependency for the Pin the Web features for 2.5.

I've been thinking about what this API might look like, and I've been struggling to think of many ways in which it is different to the existing Apps API in terms of its interface.

We might say that the biggest difference is that a site no longer requires a manifest in order to be pinned/installed. But all of the features we've talked about which require Gecko to know about a pinned site actually require a manifest anyway. How will this API be substantially different than the existing Apps API? How much of this could we achieve with the existing API for "pinned apps" (i.e. sites with a manifest) and a DataStore for "pinned sites" (i.e. sites without a manifest), then just amalgamate them together in the system app like we already do for apps and bookmarks, but hide the differences from users?

Should we be focusing on removing separate data jars rather than re-implementing the Apps API?

Ben

[Benjamin Francis]

On 16 June 2015 at 00:59, Jonas Sicking <jo...@sicking.cc> wrote:

One thing that is tricky is that in the new manifest spec the manifest
URL is meaningless. In order to update an icon we need to re-download
the HTML file and see what manifest URL it points to.

This is especially problematic if the user "pinned the website" rather
than "pinned the page", i.e. if we bookmarked the start_url rather
than the current page URL. What do we do if the page on the start_url
doesn't link to a manifest? Or links to a different manifest? Does the
spec address this?

My understanding is that the spec assumes that the manifest URL does not change, including when stored on a CDN. It says that the user agent may "periodically check if the contents of a manifest has been modified (e.g., by honoring HTTP cache directives associated with the manifest or by checking for updates after the web application has been launched)".

You do need to keep a record of the URL of the document the app was installed from (or at least its origin), in order to check that any future start_url or scope are on the same origin as that document, because these are steps you need to take when processing any new or updated manifest.

There are quite a lot of assumptions here though so I have filed an issue to ask for clarification https://github.com/w3c/manifest/issues/384

[Benjamin Francis]

On 12 June 2015 at 18:41, Benjamin Francis <bfra...@mozilla.com> wrote:

Yeah that's really what I'm getting at. Do we still need window.navigator.mozApps.install() ? If not then how do we do all the other things that we currently register at install time? Do we need new (internal?) APIs for those things?

* activities

* appcache_path (I guess we will deprecate this, but we might want to replace it with service worker integration)

* customizations (I guess we can keep this for the manifests of addons)

* datastores-owned/datastores-access (I hope we can deprecate DataStore eventually but we need it for now)

* fullscreen (I think we can replace this with display_mode and handle it in Gaia for pinned sites only)

* messages (as you mentioned)

* orientation (I think we can support this in Gaia for pinned sites)

I'm starting to talk to myself a little bit now (sorry), but it occurs to me that maybe it's only pinning Firefox Apps (i.e. Firefox specific signed content) that Gecko needs to know about, I can't think of any reason Gecko needs to know when the user pins a web app/web site.

Maybe we need to keep the Apps API for Firefox Apps (and addons), and web apps can be handled entirely in Gaia?

Is there some reason Gecko needs to know when a web app/web site is pinned? Web apps can't use proprietary APIs like Web Activities, Data Stores and system messages that need registering at pin time. Are there any differences in storage quotas etc. that mean Gecko needs to know? Maybe if we added Service Worker integration into the manifest Gaia would need a way to trigger a Service Worker registration in Gecko? With the current approach to app scope we can get away with not needing to register scope with Gecko.

Ben

[Jonas Sicking]

On Wed, Jul 1, 2015 at 2:11 PM, Benjamin Francis <bfra...@mozilla.com> wrote:
> On 16 June 2015 at 00:59, Jonas Sicking <jo...@sicking.cc> wrote:
>>

>> We will definitely need a FirefoxOS-specific API which tracks the list
>> of URLs that have been bookmarked, and which maybe caches manifest
>> information as needed. Likely this API would also need to send
>> notifications when pins are added/removed.
>
> I think this API is now the biggest unknown platform dependency for the Pin
> the Web features for 2.5.
>
> I've been thinking about what this API might look like, and I've been
> struggling to think of many ways in which it is different to the existing
> Apps API in terms of its interface.

We currently have two registries: Apps and bookmarks.

Things that go into the app registry are given additional permissions
but are also put in a separate cookie jar. This cookie jar is extra
aggressive given that anything linked from within the cookie jar stays
within the cookie jar.

Things that go into the bookmark registry stay within the browser
cookie jar, but are not given additional permissions.


What we *want* should happen when the user pins a website is:

* Grant the origin unlimited persistent storage. This requires
implementing at least the backend pieces of
https://storage.spec.whatwg.org/#api as well as make localStorage run
on the quota manager backend.
* Grant ability to use Push, BackgroundSync and Alarm APIs. But right
now these rely on signing and system messages. I'm somewhat reluctant
to spend the effort to make these work for normal websites given that
there are standardized versions of these based on ServiceWorkers
coming soon.
* *Maybe* grant the ability to use notifications. I'll leave this to
the UX team. I think we have an opportunity to differentiate by
creating a device which doesn't constantly notify you about crap you
don't care about.
* Register webactivities.
* More aggressively update the ServiceWorker.
* Do something for websites that use a display-mode and/or an
orientation. It's a bit unclear to me what we should actually do here,
especially for manifests that don't have a scope.

However only the first one is really going to affect websites given
that extremely few websites use any of the other features.

So in practice, if we don't do any of this, and simply treat pins as
bookmarks, there will be relatively few downsides.


If, on the other hand, we stick pins in the application registry, the
main thing that would immediately happen is that pinned content would
get its own cookie jar. I.e. as soon as you pin something you'd get
automatically logged out.

We could definitely change that. But given the relatively few
downsides of just using the bookmarks database I think we should do
that for now. That way we'll have more freedom to modify what is
stored since it can be handled entirely within Gaia.

Once we have that more figured out, and once we have more of the new
security model in place, it definitely sounds like a plausible option
to simply use the apps registry to store pins. Part of that will
definitely be to remove the existing data-jar-per-app policy that we
currently have.

That sounds like something that should fit in the 3.0 release without
too much hassle.

> We might say that the biggest difference is that a site no longer requires a
> manifest in order to be pinned/installed. But all of the features we've
> talked about which require Gecko to know about a pinned site actually
> require a manifest anyway.

I don't think that's true. We can get a nice-looking icon and a icon
name using just <meta> and <title> tags. I would expect that that
covers by far most of what people will put in the manifest.

And there are *far* more websites out there that has that has
apple-touch-icon <meta> information, than have manifests.

/ Jonas

[Jonas Sicking]

On Wed, Jul 1, 2015 at 2:20 PM, Benjamin Francis <bfra...@mozilla.com> wrote:
> On 16 June 2015 at 00:59, Jonas Sicking <jo...@sicking.cc> wrote:
>>

>> One thing that is tricky is that in the new manifest spec the manifest
>> URL is meaningless. In order to update an icon we need to re-download
>> the HTML file and see what manifest URL it points to.
>>
>> This is especially problematic if the user "pinned the website" rather
>> than "pinned the page", i.e. if we bookmarked the start_url rather
>> than the current page URL. What do we do if the page on the start_url
>> doesn't link to a manifest? Or links to a different manifest? Does the
>> spec address this?
>
>
> My understanding is that the spec assumes that the manifest URL does not
> change, including when stored on a CDN.

Generally speaking, CDNs use a new URL any time you want to change the
file contents. So while the spec might want developers to keep the
manifest at a stable URL, I doubt that authors will.

Especially if the only downside of changing the URL is that the icon
in FirefoxOS won't update.

Very few developers will think about how to update the icon when they
originally deploy the website. That won't be something that they think
about until it actually comes time to update the branding, which often
won't be long after they first deploy.

If they at that point have the manifest hosted on the CDN (as also
encouraged by the spec), then they won't have any choice but to change
the manifest URL.

And even fewer developers will worry about what FirefoxOS does or does
not do. At least until we get significantly larger marketshare.

> It says that the user agent may
> "periodically check if the contents of a manifest has been modified (e.g.,
> by honoring HTTP cache directives associated with the manifest or by
> checking for updates after the web application has been launched)".

Does any other implementation actually do that?

But more importantly, developers will only realize that this is what's
happening long after it's too late.

> There are quite a lot of assumptions here though so I have filed an issue to
> ask for clarification https://github.com/w3c/manifest/issues/384

Yeah, the spec definitely needs to be updated here. As long as the
spec allows the manifest to be hosted on a different origin, and thus
on a CDN, we should assume that the manifest URL is meaningless and
won't be kept valid for any extended period of time.

Nothing that the spec says here will make a difference short of
requiring that the manifest is same-origin. Otherwise developers will
host the manifest on a CDN and will be forced to update the manifest
URL when they want to update the manifest contents.

/ Jonas

[Benjamin Francis]

On 7 July 2015 at 07:12, Jonas Sicking <jo...@sicking.cc> wrote:

So in practice, if we don't do any of this, and simply treat pins as
bookmarks, there will be relatively few downsides.


If, on the other hand, we stick pins in the application registry, the
main thing that would immediately happen is that pinned content would
get its own cookie jar. I.e. as soon as you pin something you'd get
automatically logged out.

We could definitely change that. But given the relatively few
downsides of just using the bookmarks database I think we should do
that for now. That way we'll have more freedom to modify what is
stored since it can be handled entirely within Gaia.

I think I agree... we could deliver an MVP for pinning sites which provide a W3C manifest (or no manifest) without needing to use the Apps API or a new Gecko API, it's feasible to handle them entirely within Gaia.

But how should we handle the case where the user navigates to a Firefox App in the browser (either a legacy hosted Open Web App or a new signed streamable packaged app), and then tries to pin it? Would we a) just pin it like a web site using metadata provided by the page and not provide any additional features, keeping the existing separate install flow in the Marketplace or b) require that all Firefox Apps provide a manifest link relation in their pages pointing towards a manifest with proprietary Mozilla properties, somehow detect the difference, then internally install the app via the exiting Apps API?

I would have a strong preference for b), but I'm not sure how feasible that is from a BizDev/Evangelism point of view. We'd need to be clear on how this is going to work very soon to set those balls rolling.

Once we have that more figured out, and once we have more of the new
security model in place, it definitely sounds like a plausible option
to simply use the apps registry to store pins. Part of that will
definitely be to remove the existing data-jar-per-app policy that we
currently have.

That sounds like something that should fit in the 3.0 release without
too much hassle.

I agree, I think it's likely that 2.5 is going to be a bit of an in-between state where we support both models to some extent (hopefully we can hide that from users). In 3.0 we can hopefully unify things more at the platform level.

> We might say that the biggest difference is that a site no longer requires a
> manifest in order to be pinned/installed. But all of the features we've
> talked about which require Gecko to know about a pinned site actually
> require a manifest anyway.

I don't think that's true. We can get a nice-looking icon and a icon
name using just <meta> and <title> tags. I would expect that that
covers by far most of what people will put in the manifest.

And there are *far* more websites out there that has that has
apple-touch-icon <meta> information, than have manifests.

I think you may have misunderstood me a little here, I was saying that we shouldn't require a manifest in order to be pinned, but we will use any metadata provided (including a manifest), and it's mostly Firefox Apps that Gecko needs to be made aware of.

Ben

[Benjamin Francis]

On 7 July 2015 at 07:21, Jonas Sicking <jo...@sicking.cc> wrote:

Generally speaking, CDNs use a new URL any time you want to change the
file contents.

Alex Russell claims this impression is outdated, that the vast majority of modern CDNs do allow for stable URLs. How can we get some objective data on this?
 

Very few developers will think about how to update the icon when they
originally deploy the website. That won't be something that they think
about until it actually comes time to update the branding, which often
won't be long after they first deploy.

That is probably true.
 

> It says that the user agent may
> "periodically check if the contents of a manifest has been modified (e.g.,
> by honoring HTTP cache directives associated with the manifest or by
> checking for updates after the web application has been launched)".

Does any other implementation actually do that?

No not as far as I know, it's still untested. Chrome doesn't currently implement the updating, though I understand they plan to in the future. I'm trying to find out about Opera.
 

Yeah, the spec definitely needs to be updated here. As long as the
spec allows the manifest to be hosted on a different origin, and thus
on a CDN, we should assume that the manifest URL is meaningless and
won't be kept valid for any extended period of time.

I would have preferred the same-origin restriction, but I'm not entirely convinced this is true. I would really like to see some data on this.
 

Nothing that the spec says here will make a difference short of
requiring that the manifest is same-origin.

Unfortunately we lost that battle. If you want to continue to push for that in the spec, we're going to need to come up with some compelling new arguments.

Ben

[Jonas Sicking]

I don't think it's at all feasible. And it would still cause the user
to get logged out when you pin the site, which likely means that it's
worse for the user than simply doing a). I don't think very many
hosted apps use the extra permissions which means that the difference
between a) and b) is very little, other than that b) logs you out.

For streamable packaged apps in 2.5 we're most likely going to grant
apps all the needed permissions upon simply navigating to the app. So
again b) makes very little difference other than logging the user out.

>> > We might say that the biggest difference is that a site no longer
>> > requires a
>> > manifest in order to be pinned/installed. But all of the features we've
>> > talked about which require Gecko to know about a pinned site actually
>> > require a manifest anyway.
>>
>> I don't think that's true. We can get a nice-looking icon and a icon
>> name using just <meta> and <title> tags. I would expect that that
>> covers by far most of what people will put in the manifest.
>>
>> And there are *far* more websites out there that has that has
>> apple-touch-icon <meta> information, than have manifests.
>
>
> I think you may have misunderstood me a little here, I was saying that we
> shouldn't require a manifest in order to be pinned, but we will use any
> metadata provided (including a manifest),

Ok, cool, sounds like we're on the same page.

> and it's mostly Firefox Apps that
> Gecko needs to be made aware of.

I'm not sure what you mean by this. I definitely think that we should
make use of the apple meta tags to the same extent that we do
manifests. I believe the fennec team has found that the get a much
better user experience for their "bookmark to homepage" when making
use of the apple tags.

/ Jonas

[Jonas Sicking]

On Tue, Jul 7, 2015 at 6:29 AM, Benjamin Francis <bfra...@mozilla.com> wrote:
> On 7 July 2015 at 07:21, Jonas Sicking <jo...@sicking.cc> wrote:
>>
>> Generally speaking, CDNs use a new URL any time you want to change the
>> file contents.
>
>
> Alex Russell claims this impression is outdated, that the vast majority of
> modern CDNs do allow for stable URLs. How can we get some objective data on
> this?
>
>> Very few developers will think about how to update the icon when they
>> originally deploy the website. That won't be something that they think
>> about until it actually comes time to update the branding, which often
>> won't be long after they first deploy.
>
> That is probably true.
>
>> > It says that the user agent may
>> > "periodically check if the contents of a manifest has been modified
>> > (e.g.,
>> > by honoring HTTP cache directives associated with the manifest or by
>> > checking for updates after the web application has been launched)".
>>
>> Does any other implementation actually do that?
>
> No not as far as I know, it's still untested. Chrome doesn't currently
> implement the updating, though I understand they plan to in the future. I'm
> trying to find out about Opera.

Given this, and given that I don't actually see a downside to
re-downloading the HTML page again, I definitely think that that's
what we should do.

At the very least I'd like to see solid data on that CDNs most of the
time support updating the manifest in-place, and that other browsers
have near-term plans on implementing updates by re-pinging the
manifest URL.

Also keep in mind that for websites that don't use manifests, but
instead use meta tags, we'll have to re-download the HTML.

/ Jonas

[Benjamin Francis]

On 8 July 2015 at 04:36, Jonas Sicking <jo...@sicking.cc> wrote:

> But how should we handle the case where the user navigates to a Firefox App
> in the browser (either a legacy hosted Open Web App or a new signed
> streamable packaged app), and then tries to pin it? Would we a) just pin it
> like a web site using metadata provided by the page and not provide any
> additional features, keeping the existing separate install flow in the
> Marketplace or b) require that all Firefox Apps provide a manifest link
> relation in their pages pointing towards a manifest with proprietary Mozilla
> properties, somehow detect the difference, then internally install the app
> via the exiting Apps API?
> I would have a strong preference for b), but I'm not sure how feasible that
> is from a BizDev/Evangelism point of view. We'd need to be clear on how this
> is going to work very soon to set those balls rolling.

I don't think it's at all feasible. And it would still cause the user
to get logged out when you pin the site, which likely means that it's
worse for the user than simply doing a). I don't think very many
hosted apps use the extra permissions which means that the difference
between a) and b) is very little, other than that b) logs you out.

Are you you saying that un-signed hosted Firefox Apps should no longer be able to access alarms, push, audio channels, fmradio or unlimited storage, or be able to register web activities? So I will no longer be able to "share" something via Facebook/Twitter/Pinterest for example? Or are you saying that there should be two separate experiences of these apps/sites depending on whether you "pin" them from the browser chrome or "install" them from the Marketplace? Neither of these sound good to me.

For streamable packaged apps in 2.5 we're most likely going to grant
apps all the needed permissions upon simply navigating to the app. So
again b) makes very little difference other than logging the user out.

Are you saying that signed Firefox Apps will automatically be able to use alarms, push, audio channels, fmradio, unlimited storage, mozbrowser, input, mobilenetwork, nfc, systemXHR, tcp-socket, datastore, promptless fullscreen, system messages and registering web activities simply by navigating to a URL? Or will we prompt the user for every one of those features?

> and it's mostly Firefox Apps that
> Gecko needs to be made aware of.

I'm not sure what you mean by this.

I meant that it seems like for many of the features listed above, pinning a Firefox App needs to call some API in Gecko to activate those features only when the app is pinned. But that for pinning a web site/web app with a W3C manifest we can probably deliver an MVP without needing to call a Gecko API.

 

I definitely think that we should
make use of the apple meta tags to the same extent that we do
manifests. I believe the fennec team has found that the get a much
better user experience for their "bookmark to homepage" when making
use of the apple tags.

Yes, we should make use of site/app metadata provided by meta tags, potentially including proprietary meta tags (with developer warnings) where necessary for backwards compatibility with a large volume of existing content.

My view is that for simple site/app metadata (like specifying an application-name and icon) meta tags in the <head> of an HTML page are sufficient (though their scope is ambiguous). But to define a more complex collection of metadata like defining a scope, display mode, default orientation, start_url, default theme_color, a map of icons and a map of splash screens, a centralised JSON manifest is more practical.

Incidentally I've come to think about page metadata in the same way. For simple metadata like type/title/description/image, simple RDFa style meta tags using the Open Graph vocabularly in <head> are sufficient. For more complex use cases like complex structured data (e.g. contact details), associated actions and linked resources, a description in JSON using a vocabularly like schema.org is more practical.

So for the more complex app features described above, I don't think we can rely on meta tags in the <head> alone (unless we invent quite a long list of new meta tags to include in each page for things like registering web activities). I think we need a page of a site/app to be able to associate itself with a manifest for use during pinning.

Ben

[Benjamin Francis]

On 8 July 2015 at 04:41, Jonas Sicking <jo...@sicking.cc> wrote:

Given this, and given that I don't actually see a downside to
re-downloading the HTML page again, I definitely think that that's
what we should do.

Actually the conclusion I drew from the W3C GitHub thread [1] was that we should check for an updated manifest for a pinned site every time the user visits a page of the site (associated with the manifest via a link relation), and download an updated manifest if the cached copy has expired. It's important to do it whilst the site is being used so that the HTTP requests can be sent in the context of the site itself to ensure any necessary cookies, sessions or credentials are set.

If the referenced manifest URL has changed, we have to treat the new manifest as belonging to a separate site, because there's no way to ensure that the new manifest refers to the same site as the old manifest.

I don't think it's safe to rely on re-downloading the page the site was pinned from in order to try to derive a new manifest URL. A site can be pinned from any page of the site and if the page you pinned the site from is removed then updates will fail. This seems just as likely as the manifest URL changing. It's also not enough to simply do an XHR from the system app to the page URL, the user may need to be logged in in order to fetch the page.

I do agree that the W3C manifest spec has a bit of a built-in footgun at the moment which risks unwary web developers using misbehaving CDNs creating manifests which users never get updates for after pinning/installing a site/app, because the default configuration of their CDN is to change the manifest URL when the manifest is updated.

I don't think the best solution is to try to work around the spec by inventing our own way of updating which may just break in different ways. I think we need to try to quantify that risk, and if we find it to be significant, propose changes to the spec again.

 

Also keep in mind that for websites that don't use manifests, but
instead use meta tags, we'll have to re-download the HTML.

It's tricky to determine from the metadata of an arbitrary page of a given site whether a change is representative of the whole site, but yes we're going to have to work around that ambiguity for sites which don't provide a manifest with a clearly defined scope. We can update site metadata when the user visits a page of the site, in the same way that we update page metadata in the places database when the user visits a page.

Ben

1. https://github.com/w3c/manifest/issues/384

[J. Ryan Stinnett]

On Tue, Jul 7, 2015 at 10:41 PM, Jonas Sicking <jo...@sicking.cc> wrote:
> At the very least I'd like to see solid data on that CDNs most of the
> time support updating the manifest in-place

I might be misreading the question, but I'll give it a try.

>From both past work experience and vendor docs, CDN vendors typically
allow a URL to cleared by both:

* CDN TTL expiring, which is derived from an HTTP header (could be a
vendor specific header)
* Manual purge request for a URL, either via an API or management console

Looking at a graph of CDN market share[1], the top 3 vendors used on
Alexa top 1M sites are:

* Amazon CloudFront
* Akamai
* MaxCDN

Let's examine their updating abilities:

# Amazon CloudFront

* TTL control via Cache-Control or an Expires header, among others[2]
* Invalidation request to purge an object[3]

# Akamai

* TTL control via headers (docs seem to require an Akamai account to view)
* Purge specific files via API[4]

# MaxCDN

* TTL control via Cache-Control[5]
* Purge specific files via management console[6]

So, at least for all of these vendors, there are several options you
can use to update the manifest without changing the URL.

[1]: http://www.datanyze.com/market-share/CDN/
[2]: http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/RequestAndResponseBehaviorS3Origin.html#RequestS3Caching
[3]: http://docs.aws.amazon.com/AmazonCloudFront/latest/APIReference/CreateInvalidation.html
[4]: https://developer.akamai.com/api/purge/ccu/overview.html
[5]: https://www.maxcdn.com/one/tutorial/using-cache-control-headers/
[6]: https://www.maxcdn.com/one/tutorial/purge-cache-on-cdn/

- Ryan

[Jonas Sicking]

Yeah, we need to figure out something to do for webactivities. One
option is to expose an API to the system app to enable it to
register/unregister activities for a given manifest.

>> For streamable packaged apps in 2.5 we're most likely going to grant
>> apps all the needed permissions upon simply navigating to the app. So
>> again b) makes very little difference other than logging the user out.
>
> Are you saying that signed Firefox Apps will automatically be able to use
> alarms, push, audio channels, fmradio, unlimited storage, mozbrowser, input,
> mobilenetwork, nfc, systemXHR, tcp-socket, datastore, promptless fullscreen,
> system messages and registering web activities simply by navigating to a
> URL?

Yes. In 2.5. In 3.0 I'd like to improve this by still prompting for
things like we normally prompt for, like fullscreen, notifications,
and unlimited storage.

>> > and it's mostly Firefox Apps that
>> > Gecko needs to be made aware of.
>>
>> I'm not sure what you mean by this.
>
> I meant that it seems like for many of the features listed above, pinning a
> Firefox App needs to call some API in Gecko to activate those features only
> when the app is pinned. But that for pinning a web site/web app with a W3C
> manifest we can probably deliver an MVP without needing to call a Gecko API.

Sounds good.

/ Jonas