API for editing existing app installations

[Matt Basta]

Hi all

This was brought up in a bug, but the conversation got muddied across a number of different issues. One thing I'd like to see is an API that allows an app (a "marketplace app") to modify the manifest URLs (for hosted apps) or mini manifest URLs (for packaged apps) of apps that the marketplace app has installed. This allows us to do a few things:

- Prevent users from getting updates to apps that are incompatible with their devices (i.e. a developer releases a new version of a packaged app that doesn't work on older devices which may have previously installed the app)
- Enable blocklisting of hosted apps (currently impossible)
- Allow hosted apps to change domains (i.e.: "acmeapp.com" moves to "getacme.com"; app changes from HTTP to HTTPS; currently impossible)

I imagine an API that's privileged-only that works something like this:

navigator.mozApps.changeManifest("http://old.manifest/url.webapp", "https://new.manifest/url.webapp")

Note the restrictions:

- Can only be used for manifest URLs which were originally installed by the marketplace app (i.e.: if the Firefox Marketplace didn't install App X, it can't change App X's manifest URL)
- For privileged apps initially (though this can probably be lifted at some point, given the previous restriction)

What are your ideas about an API like this?


-basta

[Mounir Lamouri]

On 22/05/13 17:58, Matt Basta wrote:
> Hi all
>
> This was brought up in a bug, but the conversation got muddied across a number of different issues. One thing I'd like to see is an API that allows an app (a "marketplace app") to modify the manifest URLs (for hosted apps) or mini manifest URLs (for packaged apps) of apps that the marketplace app has installed. This allows us to do a few things:
>
> - Prevent users from getting updates to apps that are incompatible with their devices (i.e. a developer releases a new version of a packaged app that doesn't work on older devices which may have previously installed the app)

How do you know that an update is incompatible? Shouldn't that
information lives in the manifest? Then, can't the runtime (ie. Firefox
OS) simply ignore the update. It could even let the user know that the
lastest update is incompatible.

> - Enable blocklisting of hosted apps (currently impossible)

Hosted application have identifier based on their manifest URL so the
runtime can take care of that. The same that Gecko has code to block
malicious addons.

> - Allow hosted apps to change domains (i.e.: "acmeapp.com" moves to "getacme.com"; app changes from HTTP to HTTPS; currently impossible)

Can't the runtime be clever and if there is an HTTP REDIRECT when trying
to update a manifest? It would automatically make that happen if the
redirection points to a valid manifest.

--
Mounir

[Matt Basta]

> How do you know that an update is incompatible?

The marketplace currently generates a feature profile, which the marketplace API can understand and decide whether the app is compatible with the device. The platform/OS have no knowledge of this, and just poll whatever URL is thrown at them for updates. In this case, the mini manifest URL is the Marketplace API, which could decide whether to serve an update or not.

Could this go in the manifest? Sure, but then the client needs to know about every single feature that we sniff for, even ones that Firefox OS/Android/Desktop don't support.

Also, warning a user that an app update isn't compatible is a little cruel: they can't get the update regardless, so why taunt them about it? "Your phone doesn't support the new version of the app. Too bad!" There's no action there. The user can't choose to do anything differently in many cases.

> Hosted application have identifier based on their manifest URL so the runtime can take care of that.

If the runtime is polling the Marketplace, that's a bad thing and works against the idea of federated marketplaces, and is partly why hosted app blocklisting doesn't exist today. Hosted app manifests don't live on Marketplace servers, and not all apps may have been installed from the Firefox Marketplace.

> Can't the runtime be clever and if there is an HTTP REDIRECT when trying to update a manifest?

Consider the case of Bit.ly a few years ago when there was that big scare that Libya was going to retract all of the LY TLDs. If an app owner suddenly lost control over their domain for legal reasons or otherwise, their app is permanently broken on all users' devices, and potentially could be hijacked by a third party in the future.


-basta

_______________________________________________
dev-webapps mailing list
dev-w...@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-webapps

[Mounir Lamouri]

On 23/05/13 16:36, Matt Basta wrote:
>> How do you know that an update is incompatible?
>
> The marketplace currently generates a feature profile, which the marketplace API can understand and decide whether the app is compatible with the device. The platform/OS have no knowledge of this, and just poll whatever URL is thrown at them for updates. In this case, the mini manifest URL is the Marketplace API, which could decide whether to serve an update or not.

Doesn't the manifest contain the list of required feature? Can't that be
used by the runtime to ignore some applications?

> Could this go in the manifest? Sure, but then the client needs to know about every single feature that we sniff for, even ones that Firefox OS/Android/Desktop don't support.

I'm not sure I understand what you meant here. Do you have examples?

> Also, warning a user that an app update isn't compatible is a little cruel: they can't get the update regardless, so why taunt them about it? "Your phone doesn't support the new version of the app. Too bad!" There's no action there. The user can't choose to do anything differently in many cases.

This is a UX issue but as a user I would like to know that I do no
longer get updates because my OS/device is getting too old. A user my
trying to push the developer of an app to get the latest version working
on its device or even change device or switch to an alternative application.

>> Hosted application have identifier based on their manifest URL so the runtime can take care of that.
>
> If the runtime is polling the Marketplace, that's a bad thing and works against the idea of federated marketplaces, and is partly why hosted app blocklisting doesn't exist today. Hosted app manifests don't live on Marketplace servers, and not all apps may have been installed from the Firefox Marketplace.

I do not think the marketplaces should be put in the centre of the
system because you can install hosted applications from anywhere so you
can't make the marketplaces owning the blacklist of applications.

As I said, having Mozilla taking care of blacklist for its runtime is
probably the best solution so if a user is using another marketplace
than the one from Mozilla, that user would be as safe as anyone else.
This is a mechanism we use for websites and extensions.

>> Can't the runtime be clever and if there is an HTTP REDIRECT when trying to update a manifest?
>
> Consider the case of Bit.ly a few years ago when there was that big scare that Libya was going to retract all of the LY TLDs. If an app owner suddenly lost control over their domain for legal reasons or otherwise, their app is permanently broken on all users' devices, and potentially could be hijacked by a third party in the future.

Isn't that a problem you have on the Web in general? If someone takes
over your domain, you are screwed. If I host a popular web page or web
service that attracts a lot of users and suddenly, I do no longer own
the domain, all my users will be in danger because they trusted me but
not the new owners. Given that hosted web applications are no more than
web applications with a manifest to make their integration in the
runtime better, trying to solve this problem does not seem to be in scope.

--
Mounir

[Matt Basta]

> Doesn't the manifest contain the list of required feature?

A very old version of the spec contains a "requiredFeatures" field, but it was never fully defined and isn't supported, and no values for the field were specified.

Permissions are available in the manifest, but that doesn't cover things that the phone doesn't consider a feature. Case in point: we are adding four items to our feature profile this week to detect the different flavors of getUserMedia (audio, video, images, and screen).

> I'm not sure I understand what you meant here. Do you have examples?

If there's a new feature we want to add compatibility testing for that some devices might already have, we can't rely on the devices to implicitly know that they support the feature. If we decide, for instance, that we want to detect support for <input type="color">, the user agent would need to be able to say, "Oh, I know what <input type="color"> is and I do[n't] support it.".

We already have this functionality in our feature profiles, so no sense in reinventing the wheel.

> I do not think the marketplaces should be put in the centre of the
> system because you can install hosted applications from anywhere so you
> can't make the marketplaces owning the blacklist of applications.

Right, and as I mentioned in my original email, a marketplace could only use this API for manifests *which it installed itself*. Relying on Mozilla to blacklist apps which were distributed by a third party who could otherwise do the blacklisting themselves seems very bad. If carriers start distributing their own branded marketplaces, making them go through us to flag and remove malware seems like a dangerous and slow process.

> Isn't that a problem you have on the Web in general? If someone takes
> over your domain, you are screwed. If I host a popular web page or web
> service that attracts a lot of users and suddenly, I do no longer own
> the domain, all my users will be in danger because they trusted me but
> not the new owners.

Of course, but we're in a different boat than the web at large. Our users are paying us to be able to install these apps, in some cases, and if a developer comes to us and says, "Hey, we lost our short URL because <insert completely valid and unfortunate reason here> and now the app doesn't work. Can we get you to update the manifests?" we should have the power to make that happen.

Granted, that's a feature which isn't on any roadmaps, but it's something that we can't currently do that an API like this would enable for us.

----- Original Message -----
From: "Mounir Lamouri" <mou...@lamouri.fr>
To: dev-w...@lists.mozilla.org
Sent: Friday, May 24, 2013 3:24:03 AM
Subject: Re: API for editing existing app installations

[Fabrice Desre]

On 05/24/2013 08:34 AM, Matt Basta wrote:
>
> Of course, but we're in a different boat than the web at large. Our users are paying us to be able to install these apps, in some cases, and if a developer comes to us and says, "Hey, we lost our short URL because <insert completely valid and unfortunate reason here> and now the app doesn't work. Can we get you to update the manifests?" we should have the power to make that happen.
>
> Granted, that's a feature which isn't on any roadmaps, but it's something that we can't currently do that an API like this would enable for us.

Just so you know, we will provide malware protection for apps like we do
for websites. And yes, that could include blocking apps we don't
distribute ourselves (see bug 863669).

Also, a couple of other points:
- You want to rely on an API that can only be exercised if the user
visits the marketplace. Unfortunately you have no guarantee that this
will happen, and we'll still check for updates at the install-time
manifest url.
- We want to support scenarii where the user browses the marketplace
from a desktop computer and from there installs an app on device. We
have a prototype doing that with the push API. How does your feature
profile work in this case?

By the way, can you point us to a description of these profiles?

Fabrice
--
Fabrice Desré
b2g team
Mozilla Corporation

[Matt Basta]

> We have a prototype doing that with the push API. How does your feature
> profile work in this case?

The feature profile is added at the time that mozApps.install[Package] is
triggered. If this is being done at the platform level, then the feature
profile is moot because we can't add it. If it's being done from within
the Marketplace, our wrapper would handle this just fine.

FWIW, we're a privileged app now, so we can do our own push notifications
to do the send-to-phone feature. That's been on our roadmap for a while
now.

> By the way, can you point us to a description of these profiles?

We don't have formal docs, but you can look at the code in our repo:

https://github.com/mozilla/fireplace/blob/master/hearth/media/js/buckets.js
https://github.com/mozilla/zamboni/blob/master/mkt/constants/features.py
https://github.com/mozilla/zamboni/blob/master/mkt/webapps/models.py#L1305

It's basically a hex-encoded bit field where each bit represents the
presence of a feature. There's also a version number and a feature count.

My profile on desktop, for instance, is "3040fe.32.1"

----- Original Message -----
From: "Fabrice Desre" <fab...@mozilla.com>
To: "Matt Basta" <mba...@mozilla.com>
Cc: "Mounir Lamouri" <mou...@lamouri.fr>, dev-w...@lists.mozilla.org
Sent: Friday, May 24, 2013 9:18:50 AM
Subject: Re: API for editing existing app installations

[Travis Choma]

Interesting, so it's taking this profile associated with the device and matching it with a feature profile associated with the app, right? To complete the picture for me, what is the flow/process for creating the feature profile on the marketplace side that is associated with the app? Also how is it specified whether a feature is required vs optional in a given app?

[Matt Basta]

When an app is submitted, we (will soon) ask the developer which APIs/features their app absolutely requires, with the list pre-populated with smart guesses through the magic of static analysis. Developers can update the app profile when they upload a new version. Reviewers will also soon have tools to inspect the profile.

We then can calculate whether a device (based on which features it has) is compatible with an app (based on which features the developer says it requires).

> Also how is it specified whether a feature is required vs optional in a given app?

We don't track optional features. If a feature isn't required for the app to function properly, we don't track that since it doesn't break compatibility or change what we'd show to the user.

[Fabrice Desre]

On 05/24/2013 11:40 AM, Matt Basta wrote:
> When an app is submitted, we (will soon) ask the developer which APIs/features their app absolutely requires, with the list pre-populated with smart guesses through the magic of static analysis. Developers can update the app profile when they upload a new version. Reviewers will also soon have tools to inspect the profile.

But that's only for packaged apps, right? Hosted app can change their
required features at will without looping in the marketplace, so that
seems useless for them.

> We then can calculate whether a device (based on which features it has) is compatible with an app (based on which features the developer says it requires).

I don't see how this will work in multi-device or pushing from desktop
to device environments if you have no platform support.

[Matt Basta]

It's an optional step for hosted apps (it's optional for packaged apps as well). The goal is to help developers avoid users from installing their apps if it's not going to work on the users' devices. The app profile is honors system, and the incentive is to help users avoid a bad experience.

> I don't see how this will work in multi-device or pushing from desktop
> to device environments if you have no platform support.

I can't see how it will work with those features either, but I also haven't seen any of the work that the platform has done. We should probably have a conversation about how these features will work with our (and other) marketplaces.

----- Original Message -----
From: "Fabrice Desre" <fab...@mozilla.com>
To: "Matt Basta" <mba...@mozilla.com>
Cc: "Travis Choma" <tch...@mozilla.com>, dev-w...@lists.mozilla.org, "Mounir Lamouri" <mou...@lamouri.fr>
Sent: Friday, May 24, 2013 11:46:25 AM
Subject: Re: API for editing existing app installations

[Mounir Lamouri]

On 24/05/13 16:34, Matt Basta wrote:
>> Doesn't the manifest contain the list of required feature?
>
> A very old version of the spec contains a "requiredFeatures" field, but it was never fully defined and isn't supported, and no values for the field were specified.

Why was that abandoned? Also, it is being worked on in the specification
so I'm not sure of what "spec" you are speaking about.

>> I'm not sure I understand what you meant here. Do you have examples?
>
> If there's a new feature we want to add compatibility testing for that some devices might already have, we can't rely on the devices to implicitly know that they support the feature. If we decide, for instance, that we want to detect support for <input type="color">, the user agent would need to be able to say, "Oh, I know what <input type="color"> is and I do[n't] support it.".
>
> We already have this functionality in our feature profiles, so no sense in reinventing the wheel.

I believe the marketplace is reinventing the wheel here. That kind of
problems have already been solved in the web and things like input types
are being polyfiled.

Granted some websites are locking out some users that don't have a
particular set of features but <input type=foo> wasn't a good example
and I believe that for those high level features that can't be
polyfilled, we could use required_features in the manifest.

>> I do not think the marketplaces should be put in the centre of the
>> system because you can install hosted applications from anywhere so you
>> can't make the marketplaces owning the blacklist of applications.
>
> Right, and as I mentioned in my original email, a marketplace could only use this API for manifests *which it installed itself*. Relying on Mozilla to blacklist apps which were distributed by a third party who could otherwise do the blacklisting themselves seems very bad. If carriers start distributing their own branded marketplaces, making them go through us to flag and remove malware seems like a dangerous and slow process.

Why wouldn't carrier or third parties being able to plug their own
blacklist server in the device?

> Of course, but we're in a different boat than the web at large.

I strongly disagree.

--
Mounir

[Matt Basta]

> Why was that abandoned?

The first it was ever mentioned was 2012 and it's not supported by any of the devices that we're shipping. It's effectively a non-feature and won't be useful since it's not backwards compatible. Whether or not it becomes a standard is moot because it won't apply to the launch devices and that's what matters.

> I'm not sure of what "spec" you are speaking about.

http://www.w3.org/2012/sysapps/manifest/#required_features-member

Tracing the origin of the "required_features" field back, it first appears in 2012.

> I believe the marketplace is reinventing the wheel here. That kind of
> problems have already been solved in the web and things like input types
> are being polyfiled.

False. If an app needs to use getUserMedia to take pictures, that's supported on Firefox for Desktop but not FXOS. WebActivities are only available on FXOS (maybe Android?). If an app doesn't target desktop, it has to choose WebActivities. Any app that only implements one and not the other is physically broken on one of the two platforms. They could implement both, but then they can't sell their app since navigator.mozPay doesn't exist for desktop (or Android) yet.

There are plenty of features that an app could require that are "proprietary" Mozilla APIs that simply don't let apps fit into the web as we know it. All of the WebAPIs, WebActivities, codec support (!), and the flavors of WebRTC are just a few which make this issue something which simply can't be addressed by conventional means. The Marketplace's feature profile addresses this issue fully and completely.

Telling developers to use polyfills isn't sufficient because the apps already exist and the developers won't do it. We can't sell apps to people if their devices won't run the apps. Simple as that. If you pay for a music app, it damn well better play music or the user is going to be pissed. If the music app uses an API that's not supported on the platform they're using, it's our (i.e.: the Marketplace's) responsibility to not show them that app. By the time the user's device gets the manifest, it's too late to check whether their device supports the app because the user has already paid for it.

> Why wouldn't carrier or third parties being able to plug their own
> blacklist server in the device?

We shouldn't make it the responsibility of the carrier to know when the Firefox Marketplace blacklists an app. That would be like saying that you can run Norton Antivirus on your PC, but the definitions come from Intel.

----- Original Message -----
From: "Mounir Lamouri" <mou...@lamouri.fr>
To: dev-w...@lists.mozilla.org

Sent: Tuesday, May 28, 2013 6:11:00 AM
Subject: Re: API for editing existing app installations

[dbuc...@mozilla.com]

Matt is correct, you can't polyfill a feature that fundamentally does not exist given a missing hardware requirement it relies upon.

Secondly, this form of feature detection and profile comparison is flexible enough that other marketplaces and apps can generate their own profile signatures that are tailored to their needs. For instance:

If Nvidia wanted to create a Tegra Zone app that mitigated the installation of games intended for various levels of hardware and performance support, they could detect a profile, of their definition, on the user's device and show games that they know will run well on a device with those perf characteristics.

I have reviewed the solution Matt presents, and it makes the most sense of anything I have seen or thought of to solve this problem. Paired with this feature - https://bugzilla.mozilla.org/show_bug.cgi?id=873599 - we can make it even easier to generate a base profile.

This discussion is moving toward the region of "we should discuss this impasse face to face". Let's get back to addressing a bulleted list of outstanding concerns instead of circular, scatter shot mix of new and previously answered questions.

TO DO:

- Each of the skeptics should formulate single, deduped, bulleted list of their outstanding concerns here: https://etherpad.mozilla.org/app-profile-answers
- Matt will provide answers for each
- Ask follow-ups inline if need be

Let's be in the business of solutions folks.

[dbuc...@mozilla.com]

Here are the Bugzilla tickets relevant to this discussion:

- Manifest modification API: https://bugzilla.mozilla.org/show_bug.cgi?id=876984
- Device/environment capability API: https://bugzilla.mozilla.org/show_bug.cgi?id=873599

[jsmith....@gmail.com]

I could go into a lot detail responding to the above discussion, but at this point, I think we have to get people into a meeting to talk about this. There's misunderstandings evident on both sides of the argument that need to be addressed.

[dbuc...@mozilla.com]

On Friday, May 31, 2013 10:15:07 AM UTC-7, jsmith....@gmail.com wrote:
> I could go into a lot detail responding to the above discussion, but at this point, I think we have to get people into a meeting to talk about this. There's misunderstandings evident on both sides of the argument that need to be addressed.

Who shall I include on the meeting invite?

[jsmith....@gmail.com]

I think Caitlin was going to organize a meeting on this. People I would suggest including would be:

* Lawrence
* Yourself
* Mounir
* Matt
* Fabrice
* Travis
* Myself
* Jonas
* David

[Harald Kirschner]

I'd be happy to join to provide real-life scenarios and personas from the partner world if needed.

---
Harald Kirschner | Partner Engineer & Web Craftsman | har...@mozilla.com (mailto:hkirs...@mozilla.com)


El lunes, junio 3, 2013 a las 12:24 PM, jsmith....@gmail.com escribió:

> On Friday, May 31, 2013 4:54:45 PM UTC-7, dbuc...@mozilla.com (http://mozilla.com) wrote:

> > On Friday, May 31, 2013 10:15:07 AM UTC-7, jsmith....@gmail.com (http://gmail.com) wrote:
> >
> > > I could go into a lot detail responding to the above discussion, but at this point, I think we have to get people into a meeting to talk about this. There's misunderstandings evident on both sides of the argument that need to be addressed.
> >
> >
> >
> > Who shall I include on the meeting invite?
>
> I think Caitlin was going to organize a meeting on this. People I would suggest including would be:
>
> * Lawrence
> * Yourself
> * Mounir
> * Matt
> * Fabrice
> * Travis
> * Myself
> * Jonas
> * David

> _______________________________________________
> dev-webapps mailing list
> dev-w...@lists.mozilla.org (mailto:dev-w...@lists.mozilla.org)
> https://lists.mozilla.org/listinfo/dev-webapps
>
>