Telemetry opt-in for WebRT

[Lawrence Mandel]

Note: Reposted from dev-privacy as the archive isn't working over there just yet. Please respond in dev-planning

It is desirable to enable Telemetry in the Apps Web runtime (WebRT) so that we can measure in the same way that we do in Firefox. I would like to solicit input on the method that should be used to prompt users to opt-in. (Desktop bug 758232, Fennec bug 766094, see the end of this post for B2G). Here are the existing proposals:

1. Prompt within each app. I *think* this is what will happen by default as each app runs in its own profile and therefore requires its own opt-in prompt as a result of the way the prompting mechanism is currently structured.

2. Inherit the Firefox preference. This means that when a user opts-in to Telemetry on Firefox they have also opted-in to Telemetry in the WebRT for all apps.

3. Ask for opt-in on app installation. There are considerations if this is to be done on the marketplace side such as the change to the single click install flow. There is also an issue in that apps can be installed directly without going through the market. This solution can also be on the client side, although that solution may be similar to #1.

Some concerns that I have:
#1, #3 - These methods will require more prompts, which I think will likely translate into a lower opt-in rate.
#2 - I don't know that it will be clear to users that opting in to Telemetry in Firefox opts them in for WebRT as well.

Thoughts? Ideas?

A note on B2G: B2G is different than desktop and Android. On B2G we will want to collect Telemetry for the entire platform. B2G contains an setup/first run wizard in which an opt-in question can be added while the user is configuring their phone for the first time.

Lawrence

[Gervase Markham]

On 28/06/12 02:34, Lawrence Mandel wrote:
> It is desirable to enable Telemetry in the Apps Web runtime (WebRT)
> so that we can measure in the same way that we do in Firefox.

Do you mean in the runtime itself, or in the apps running on the runtime?

This is important because if it's the latter, app authors may have
concerns about us collecting stats about the internal state of their apps.

If it's the former, then IMO it's a part of Firefox, and it's OK for the
Firefox preference to govern.

> A note on B2G: B2G is different than desktop and Android. On B2G we
> will want to collect Telemetry for the entire platform. B2G contains
> an setup/first run wizard in which an opt-in question can be added
> while the user is configuring their phone for the first time.

Same question as above for B2G?

Gerv

[Gavin Sharp]

On Thu, Jun 28, 2012 at 12:38 AM, Gervase Markham <ge...@mozilla.org> wrote:
> Do you mean in the runtime itself, or in the apps running on the runtime?
>
> This is important because if it's the latter, app authors may have
> concerns about us collecting stats about the internal state of their apps.
>
> If it's the former, then IMO it's a part of Firefox, and it's OK for the
> Firefox preference to govern.

The simple answer is that it's the former - we'd get the same
information that we get from Firefox (though it obviously wouldn't
include the Firefox front-end telemetry information, just Gecko data).
Given this, I agree with you that we should just inherit the Firefox
pref - this distinction seems unlikely to be one that matters enough
to expose to users.

But there's a more complicated answer: given the nature of the web
runtime, the majority of the pages loaded in it are going to be from a
single origin (the given web app's), so in theory we could collect
app-specific performance characteristics (e.g. how Gecko performs for
a specific web app). That assumes we'd have some way to tie the data
to a specific app, rather than collecting data from all web app run
time instances (and thus all apps run in them) into a single bucket. I
don't know offhand whether the telemetry reports we'd get would allow
us to determine which app was in use for a given report (either
directly or indirectly).

Gavin

[Lawrence Mandel]

> On 28/06/12 02:34, Lawrence Mandel wrote:
> > It is desirable to enable Telemetry in the Apps Web runtime (WebRT)
> > so that we can measure in the same way that we do in Firefox.
>
> Do you mean in the runtime itself, or in the apps running on the
> runtime?

The measurement that I'm talking about is for WebRT itself. However, another question is whether we can/should include an app id in each Telemetry ping so that we can filter WebRT measurements based on app id. (This will allow us to see how WebRT functions with a given app.) Note that sending the app id would be a change from our current practice of not sending any information about the site/app that you are using during your Firefox session.

Providing app authors the ability to measure the performance of their apps would be a logical next step. However, I think this is a secondary conversation to have once we work out the logistics of enabling Telemetry on WebRT.

>
> This is important because if it's the latter, app authors may have
> concerns about us collecting stats about the internal state of their
> apps.
>
> If it's the former, then IMO it's a part of Firefox, and it's OK for
> the
> Firefox preference to govern.
>
> > A note on B2G: B2G is different than desktop and Android. On B2G we
> > will want to collect Telemetry for the entire platform. B2G
> > contains
> > an setup/first run wizard in which an opt-in question can be added
> > while the user is configuring their phone for the first time.
>
> Same question as above for B2G?

The goal at this point is the same on all platforms - to measure the Mozilla pieces. We do not have anything in place today to measure the content from a specific site/app.

Lawrence

[Lawrence Mandel]

> On Thu, Jun 28, 2012 at 12:38 AM, Gervase Markham <ge...@mozilla.org>
> wrote:

> > Do you mean in the runtime itself, or in the apps running on the
> > runtime?
> >

> > This is important because if it's the latter, app authors may have
> > concerns about us collecting stats about the internal state of
> > their apps.
> >
> > If it's the former, then IMO it's a part of Firefox, and it's OK
> > for the
> > Firefox preference to govern.
>

> The simple answer is that it's the former - we'd get the same
> information that we get from Firefox (though it obviously wouldn't
> include the Firefox front-end telemetry information, just Gecko
> data).
> Given this, I agree with you that we should just inherit the Firefox
> pref - this distinction seems unlikely to be one that matters enough
> to expose to users.
>
> But there's a more complicated answer: given the nature of the web
> runtime, the majority of the pages loaded in it are going to be from
> a
> single origin (the given web app's), so in theory we could collect
> app-specific performance characteristics (e.g. how Gecko performs for
> a specific web app). That assumes we'd have some way to tie the data
> to a specific app, rather than collecting data from all web app run
> time instances (and thus all apps run in them) into a single bucket.
> I
> don't know offhand whether the telemetry reports we'd get would allow
> us to determine which app was in use for a given report (either
> directly or indirectly).

The Telemetry reports do not have this capability today but it is a capability that we can add.

Reporting on the individual app level may be useful for app authors as well as this will provide them with real world data that they can use to performance tune their apps.

Lawrence

[Gervase Markham]

On 28/06/12 17:16, Lawrence Mandel wrote:
> The measurement that I'm talking about is for WebRT itself. However,
> another question is whether we can/should include an app id in each
> Telemetry ping so that we can filter WebRT measurements based on app
> id. (This will allow us to see how WebRT functions with a given app.)

I think "Mozilla gets told what apps you are using" is something which
would definitely require a specific opt-in.

It's an interesting question about whether the app author could be the
one to opt in. After all, they could do telemetry and analysis in their
own code if they chose, perhaps without informing the user.

But we definitely need an opt-in from someone. If it's the user, perhaps
it could be for all apps. But perhaps app authors could opt in on a
per-app basis.

Gerv

[Justin Dolske]

On 6/28/12 8:58 AM, Gervase Markham wrote:
> On 28/06/12 17:16, Lawrence Mandel wrote:
>> The measurement that I'm talking about is for WebRT itself. However,
>> another question is whether we can/should include an app id in each
>> Telemetry ping so that we can filter WebRT measurements based on app
>> id. (This will allow us to see how WebRT functions with a given app.)
>
> I think "Mozilla gets told what apps you are using" is something which
> would definitely require a specific opt-in.

That might be, but I think it's an assumption worth challenging (in the
intellectual, where-is-the-line really sense)!

So far as privacy invasions go it's pretty minor. I presume all the
other major/popular "app" platforms are doing similar things to some
degree (and likely in a more brazen fashion, given their histories).

I would submit that it's well withing the realm of consideration, as
long as we're (1) careful and responsible in what we collect (as
Telemetry is today in Firefox), and (2) we have have an appropriate
privacy policy in place (saying that it's all aggregated, and there's no
per-user tracking). Also, Web Apps are basically a new thing with new
(and likely often unformed) expectations.

I'd be in favor of going with opt-out. If we miss something and find
that it's actually a serious privacy issue, it's easy enough to
immediately disable it and build something opt-in.

Justin

[Gervase Markham]

On 03/07/12 03:18, Justin Dolske wrote:
> That might be, but I think it's an assumption worth challenging (in the
> intellectual, where-is-the-line really sense)!

Sure :-)

> So far as privacy invasions go it's pretty minor. I presume all the
> other major/popular "app" platforms are doing similar things to some
> degree (and likely in a more brazen fashion, given their histories).

Well, other platforms have app stores, and obviously they know what
you've downloaded. On iPhone, therefore, Apple knows pretty much every
app everyone has installed (except developers). On Android, though, one
can install 3rd party apps, perhaps via another store. Does Google get
sent details of those? (Genuine question; I don't know.)

And even if they do know what apps you have installed, they don't know
which ones you use and for how long (which the telemetry data would tell
us).

> I would submit that it's well withing the realm of consideration, as
> long as we're (1) careful and responsible in what we collect (as
> Telemetry is today in Firefox), and (2) we have have an appropriate
> privacy policy in place (saying that it's all aggregated, and there's no
> per-user tracking).

Aggregated on a per-app basis? Surely it would need to be more
fine-grained than that; you can't lump together telemetry measurements
from wildly disparate hardware. And what if you have a few outliers
where performance is poor, and want to look at them individually in more
detail?

If such restrictions are part of the deal we make when we turn this on,
we need to remember that, and not then loosen them later because the
data is tempting.

> Also, Web Apps are basically a new thing with new
> (and likely often unformed) expectations.

I'm not sure that's an argument for either side. If what we are
proposing is not a good thing for other reasons, then the idea that we
may be able to get users to swallow it because they have unformed
expectations is not reassuring! :-)

Gerv