recording use counters for web features

[Nathan Froyd]

Bug 968923, "Implement some equivalent of Chrome's use counters", has
landed on mozilla-central. We are now able to report statistics on whether
individual documents use a given:

- WebIDL method
- WebIDL attribute (getters and setters reported separately)
- CSS property
- operation listed in dom/base/nsDeprecatedOperationList.h

These statistics are reported through Telemetry.

Deprecated operations are handled automatically. CSS properties require
modifying dom/base/UseCounters.conf. WebIDL methods and attributes require
modifying dom/base/UseCounters.conf and adding a [UseCounter] attribute to
the appropriate entity in the corresponding WebIDL file. (Bug 1195959
tracks eliminating the WebIDL file modification, if you are curious.)

Please see dom/base/UseCounters.conf for more information, and come find
people in #content if you have questions.

Thanks,
-Nathan

[Bobby Holley]

This is awesome - thanks Nathan!

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>

[Benjamin Smedberg]

On 8/19/15 11:35 AM, Nathan Froyd wrote:
> These statistics are reported through Telemetry.

Have the in-tree docs been updated to document this? I don't recall
being asked to review the final data collection proposal for this mechanism.

In particular:

* Will this only collect to the opt-in (prerelease) population, or does
this also affect the release population?
* This involves collecting a numerator and denominator as separate
histograms, correct?
* Is there reporting in place for this mechanism already? If not, who is
going to write that?
* How are we going to do data-collection reviews for additions,
especially if the collection is added through WebIDL/UseCounters?
* Is there a mechanism for auto-expiring data collection like there is
for histograms?

--BDS

[Ehsan Akhgari]

On 2015-08-19 11:35 AM, Nathan Froyd wrote:
> Bug 968923, "Implement some equivalent of Chrome's use counters", has
> landed on mozilla-central.

Hurray! \o/

> - WebIDL method
> - WebIDL attribute (getters and setters reported separately)
> - CSS property
> - operation listed in dom/base/nsDeprecatedOperationList.h

Is there support for counting features that don't really fall into any
of the above categories? Basically counting things programatically?

> These statistics are reported through Telemetry.

Where does the data go? How can one view them?


Cheers,
Ehsan

[Patrick Brosset]

[Cc dev-developer-tools because this is relevant to devtools]

On Wed, Aug 19, 2015 at 5:35 PM, Nathan Froyd <nfr...@mozilla.com> wrote:

> Bug 968923, "Implement some equivalent of Chrome's use counters", has

> landed on mozilla-central. We are now able to report statistics on whether
> individual documents use a given:
>

> - WebIDL method
> - WebIDL attribute (getters and setters reported separately)
> - CSS property
>

Awesome! This is going to be useful for bug 1168246
<https://bugzilla.mozilla.org/show_bug.cgi?id=1168246> that aims at being
smart about ordering CSS properties when showing the auto-complete list in
the inspector.

> - operation listed in dom/base/nsDeprecatedOperationList.h
>

> These statistics are reported through Telemetry.
>

Could you give details about this please? How will we be able to access
this data?

[Nathan Froyd]

[Responding to the list this time...]

On Thu, Aug 20, 2015 at 4:05 AM, Patrick Brosset <pbro...@mozilla.com>
wrote:

> [Cc dev-developer-tools because this is relevant to devtools]
>
> On Wed, Aug 19, 2015 at 5:35 PM, Nathan Froyd <nfr...@mozilla.com> wrote:
>>
>> - CSS property
>>
>
> Awesome! This is going to be useful for bug 1168246
> <https://bugzilla.mozilla.org/show_bug.cgi?id=1168246> that aims at being
> smart about ordering CSS properties when showing the auto-complete list in
> the inspector.
>

It's worth noting that we don't collect information on very many CSS
properties (just enough to be able to write useful tests, at the moment...).

> These statistics are reported through Telemetry.
>>
>
> Could you give details about this please? How will we be able to access
> this data?
>

They are regular histograms, so you ought to be able to access them through
the Telemetry dashboards at http://telemetry.mozilla.org/ .

That being said, they are not available there yet, possibly due to the
slightly unorthodox way we generate the histograms. So you may have to
wait a little while we we get that sorted out.

I have plans to write a custom dashboard for the use counter histograms, so
that you can see them in the more understandable form "X% of documents use
$FEATURE"...but one thing at a time. :)

-Nathan

On Thu, Aug 20, 2015 at 4:05 AM, Patrick Brosset <pbro...@mozilla.com>
wrote:

[Nathan Froyd]

On Wed, Aug 19, 2015 at 7:49 PM, Ehsan Akhgari <ehsan....@gmail.com>
wrote:

> On 2015-08-19 11:35 AM, Nathan Froyd wrote:
>

>> - WebIDL method
>> - WebIDL attribute (getters and setters reported separately)
>> - CSS property

>> - operation listed in dom/base/nsDeprecatedOperationList.h
>>
>

> Is there support for counting features that don't really fall into any of
> the above categories? Basically counting things programatically?
>

This is bug 1175033, so no. :) It should not be terribly hard to write.

> These statistics are reported through Telemetry.
>>
>

> Where does the data go? How can one view them?
>

They are just normal boolean histograms, so all that infrastructure ought
to be sufficient for viewing them.

-Nathan

[Nathan Froyd]

On Wed, Aug 19, 2015 at 7:11 PM, Benjamin Smedberg <benj...@smedbergs.us>
wrote:

> On 8/19/15 11:35 AM, Nathan Froyd wrote:
>
>> These statistics are reported through Telemetry.
>>

> Have the in-tree docs been updated to document this? I don't recall being
> asked to review the final data collection proposal for this mechanism.
>

They have not. I didn't realize there were documents to update or that
adding new histograms required separate review.

> In particular:
>
> * Will this only collect to the opt-in (prerelease) population, or does
> this also affect the release population?
>

Since these are just histograms within Telemetry, I assume this affects
prerelease and release populations. (That was the intent, anyway; it's
possible I didn't flip the right switches for this to happen.)

* This involves collecting a numerator and denominator as separate
> histograms, correct?
>

I don't know what numerator and denominator refer to here, maybe # of
pages using feature and # of total pages, respectively?

For each defined use counter, there are two separate boolean histograms.
One describes the use of that thing for individual documents, and the other
describes the use of that thing for "top-level" pages: basically what we
think of as a "web page".

So, scenarios:

- Site X uses use counter Y. We report "used" (true) in both histograms.
- Site X does not use use counter Y. We report "unused" (false) in both
histograms.
- Site X has an iframe for site W. Site W uses use counter Y, but Site X
does not. We report one "used" and one "unused" in the individual
documents histogram and one "used" in the "top-level pages" histogram.

We went with two separate histograms because the second gives us a measure
of "how many sites would be affected by removing this feature" that we
couldn't get from the first one alone.

* Is there reporting in place for this mechanism already? If not, who is
> going to write that?
>

Since these are just histograms, reporting should not be a problem. There
have been problems on the server-side when the first part of this landed
back in May (bug 1168409), so it's possible that there are some changes
needed on the server side for histogram validation, etc.

* How are we going to do data-collection reviews for additions, especially
> if the collection is added through WebIDL/UseCounters?
>

I do not know. hg push hooks?

> * Is there a mechanism for auto-expiring data collection like there is for
> histograms?
>

The histograms are currently set to never expire. It is probably worth
rethinking that.

-Nathan

[Benjamin Smedberg]

On 8/20/2015 12:56 PM, Nathan Froyd wrote:
> On Wed, Aug 19, 2015 at 7:11 PM, Benjamin Smedberg
> <benj...@smedbergs.us <mailto:benj...@smedbergs.us>> wrote:
>
> On 8/19/15 11:35 AM, Nathan Froyd wrote:
>
> These statistics are reported through Telemetry.
>
> Have the in-tree docs been updated to document this? I don't
> recall being asked to review the final data collection proposal
> for this mechanism.
>
>
> They have not. I didn't realize there were documents to update or
> that adding new histograms required separate review.

Yes. The Firefox data stewards (myself and peers Vladan and Ally) review
all changes to Firefox data collection. See the policy doc at
https://wiki.mozilla.org/Firefox/Data_Collection

This really does need some docs. I don't particularly care whether this
ends up in the telemetry document tree at
http://mxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/docs/
or add a new area for DOM documentation. But we absolutely need to
describe the system and the histograms it produces.

I do think we should end up documenting the use counters in the
auto-generated docs, but that can be a followup bug. Bug 1194287 is
currently filed to get Histograms.json itself included in the
auto-generated docs.

>
> In particular:
>
> * Will this only collect to the opt-in (prerelease) population, or
> does this also affect the release population?
>
>
> Since these are just histograms within Telemetry, I assume this
> affects prerelease and release populations. (That was the intent,
> anyway; it's possible I didn't flip the right switches for this to
> happen.)

Will that solve all of the use-cases? I'm perfectly fine with being able
to specify that use counters are opt-out (FHR), but opt-in/telemetry is
the default in Histograms.json. In any case, this should be tested as
part of QA verification for the bug.

>
> * This involves collecting a numerator and denominator as separate
> histograms, correct?
>
>
> I don't know what numerator and denominator refer to here, maybe # of
> pages using feature and # of total pages, respectively?

Isn't that the use-case that this was trying to solve?

>
> For each defined use counter, there are two separate boolean
> histograms. One describes the use of that thing for individual
> documents, and the other describes the use of that thing for
> "top-level" pages: basically what we think of as a "web page".

Interesting, this is not how I imagined this would work. I figured we'd
have a couple of global counters:

* counter: number of documents
* counter: number of toplevel documents

And then for each feature we'd have a couple of counters:

* counter: per-document feature usage
* counter: per-toplevel-document feature usage

In terms of the total data collected, it appears to be equivalent. But
one reason for doing it this is that it can significantly reduce the
telemetry payload size for measuring infrequently-used features: we
don't include unused histograms at all in the telemetry payload.

>
> * Is there reporting in place for this mechanism already? If not,
> who is going to write that?
>
>
> Since these are just histograms, reporting should not be a problem.
> There have been problems on the server-side when the first part of
> this landed back in May (bug 1168409), so it's possible that there are
> some changes needed on the server side for histogram validation, etc.

Has somebody verified that the current histogram displays on
telemetry.mozilla.org can answer the questions you care about?

>
> * How are we going to do data-collection reviews for additions,
> especially if the collection is added through WebIDL/UseCounters?
>
>
> I do not know. hg push hooks?

Let's start out by just stating the obvious, that you cannot add a new
use counter without data collection review. If that stops working, we
can think about the more complicated hook process.

>
> * Is there a mechanism for auto-expiring data collection like
> there is for histograms?
>
>
> The histograms are currently set to never expire. It is probably
> worth rethinking that.

Yes, I think this is a requirement. Very few of the use-cases for use
counters are candidates for permanent data collection, and so we should
explicitly be able to specify the expiration.

--BDS

[Ehsan Akhgari]

On 2015-08-24 11:45 AM, Benjamin Smedberg wrote:
>> * How are we going to do data-collection reviews for additions,
>> especially if the collection is added through WebIDL/UseCounters?
>>
>>
>> I do not know. hg push hooks?
>
> Let's start out by just stating the obvious, that you cannot add a new
> use counter without data collection review. If that stops working, we
> can think about the more complicated hook process.

What specific things are you planning to review? Immediately, it's not
obvious to me at all that the Firefox data steward group is the right
group to review use counter collections based on the goals stated in
<https://wiki.mozilla.org/Firefox/Data_Collection>.

For example, reading the "Function" section under Data Collection
Properties there, I don't think individual use counters really map to
any of the questions asked there. For other concerns, such as privacy
or legal review, it seems that a one-time analysis of use counters is
sufficient, and reviewing individual use counters from those
perspectives may not make much sense.

There are however other questions that I think will be satisfied better
by requiring different review requirements (I would imagine that DOM
peer review would be suited for those), such as: whether it's necessary
and useful to collect a piece of data. For example, collecting use
counters on Document.getElementById() doesn't seem very useful since we
know that the API is in wide usage and we don't plan to change or remove
the API in the foreseeable future. I don't think that answering
questions like this is best achieved by the Data Collection
owners/peers, since the answer often requires a good understanding of
the API surface we expose and also the specific changes planned for the
future of the APIs in question.

Cheers,
Ehsan

[Jeff Griffiths]

This looks like it will be amazing. I would like to talk about
expanding css coverage as much as possible, as a followup to the
initial work.

On Thu, Aug 20, 2015 at 9:38 AM, Nathan Froyd <nfr...@mozilla.com> wrote:
> [Responding to the list this time...]
>
> On Thu, Aug 20, 2015 at 4:05 AM, Patrick Brosset <pbro...@mozilla.com>
> wrote:
>
>> [Cc dev-developer-tools because this is relevant to devtools]
>>
>> On Wed, Aug 19, 2015 at 5:35 PM, Nathan Froyd <nfr...@mozilla.com> wrote:
>>>
>>> - CSS property
>>>
>>
>> Awesome! This is going to be useful for bug 1168246
>> <https://bugzilla.mozilla.org/show_bug.cgi?id=1168246> that aims at being
>> smart about ordering CSS properties when showing the auto-complete list in
>> the inspector.
>>
>
> It's worth noting that we don't collect information on very many CSS
> properties (just enough to be able to write useful tests, at the moment...).
>
>

>> These statistics are reported through Telemetry.
>>>
>>

>> Could you give details about this please? How will we be able to access
>> this data?
>>
>
> They are regular histograms, so you ought to be able to access them through
> the Telemetry dashboards at http://telemetry.mozilla.org/ .
>
> That being said, they are not available there yet, possibly due to the
> slightly unorthodox way we generate the histograms. So you may have to
> wait a little while we we get that sorted out.
>
> I have plans to write a custom dashboard for the use counter histograms, so
> that you can see them in the more understandable form "X% of documents use
> $FEATURE"...but one thing at a time. :)
>
> -Nathan
>
> On Thu, Aug 20, 2015 at 4:05 AM, Patrick Brosset <pbro...@mozilla.com>
> wrote:
>
>> [Cc dev-developer-tools because this is relevant to devtools]
>>
>> On Wed, Aug 19, 2015 at 5:35 PM, Nathan Froyd <nfr...@mozilla.com> wrote:
>>
>>> Bug 968923, "Implement some equivalent of Chrome's use counters", has
>>> landed on mozilla-central. We are now able to report statistics on
>>> whether
>>> individual documents use a given:
>>>

>>> - WebIDL method
>>> - WebIDL attribute (getters and setters reported separately)
>>> - CSS property
>>>
>>

>> Awesome! This is going to be useful for bug 1168246
>> <https://bugzilla.mozilla.org/show_bug.cgi?id=1168246> that aims at being
>> smart about ordering CSS properties when showing the auto-complete list in
>> the inspector.
>>
>>

>>> - operation listed in dom/base/nsDeprecatedOperationList.h
>>>

>>> These statistics are reported through Telemetry.
>>>
>>

>> Could you give details about this please? How will we be able to access
>> this data?
>>
>>
>>>
>>> Deprecated operations are handled automatically. CSS properties require
>>> modifying dom/base/UseCounters.conf. WebIDL methods and attributes
>>> require
>>> modifying dom/base/UseCounters.conf and adding a [UseCounter] attribute to
>>> the appropriate entity in the corresponding WebIDL file. (Bug 1195959
>>> tracks eliminating the WebIDL file modification, if you are curious.)
>>>
>>> Please see dom/base/UseCounters.conf for more information, and come find
>>> people in #content if you have questions.
>>>
>>> Thanks,
>>> -Nathan
>>> _______________________________________________
>>> dev-platform mailing list
>>> dev-pl...@lists.mozilla.org
>>> https://lists.mozilla.org/listinfo/dev-platform
>>>
>>
>>

> _______________________________________________
> dev-developer-tools mailing list
> dev-devel...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-developer-tools

[Karl Dubost]

Nathan,

Wonderful news! About:

Le 20 août 2015 à 00:35, Nathan Froyd <nfr...@mozilla.com> a écrit :
> Please see dom/base/UseCounters.conf for more information, and come find
> people in #content if you have questions.

Is there a plan for a step by step guide on how to

1. implement a new counter
2. access the data once the counter is in place
3. make sense of the stats once it has ran for a while

on MDN, wiki.mozilla.org or hacks.mozilla.org?


--
Karl Dubost, Mozilla
http://www.la-grange.net/karl/moz

[Josh Matthews]

Servo is particularly interested in getting usage data for just about
every DOM API that exists in Gecko. We'd like to use this to inform our
priorities for implementing missing features in Servo. Is this a
realistic request? If not, what about instrumenting all SVG APIs to
figure out if there's a subset that would give us maximum compatibility?
How much of a burden will these requests be on developers and users?

Cheers,
Josh

[Nathan Froyd]

On Mon, Aug 31, 2015 at 1:11 PM, Josh Matthews <jo...@joshmatthews.net>
wrote:

> Servo is particularly interested in getting usage data for just about
> every DOM API that exists in Gecko. We'd like to use this to inform our
> priorities for implementing missing features in Servo. Is this a realistic
> request? If not, what about instrumenting all SVG APIs to figure out if
> there's a subset that would give us maximum compatibility? How much of a
> burden will these requests be on developers and users?

That'd be a lot of extra use counters! I think you'd have to file a bug
and/or ask folks on the DOM team how much of a performance hit they're
willing to take by recording use counters for everything. Instrumenting
all the SVG APIs sounds sort of reasonable to me...

I'm not sure how to interpret "How much of a burden will these requests be
on developers and users?" If you're asking about performance penalties,
then yes, there is a performance penalty associated with them...

Thanks,
-Nathan

[Nathan Froyd]

On Mon, Aug 24, 2015 at 8:40 PM, Karl Dubost <kdu...@mozilla.com> wrote:

> Is there a plan for a step by step guide on how to
>
> 1. implement a new counter
>

I plan on writing some documentation for
https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Adding_a_new_Telemetry_probe
very soon.

> 2. access the data once the counter is in place

3. make sense of the stats once it has ran for a while
>

Yes. We're still tweaking the way use counters are recorded, which will
have a bearing on how you access them and make sense of them. I'm hoping
to write a custom dashboard that takes the raw histogram data and it in a
more meaningful way (e.g. % of documents that use a given feature, rather
than counts of use/don't use).

Thanks,
-Nathan