UI Telemetry Dashboards

[J. Ryan Stinnett]

Mark Finkle recently blogged[2] about Fennec's UI telemetry system[1],
which looks like a quite a powerful tool for understand how their
features are used. For example, the "Settings" tab tracks how many
users have flipped each option, and I was able to see what percent of
users have enabled USB and WiFi debugging for Fennec.

Brian also linked to a similar system for desktop Firefox[3] in a
recent meeting.

We've talked several times about having something similar for
DevTools, but it would take some time to add probes everywhere and
build a way to understand the data. It does seem like it would be
hugely valuable for prioritizing future time spent on features, etc.

Thoughts?

[1]: http://people.mozilla.org/~mfinkle/uitelemetry/
[2]: http://starkravingfinkle.org/blog/2015/10/fun-with-telemetry-url-suggestions/
[3]: https://useradvocacy.mozilla.org/dashboards/telemetry-ui/ (LDAP required)

- Ryan

[mratc...@mozilla.com]

I have suggested this before and would be happy to work on it. In a nutshell they use something like:
Telemetry.sendUIEvent(TelemetryContract.Event.ACTION, TelemetryContract.Method.MENU, itemId);

It would be really easy for us to listen for click events and act on the button id to automatically create detailed telemetry entries for each tool.

The reason people have said no in the past is because of privacy but we could easily make it opt-in. One advantage of having our own system would be that we could gather stats for release versions.

[Joe Walker]

What questions do we have that need the data gathered by this?
Joe.

> _______________________________________________
> dev-developer-tools mailing list
> dev-devel...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-developer-tools
>

[J. Ryan Stinnett]

On Tue, Oct 6, 2015 at 1:20 PM, Joe Walker <jwa...@mozilla.com> wrote:
> What questions do we have that need the data gathered by this?

The main questions I am after are "does anyone use feature X?". This
can help guide decisions on the priority of marketing, adding related
functionality, and maintenance work for features. In the vein of
"Great or Dead", it may also lead us to remove things no one uses.

Today, our main data collection is:

* Was an entire panel opened ever by a user?
* How many times is an entire panel opened?
* How much time is spent in an entire panel?

We could be collecting much more specific data. We have so many
buttons, options, menu items, tabs, modes etc. which are specific to
particular features.

To pick a random example, the Debugger has an options menu with
off-by-default things like "Auto Prettify Minified Sources". I have
currently have no idea how many people use that. If it breaks, it's
hard to say how important that is relative to everything else.

This data can help guide decisions like this. I am not saying it
determines the answer, there are always many variables to consider,
but I think helps to have some data as opposed to none.

- Ryan

[Dylan Barrell]

As an independent developer of dev tools plugins, I am also interested
in this sort of data. It would be of great value to have a telemetry
feature I could use that would deliver data into my dashboard on
addons.mozilla.org

--Dylan

[Joe Walker]

On Tue, Oct 6, 2015 at 9:48 PM J. Ryan Stinnett <jry...@gmail.com> wrote:

> On Tue, Oct 6, 2015 at 1:20 PM, Joe Walker <jwa...@mozilla.com> wrote:
> > What questions do we have that need the data gathered by this?
>
> The main questions I am after are "does anyone use feature X?". This
> can help guide decisions on the priority of marketing, adding related
> functionality, and maintenance work for features. In the vein of
> "Great or Dead", it may also lead us to remove things no one uses.
>

But when G||D was announced, we talked about removing things that are not
used and didn't come up with anything.

Maybe Jeff has more of an idea how often he's needed more detailed data,
but in general I think we should have still more concrete questions for
which we need more data before we gather it. Otherwise we spend time
collecting data that we probably wont use and risk that when we need it we
find that the data isn't quite measuring the right thing.

Today, our main data collection is:
>
> * Was an entire panel opened ever by a user?
> * How many times is an entire panel opened?
> * How much time is spent in an entire panel?
>
> We could be collecting much more specific data. We have so many
> buttons, options, menu items, tabs, modes etc. which are specific to
> particular features.
>
> To pick a random example, the Debugger has an options menu with
> off-by-default things like "Auto Prettify Minified Sources". I have
> currently have no idea how many people use that. If it breaks, it's
> hard to say how important that is relative to everything else.
>
> This data can help guide decisions like this. I am not saying it
> determines the answer, there are always many variables to consider,
> but I think helps to have some data as opposed to none.
>

>From an overall cost point of view we pay for the data in further delays to
getting read-write to the storage panel (assuming Mike does it), and also
some measurable code bloat if every action grows by 3-5 lines.

Hence - time-limited telemetry to answer a specific need sounds great, but
lets be specific about the actions we'll take as a result of the data we're
gathering. Also lets consult Jeff.

Joe.

[Jordan Santell]

We recently added telemetry[0] to specific performance tool features and
potential pain points.

* All "recording options" used -- there are many recording options (record
framerate, memory, display JIT info, record allocations) and some work
better than others. If we can find out if a lot of people switch their
defaults to something, or never using a feature, then that could be a
candidate to focus on or cut.
* Multiple recordings is unique to our tools' timeline, and makes the UI
code signficantly more complex, as well as takes up valuable real estate.
If our users are only really using one recording at a time, this is
something we could remove.
* Importing/exporting profiles -- a simple feature, but ties into the
multiple recordings a bit.
* Length of recordings -- there are very different use cases for a 5 second
recording and a 5 minute recording. We should know what to focus on.
* `console.profile` recordings are a huge source of complexity for the
tool. While not something we can/want to remove, if no one is using this,
then we can focus on less chaotic bugs.

These were added not too long ago, so haven't seen the data yet, but
hopefully this'll inform of us of things we can get rid of and focus on.

[0]
https://github.com/mozilla/gecko-dev/blob/2e4ec377a390f58b170ede75a6f1100e9efcd2a8/toolkit/components/telemetry/Histograms.json#L7465-L7505

> _______________________________________________
> dev-developer-tools mailing list
> dev-devel...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-developer-tools
>

--
Jordan Santell
Sr. Software Engineer | DevTools Performance & Memory

[Mike Ratcliffe]

FWIW: I am with Joe on this... I don't see any value in knowing every
click. Choosing the exact data that we want is a far better in my opinion.

/Mike

> <mailto:dev-devel...@lists.mozilla.org>