Re: bugs I referenced for adding a new ping:

[Matthew N.]

+autofill +rweiss

Side note: Let's try to move discussions about individual probes (especially P1s) to their specific bugs now since we have them in a few other places (PRD and some other docs). Let's keep the "User Story" field in the bugs with an up-to-date description of what exactly we will measure. We have the meta bug and here is a view of bugs under it: https://mnoorenberghe.github.io/bz-dependency-buglist/?list=1333344

From talking to Steve yesterday, I realized we still need some clarity on what exactly we will measure for each probe and given that I wonder if we should work with Rebecca Weiss to get her input on it. Many of the probes have the question of whether we want to record an event (e.g. when state changes) or a scalar (one value per session) and I think her experience could be useful in deciding between them. Due to timezones maybe it would be easiest to discuss with Rebecca over email?

Rebecca, would you prefer a meeting to discuss how to measure the success of Form Autofill or would email+bugzilla work? Most of the team is in Taipei btw. If you prefer a live meeting and late afternoon or evening meetings don't work for you then perhaps just the two of us can meet?

Thanks,

Matthew

On Sun, Mar 12, 2017 at 9:08 PM, Joe Cheng <jch...@mozilla.com> wrote:

These are the P1 metrics we said we will implement. I don't think we are collecting anything out of the ordinary but we can talk further about it during our weekly call.

I'd like to also talk about how we can collect just enough data so we know how good/bad the local heuristics is doing

• [P1] [Form Autofill] Collect information on how many people opt-out of Form Autofill

• [P1] [Form Autofill] Collect information on how many people autofill forms

• [P1] [Form Autofill] Collect information on how many people submit forms

• Can be covered by "how much time users spent on page with forms" below

• [P1] [Form Autofill] Collect information on how much time users spent on page with forms (with and without form autofill). - before 56

• [P1] [Form Autofill - Payment] Collect information on how many people create a credit card profile

• [P1] [Form Autofill - Payment] Collect information on how many people use credit card autofill
  

On Sat, Mar 11, 2017 at 2:37 AM, Vance Chen <vc...@mozilla.com> wrote:

​<snip>

Option A: Review before landing

If the team already knows exactly what it wants to collect, and it knows how to analyze the data coming in, then only need a simple review before land the patch  - your patch modifies the histograms/scalars files or the data documentation, and the data peers review that change.

Option B: assisted process

If your team needs help figuring out how to collect data, or how to analyze it, or the data you're collecting has any "special" privacy characteristics, there is a more detailed "data kickoff" process that we use to help you get started and make sure you have the help you need. Rebecca Weiss has been getting this process started. This process is better for teams that have complex questions, want analyst support, or haven't done this before and want extra help. There is still a data review at the end before the code lands, but the guided process helps warn about "gotchas" that tend to trip teams up.

I thought we are going to go with Option B but now it seems like it is not necessarily so. So as Matt suggested, let's decide first how we going to add new probes from a system add-on and then we can decide which review option better suit our needs.

Thanks

Vance

On Sat, Mar 11, 2017 at 2:15 AM, Matthew N. <Ma...@mozilla.com> wrote:

I think the first thing to figure out is what we discussed in Taipei: How will we add new probes from a system add-on? It would be great to get that discussion going (perhaps on our telemetry meta bug).

I guess the approval you're talking about is [1]

If we end up using the existing infrastructure and "main" ping (specify the probes in mozilla-central). Then the data collection review is something that is normally done when the patch is written. If we implement our own ping then we will need a larger review on that first and then adding scalars/histograms to it should be easy.

[1] https://wiki.mozilla.org/Firefox/Data_Collection#Requesting_Approval