Finch alert: Study RemoveNonStandardAppearanceValue - heartbeat metrics differences first seen on 2023-08-11

[finch...@google.com]

Chirp thinks your experiment is having an effect.
Analysis based on 3 days' data from 2023-08-09 to 2023-08-11

EventLatency.GestureScrollUpdate.Touchscreen.TotalLatency (0.5 quantile)

Platform	Experiments	Quantile	Change	Effect size	Samples (approx.)	Test
M117 on Mac (DEV)	Enabled vs Control	50	⇘10094 (12437 +/- 3824) microseconds	big	88 k	T-test with jackknife resampling

---

Additional remarks.

Some subpopulations were not tested since one or more group in each of them were either completely missing or had too few data points.

Problems with the metric EventLatency.GestureScrollUpdate.Touchscreen.TotalLatency.

Message	Platform	Experiments	Samples (approx.)	Test
Observed significant change of EventLatency.GestureScrollUpdate.Touchscreen.TotalLatency in N=1 subpopulations. Technical problems (insufficient counts) prevented testing EventLatency.GestureScrollUpdate.Touchscreen.TotalLatency in N=1 subpopulations. Groups defined in gcl_studies/RemoveNonStandardAppearanceValue.gcl?plan.CANARY_DEV_50.canary_dev. Test run: T-test with jackknife resampling.	M118 on Win (CANARY)	Enabled	nan	T-test with jackknife resampling

Statistical significance: p=1.67e-03 or lower across all results.

You can update this incident to associate it with a bug or snooze it.

Feedback?

---

More about "missing data" warnings

• Why they are emitted: The offending metric is reported far less frequently for one of field trial's groups than it is for the others. It may be that the associated branch of code is not entered or that it is not instrumented.
• How to fix the problem: If the situation is unexpected, you should examine the code path of the offending group and add necessary telemetry code. If the reporting works as intended, you can either (1) ignore the problem (other groups report just fine and you want to keep watching this) or (2) consider removing the histogram from the relevant Finch config file.

Terminology explained...

• Samples (approx.): Approximate number of observations used in the test. Note that usually each client contributes many observations each day.
• Change: How much the observed variable changed between groups in absolute terms. The median and deviation of the entire population are given in parentheses for reference.
• Effect size: It is a relative measure of the strength of the relationship between group assignment (the independent variable here) and the metric we look at (the dependent variable).
• Test: The statistical hypothesis test used.
• Warning! Chirp will not alert again for a week unless something changes.