Stability of telemetry metrics

[Bartosz Wiklak]

Hi, I wonder how is the stability of benchmark metrics is assured. Is it safe to assume that when we run the same benchmark on successive chromium checkouts results are always comparable i.e. if we see a change in metrics one can be sure that's a change in product being benchmarked not change in telemetry or storyset? 

[Caleb Rouleau]

If you run the same story on successive chromium checkouts then the results should be comparable. However, if you run the same benchmark and use some average of the available stories, then that average could shift since stories are frequently added or removed (especially from system_health benchmarks).

I hope this helps!

[Bartosz Wiklak]

Thanks, is there any formal check for that? I can imagine that some change in core components can have an affect on metrics.

[Christopher C]