Proposal: Turning off Eideticker perf testing

[William Lachance]

The Eideticker project (http://eideticker.mozilla.org) was started
almost 4 years ago, in an effort to better measure the performance of
Firefox for Android by directly capturing video of the browser
performing various actions. It later evolved to encompass testing
FirefoxOS performance. It's an interesting approach and has produced
useful results, especially during a period in 2012-2013 when we were
tuning Firefox for Android to reduce checkerboarding and start up time.

However, it hasn't received much maintenance over the last year, and
some parts of the code (especially those to do with bisection and
backfilling, which are critical to a useful performance harness) have
bitrotted and no longer work. People have (correctly) criticized the set
of tests that we run for lacking relevance: the content we use is old
and sometimes esoteric and the Galaxy Nexus hardware we use is seriously
out of date at this point. For these and other reasons, it's been over a
year since a regression has been filed based on Eideticker data that has
subsequently been addressed.

Given enough time and resources, these issues could be fixed, but we
just don't have them right now. My own attention has shifted to working
on Perfherder and Talos, which has a much larger scope and impact in
terms of the products we ship. We've made a fair bit of progress in this
project over the last couple of quarters (see the recent announcement on
48-hour backouts), but for that to continue it needs more of my attention.

Thus, I'd like to thus propose that we shut this system off for now. It
goes without saying that the code
(https://github.com/mozilla/eideticker) will live on, if we find out
that we want to reactivate the project at some future point that should
be totally possible.

Will

[Kartikaya Gupta]

FWIW one of the original driver behind Eideticker (tuning Fennec for
checkerboarding during scrolling) will become relevant again in the
next couple of months as we transition Fennec to using the C++ APZ
code and off the old Java pan/zoom code. While it would be nice to
have Eideticker around to give us data on performance characteristics
during this transition, I'm not sure if it would be worth the effort
to get it up and running again.

kats

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform

[William Lachance]

On 2015-08-17 7:21 PM, Kartikaya Gupta wrote:
> FWIW one of the original driver behind Eideticker (tuning Fennec for
> checkerboarding during scrolling) will become relevant again in the
> next couple of months as we transition Fennec to using the C++ APZ
> code and off the old Java pan/zoom code. While it would be nice to
> have Eideticker around to give us data on performance characteristics
> during this transition, I'm not sure if it would be worth the effort
> to get it up and running again.

It wouldn't be an enormous amount of effort to keep it running (it's
still going against m-c, as long as the device I'm using doesn't fall
over, which happens depressingly often): I guess the question is whether
the results it produced would really be relevant for you. As I said, the
tests that we run for checkerboarding are pretty out of date at this
point. Who really cares about a copy of CNN.com from mid 2012?

This is fixable, of course, but then the question becomes whether cycles
would be better spent on Eideticker or something else. It's nice to have
the video logs, etc. but at this point I'm feeling like synthetic
benchmarks like Talos or Autophone offer better bang for the buck,
especially now that they're hooked up to treeherder/perfherder and are
thus useable via try, etc.

I know that tcheck is still running in automation
(https://treeherder.mozilla.org/perf.html#/graphs?timerange=7776000&series=[mozilla-inbound,c233ba1133abbd544002dfbc29d9e63ced42a20e,1]),
maybe we could just expand on or modify that?

Will