So I just blogged about where I'm at right now with Eideticker:
I talked to cjones a few weeks ago about where we should take things
next, and we agreed that the main priority should be bringing things
into a state such that it would be easy for other people to hack on
tests and the framework (so that work doesn't bottleneck on me or the
ateam). To that end, I've been spending quite a bit of time focusing on
creating a web interface for analysis, as well as streamlining the
process of writing tests. My current plan is to spend most of the rest
of the quarter continuing to do that, focusing mainly on the following
1. Getting the panda board working with the decklink capture card (see
blog post for a few more details on this).
2. Improving the capture analysis API to be more flexible/faster/more
3. Further improvements to the analysis web interface.
4. Better test infrastructure (the new harness is basically just a
loader for HTML pages-- we probably want something a bit more
sophisticated than this -- for instance supporting custom actions on
loaded web pages using Marionette).
Although my *focus* will be on the above, sometimes working through
actual user stories for eideticker can be helpful in developing the
framework. To that effect, we talked about a few areas to look at when
developing the framework. Here's where we're at with them:
> (1) investigate image diff issues
It seems there is simply some noise in the HDMI output. Strangely
enough, if I get fennec to just output a solid colour page, I sometimes
don't see this, however it seems to be always present in the tests that
I run. Not really sure what to do about this-- for now, I think we'll
just have to live with it in our post-processing step.
> (2) get some cool data from popular benchmarks
I spent some time looking into this, the fishtank demo I mention in my
blog post is obviously an example of this. Unfortunately the results
here are not super-interesting, mainly because most of these speed tests
seem to work poorly or not at all on fennec (both the XUL version and
the newer native version).
I'm feeling like it might be best to shelve this for now, at least until
performance on mobile improves a bit (I'll make a point of filing some
> (3) checkerboard analysis
Haven't looked into this yet, but I think it's an interesting area to
explore. I'd be interested in getting feedback from folks working
directly on this problem. Proofs of concept are great, but it would be
even better to be doing work which would actually be useful to people.
> (4) page load
Same as for (3)