General observation: two years ago we had big "classes" of compat
issues (UA sniffing for FxOS, -webkit- prefixes for mobile content
generally). Subsequent development sorted out much of those problems -
today issues tend to be more specific and detailed, and less easy to
classify into big piles or find by automated tools. I'm still
interested in how we can push the Compatipede tool forward and make it
useful also to guarantee against compat *regressions*, particularly
when we want to test the waters with new features such as ES-future or
new CSS.
Some things that would be interesting to me on the automation frontier
to work on in London:
* Compatipede-2 plug-ins: the ones we have (demo), how to write them (workshop)
* Looking at the DOM, HTTP, CSS and JS: what are interesting data
points to record for comparison and tracking over time? (Discussion -
good to do before that workshop - we'd write plug-ins to track new
data points).
* Matching/diffing error messages from Gecko and WebKit - can we write
some code that can give us a meaningful diff and tell us what errors
happened in engine A but not in B? (new Compatipede analysis module?)
* What should our approach be to automating *interactivity* for
automated tests, such as automated log-in? (discussion, hack session)
* We can create some spin-off statistics for purposes that are not
strictly compat but "nearby" like memory usage. How, and what's
important? Memory usage? Can we measure jank? Scroll performance?
* Explore working on the proposed WebExtensions-based Firefox/Chrome
add-on to control the browser on behalf of Compatipede, to move off
SlimerJS and Phantom (there's some initial code somewhere on GitHub -
a Boar branch I think)
* Should we aim for testing on real devices? If yes, how do we bridge
the gap between Compatipede and the Firefox instance on the device?
Marionette (whenever it comes to Fennec..)? WebExtensions (whenever it
comes to Fennec..)? Nodejs-Marionette (testing on FxOS devices)? A
custom-written devtools protocol client? Something else?
Other things (non-Compatipede):
* Usability testing of our various "report a site problem" features
(aiming to streamline as much as possible). I suggest we list or set
up couple of pages that have known bugs/brokenness, and challenge
random all-hands participants to load URLs and report the problems,
then observe them using either extensions, the firefox:input feature
or the
webcompat.com form.
* Better filtering UI on
webcompat.com is very important.. I'll keep
nagging about
https://github.com/webcompat/webcompat.com/issues/795
until we get it right.
* Devtools JS debugger QA - how can we contribute more, since we
probably use this tool more intensively than most?
On Wed, Mar 23, 2016 at 4:37 PM, Mike Taylor <
mi...@mozilla.com> wrote:
> With the London all-hands approaching, let's brainstorm some things we'd
> like to discuss as a team, or across teams.
>
> Feel free to reply to this thread with ideas and we can put them in the wiki
> when the thread dies down.
>
> Discussion topics:
>
> * Measuring compatibility (and progress)
> * Structured Triage (bugzilla and
webcompat.com)
> *
webcompat.com development conventions -- what works and what can be
> improved
> * Training contributors to triage and do outreach
>
> Hack projects:
>
> * Moving bug stage labels to milestones for
webcompat.com
>
> --
> Mike Taylor
> Web Compat, Mozilla