info
Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
mobile random orange now has parity with desktop
14 views
Skip to first unread message
Brad Lassey
Oct 24, 2011, 3:43:09 PM10/24/11
to mozilla.de...@lists.mozilla.org, mozilla.dev.pl...@lists.mozilla.org
[cross posted to m.d.platform and m.d.platforms.mobile, please reply to m.d.tree-management]
tl;dr: Mobile random orange has been reduce to levels comparable to desktop. Now when you see a mobile orange, treat it like you would a desktop orange (backout, closed tree, etc)
Longer version:
Over the summer we decided to stop work on bringing up new test suites on mobile and instead concentrate on driving the number of random oranges down. The stated goal was to get to desktop parity (which we defined as 5% at the time, since that could obviously be a moving target).
Last week we decided to hide two test suite subsets, js reftests subset #1 and reftests subset #2 because the two combined to make up 57% of our oranges. With those two test suite subsets hidden, we've seen 4.46% orange over the last few days (mission accomplished). Please see Joel's blog post for more details [http://elvis314.wordpress.com/2011/10/24/android-orange-rate-is-4-46/].
That means a couple things. First, for all developers, you should start treating all unstarred mobile oranges as show stoppers. There are obviously still random oranges, just like there are on desktop, and they should be starred and investigated as usual.
Second, this unblocks bringing new tests into automation. The reason for the temporary moratorium is that it was hard to tell if the new test was giving us reliable results or not with all the noise. That should not be as big of a problem now.
Finally, for those two subsets that have been hidden, work continues to fix them such that they can be brought back to life. Since they were hidden and not disabled, we're still running them and can tell if they regress to become more orange in the mean time.
-Brad
tl;dr: Mobile random orange has been reduce to levels comparable to desktop. Now when you see a mobile orange, treat it like you would a desktop orange (backout, closed tree, etc)
Longer version:
Over the summer we decided to stop work on bringing up new test suites on mobile and instead concentrate on driving the number of random oranges down. The stated goal was to get to desktop parity (which we defined as 5% at the time, since that could obviously be a moving target).
Last week we decided to hide two test suite subsets, js reftests subset #1 and reftests subset #2 because the two combined to make up 57% of our oranges. With those two test suite subsets hidden, we've seen 4.46% orange over the last few days (mission accomplished). Please see Joel's blog post for more details [http://elvis314.wordpress.com/2011/10/24/android-orange-rate-is-4-46/].
That means a couple things. First, for all developers, you should start treating all unstarred mobile oranges as show stoppers. There are obviously still random oranges, just like there are on desktop, and they should be starred and investigated as usual.
Second, this unblocks bringing new tests into automation. The reason for the temporary moratorium is that it was hard to tell if the new test was giving us reliable results or not with all the noise. That should not be as big of a problem now.
Finally, for those two subsets that have been hidden, work continues to fix them such that they can be brought back to life. Since they were hidden and not disabled, we're still running them and can tell if they regress to become more orange in the mean time.
-Brad
0 new messages