Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
2014-10-02 Performance Acceptance
9 views
Skip to first unread message
Geo Mealer
Oct 7, 2014, 2:39:45 PM10/7/14
to dev-...@lists.mozilla.org
The 2.1 acceptance run for performance has been completed using the Oct 02 build. The automated run itself took over 24 hours to complete because of the large result set, not including the original baselining run for 2.0. The collected results were closely analyzed to ensure that the numbers are solid.
The results, with graphs and comments as well as methodology and data, can be found at:
https://wiki.mozilla.org/B2G/QA/2014-10-02_Performance_Acceptance
Pass/Fail is based on the Startup->Visually Complete test, using the release acceptance criteria at https://wiki.mozilla.org/FirefoxOS/Performance/Release_Acceptance. This can be summarized as “median startup time to stable UI in under 1000 ms”.
Exec Summary:
Passed:
* Contacts
* Dialer
* Email
* FM Radio
* Video
With the exception of Video, all of these apps did show at least a small regression in startup time from 2.0, but remain under 1000 ms.
Failed:
* Calendar
* Camera
* Clock
* Gallery
* Music
* Settings
* SMS
All of these applications were over the 1000 ms guideline. SMS and Settings have improved times from 2.0 to 2.1, the latter vastly. Music had no 2.0 results (the visually-complete test doesn’t report in that branch) but the 2.1 results are over 1000 ms. The others have regressed at least slightly from 2.0. It’s worth noting that Music, Gallery and Clock are very close to 1000 ms.
Cost Control testing did not complete due to an environmental issue, and will be followed up separately. Current Datazilla results suggest that it will almost certainly be over 1000 ms and that it has regressed significantly from 2.0.
I’ll file “release acceptance" bugs for the failed apps for consideration for 2.1 blocking and/or discussion of whether a goal should be adjusted case-by-case.
One last note:
Apps may have changed where the “visually complete" event occurred between 2.0 and 2.1, if they were not correctly emitting it at the right time in 2.0. In this case a result may be worse by the numbers, but the actual startup behavior is either unchanged or better from 2.0. I’ll leave this up to individual app teams to call out in their bugs.
Geo
The results, with graphs and comments as well as methodology and data, can be found at:
https://wiki.mozilla.org/B2G/QA/2014-10-02_Performance_Acceptance
Pass/Fail is based on the Startup->Visually Complete test, using the release acceptance criteria at https://wiki.mozilla.org/FirefoxOS/Performance/Release_Acceptance. This can be summarized as “median startup time to stable UI in under 1000 ms”.
Exec Summary:
Passed:
* Contacts
* Dialer
* FM Radio
* Video
With the exception of Video, all of these apps did show at least a small regression in startup time from 2.0, but remain under 1000 ms.
Failed:
* Calendar
* Camera
* Clock
* Gallery
* Music
* Settings
* SMS
All of these applications were over the 1000 ms guideline. SMS and Settings have improved times from 2.0 to 2.1, the latter vastly. Music had no 2.0 results (the visually-complete test doesn’t report in that branch) but the 2.1 results are over 1000 ms. The others have regressed at least slightly from 2.0. It’s worth noting that Music, Gallery and Clock are very close to 1000 ms.
Cost Control testing did not complete due to an environmental issue, and will be followed up separately. Current Datazilla results suggest that it will almost certainly be over 1000 ms and that it has regressed significantly from 2.0.
I’ll file “release acceptance" bugs for the failed apps for consideration for 2.1 blocking and/or discussion of whether a goal should be adjusted case-by-case.
One last note:
Apps may have changed where the “visually complete" event occurred between 2.0 and 2.1, if they were not correctly emitting it at the right time in 2.0. In this case a result may be worse by the numbers, but the actual startup behavior is either unchanged or better from 2.0. I’ll leave this up to individual app teams to call out in their bugs.
Geo
0 new messages