Define Stability and Performance measurements on Daily builds

[Tony Chung]

Hi,

Our current smoketests do not rigorously define the quality of performance and stability of the builds. They mainly focus on functional regressions. Given the builds have are still crashy, lagging performance, and reboots randomly, please share your thoughts on how can improve the build quality.

* What factors define stability to you? How would you measure it? (there is crash reporting, but the crash signatures are not hooked up to socorro yet)
* What factors define performance to you? How would you measure it? (eg. Number of phone reboots? App startup time?)

Note: The current smoketests do track the # of reboots and # of crashes within a testrun.

If there are tools and tips that you know would help, please do share.

Thanks,

Tony
Mozilla QA

[Josh Carpenter]

Hi Tony,

Very glad that you've started this thread! This is a top priority for UX between now and Jan 15. We're very eager to do what we can to help improve performance. The finest drop shadows in the world aren't worth much if they draw at 7 fps.

> * What factors define performance to you? How would you measure it? (eg. Number of phone reboots? App startup time?)

* Frames per second (ideally we achieve some degree of consistency throughout the UX, even if it's sub-60fps)
* Responsiveness (the speed of the interface's response to user inputs. Ideally we prioritize and respond to user inputs over system events)
* Core user stories (these are high priority areas. Both load time and transition FPS are important):
** Switch to Home screen
** Switch to Task Manager
** Unlock device from Lock screen
** Trigger an activity (both window and inline dispositions)
** Open an app
** Scrolling a list

That's a very quick list. The UX team will spend some more time on this and follow up…

Thanks!

—
Josh Carpenter
UX Designer, FxOS
Mozilla

> _______________________________________________
> dev-gaia mailing list
> dev-...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-gaia

[onecy...@gmail.com]

Performance and Stability:

Is anyone working on this on the gaia or platform side.

I remember with fennec there was a bit of work to get both perf and stability tests up and running, and quantifiable.

I think this topic would be worth carving out some meeting time to discuss what we could do in the short / medium term, and definitely something that we track / triage as part of the v1 delivery.



-David

[onecy...@gmail.com]

Performance and Stability:

Is anyone working on this on the gaia or platform side.

I remember with fennec there was a bit of work to get both perf and stability tests up and running, and quantifiable.

I think this topic would be worth carving out some meeting time to discuss what we could do in the short / medium term, and definitely something that we track / triage as part of the v1 delivery.



-David



On Tuesday, October 9, 2012 11:54:49 PM UTC-7, Josh Carpenter wrote:

[Andrew Overholt]

On Wed 10 Oct 2012 08:03:52 PM EDT, onecy...@gmail.com wrote:
> Performance and Stability:

Operation Slim Fast is relevant:

https://bugzilla.mozilla.org/show_bug.cgi?id=797189

Andrew

[David Clarke]

Operation Slim Fast looks to be a good place to start on the performance end of things.

I spoke with jlebar on irc, and there is now a tool in mozilla central that spits out a memory usage report.

Do we have a thought process going how we will take this data, graph, and monitor ?

I think ctalbert, dhunt, have been involved in tasks similar in the past, is any of that infrastructure reusable.

Andrew: are you interested in pulling together a meeting next week to address some of these things / find owners ?

How would we like to proceed ?


-David

[Andrew Overholt]

> Do we have a thought process going how we will take this data, graph, and monitor ?
>
> I think ctalbert, dhunt, have been involved in tasks similar in the past, is any of that infrastructure reusable.
>
> Andrew: are you interested in pulling together a meeting next week to address some of these things / find owners ?

I'm sorry for taking so long to get back to you on this. Since I
obviously suck, it would be outstanding if you or someone else wanted
to spearhead this, David.

Andrew

[Martijn]

On Tue, Oct 23, 2012 at 11:30 PM, Andrew Overholt <over...@mozilla.com> wrote:
>> Andrew: are you interested in pulling together a meeting next week to
>> address some of these things / find owners ?
>
>
> I'm sorry for taking so long to get back to you on this. Since I obviously
> suck, it would be outstanding if you or someone else wanted to spearhead
> this, David.

I think David already put out a meeting out for Thursday, 25 october.
Under here, the details.

Regards,
Martijn

On Mon, Oct 22, 2012 at 6:58 PM, David Clarke <dcl...@mozilla.com> wrote:

The following is a new meeting request:

Subject: B2G V1 Performance Checkpoint
Organizer: "David Clarke" <dcl...@mozilla.com>

Location: "VidyoRoom - QA Ext 245" <vidy...@mozilla.com>; "SFO
Commons" <sfo-c...@mozilla.com>
Resources: "VidyoRoom - QA Ext 245" <vidy...@mozilla.com>; "SFO
Commons" <sfo-c...@mozilla.com>
Time: Thursday, October 25, 2012, 1:00:00 PM - 2:30:00 PM GMT
-08:00 US/Canada Pacific

Purpose:
Generate a working definition of how we want to improve the
performance on B2G for V1 ship date.

1) Discuss
2) Find Owners
3) Setup Tracking Bugs
4) How do we solve, type of questions ?
How to measure device performance ?
How do we measure stability ?
How do we report preformance issues ?
How do we... etc..etc


QA Vidyo room

Dial-in: conference# 9245
US/International: +1 650 903 0800 x92 Conf# 9245
US toll free: +1 800 707 2533 (pin 369) Conf# 9245
Canada: +1 416 848 3114 x92 Conf# 9245

Join irc.mozilla.org #webapi for back channel


Gaia:
Areas of focus
Tracking bug: ???
Gecko:
Platform specific issues.
Tracking Bug: Operation Slimfast
Tools:
Performance Measurement
Memory Management
Tracking Bug: ???
QA:
Stability of Builds
Results monitoring
Crash Reporter


_______________________________________________
Qa-b2g mailing list
Qa-...@mozilla.org
https://mail.mozilla.org/listinfo/qa-b2g




--
Martijn Wargers - Help Mozilla!
http://quality.mozilla.org/
http://wiki.mozilla.org/Mozilla_QA_Community
irc://irc.mozilla.org/qa - /nick mw22

> Andrew
>
> _______________________________________________
> dev-b2g mailing list
> dev...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-b2g



--
Martijn Wargers - Help Mozilla!
http://quality.mozilla.org/
http://wiki.mozilla.org/Mozilla_QA_Community
irc://irc.mozilla.org/qa - /nick mw22