Benchmark ideas

[Malini Das]

Hey all,

We (the automation and tools team) are interested in getting an idea of
the kinds of benchmarks you'd like to measure for B2G.

We currently have efforts to get performance metrics
(https://bugzilla.mozilla.org/show_bug.cgi?id=744902, but data gathering
has yet to be hooked up) and eideticker support for b2g
(http://www.youtube.com/watch?v=lrmIkdO4s0o) so we can do video analysis
and get things like fps, but we'd like to know what else you'd like to
measure, and what kind of tests you'd like to see so we can tackle these
problems early on.

What kind of benchmarks would you like to see? How do you envision
comparing multiple phones?

--
- Malini Das

[Malini Das]

Hah, wrong link for eideticker tracking bug:
https://bugzilla.mozilla.org/show_bug.cgi?id=781178

Well, at least you now know one of my preferred genres.

--
- Malini Das

[Jonas Sicking]

On Thu, Aug 23, 2012 at 1:12 PM, Malini Das <md...@mozilla.com> wrote:
> Hey all,
>
> We (the automation and tools team) are interested in getting an idea of the
> kinds of benchmarks you'd like to measure for B2G.
>
> We currently have efforts to get performance metrics
> (https://bugzilla.mozilla.org/show_bug.cgi?id=744902, but data gathering has
> yet to be hooked up) and eideticker support for b2g
> (http://www.youtube.com/watch?v=lrmIkdO4s0o) so we can do video analysis and
> get things like fps, but we'd like to know what else you'd like to measure,
> and what kind of tests you'd like to see so we can tackle these problems
> early on.
>
> What kind of benchmarks would you like to see? How do you envision comparing
> multiple phones?

One thing that I think would be great to measure is the startup time
for various apps. This is likely similar to the talos page-load tests
that we have.

While we should start with the most critical built-in apps (phone,
sms, browser), we should also do 3rd party apps, both packaged and
hosted ones.

/ Jonas

[Marco]

mdas於 2012年8月24日星期五UTC+8上午4時12分51秒寫道：

Some brand name companies may ask for the performance of file reading/writing on different file sizes and file numbers.

[Michael Treese]

Batteries , batteries, batteries...

Most cell phones provide 'Talk time' and 'Stand by time' estimates, but to my knowledge there are no industry standard benchmarks for battery life. How about a couple of profiles that are closer to real life:

1. Make a 3 min. phone call, wait 3 min. and make another - repeat until the battery is done. (time could varry between 3 and 30 min.)
2. Make a 10 min. phone call, send 10 SMS messages, take 10 pictures, play 10 videos - repeat until the battery is done.
3. Test Battery life with other applications - Browser, FM Radio, FB, etc.

It would be great to know how many amps an applications consumes during normal usage.

-Michael


----- Original Message -----

From: "Malini Das" <md...@mozilla.com>
To: mozilla...@lists.mozilla.org
Sent: Thursday, August 23, 2012 1:46:20 PM
Subject: Re: [b2g] Benchmark ideas

On 12-08-23 4:12 PM, Malini Das wrote:

> Hey all,
>
> We (the automation and tools team) are interested in getting an idea of
> the kinds of benchmarks you'd like to measure for B2G.
>
> We currently have efforts to get performance metrics
> (https://bugzilla.mozilla.org/show_bug.cgi?id=744902, but data gathering
> has yet to be hooked up) and eideticker support for b2g
> (http://www.youtube.com/watch?v=lrmIkdO4s0o) so we can do video analysis
> and get things like fps, but we'd like to know what else you'd like to
> measure, and what kind of tests you'd like to see so we can tackle these
> problems early on.
>
> What kind of benchmarks would you like to see? How do you envision
> comparing multiple phones?
>

Hah, wrong link for eideticker tracking bug:
https://bugzilla.mozilla.org/show_bug.cgi?id=781178

Well, at least you now know one of my preferred genres.

--
- Malini Das

_______________________________________________
dev-b2g mailing list
dev...@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-b2g

[Andreas Gal]

I think much of this is OEM testing, and we shouldn't waste too much cycles on that. What we should focus on is software battery consumption. The starting point for that is definitely standby time. Next is videos and apps. Active Radio use is for the OEM to optimize. We should focus on the parts we can actually fix (standby, software stack).

Andreas

[Andrew Overholt]

Can we make any use of PowerTOP?

http://www.lesswatts.org/projects/powertop/

Andrew

[Alex Keybl]

>> What kind of benchmarks would you like to see?

Do you have a list of benchmarks that are already planned? Things that immediately come to mind (and you've probably already though of): startup/shutdown time, cold app launch time, warm app launch time (?), WebGL tech demo framerates, JS perf tests, dropped frames in video playback, memory usage all the while, and app eviction frequency (?).

>> How do you envision comparing multiple phones?

That's a really interesting question if I'm understanding you correctly. Will we ever be comparing B2G app performance to the same apps running in mobile browsers, or native apps? I'm not sure if we'd compare a WebGL demo to native OpenGL demos on Blackberry (for instance) or to the same WebGL demo in the Blackberry browser.

-Alex

On Aug 23, 2012, at 1:46 PM, Malini Das <md...@mozilla.com> wrote:

> On 12-08-23 4:12 PM, Malini Das wrote:

[Kevin Hu]

How about CPU usage and memory usage?
I think we have a similar Bugzilla case before: https://bugzilla.mozilla.org/show_bug.cgi?id=756379

Cheers,




Best regards,
Kevin Hu

----- Original Message -----
From: "Andreas Gal" <g...@mozilla.com>
To: "Michael Treese" <mtr...@mozilla.com>
Cc: "Malini Das" <md...@mozilla.com>, mozilla...@lists.mozilla.org
Sent: 2012年8月24日 星期五 12:47:17
Subject: Re: [b2g] Benchmark ideas


I think much of this is OEM testing, and we shouldn't waste too much cycles on that. What we should focus on is software battery consumption. The starting point for that is definitely standby time. Next is videos and apps. Active Radio use is for the OEM to optimize. We should focus on the parts we can actually fix (standby, software stack).

Andreas

On Aug 24, 2012, at 12:42 PM, Michael Treese <mtr...@mozilla.com> wrote:

> Batteries , batteries, batteries...
>
> Most cell phones provide 'Talk time' and 'Stand by time' estimates, but to my knowledge there are no industry standard benchmarks for battery life. How about a couple of profiles that are closer to real life:
>
> 1. Make a 3 min. phone call, wait 3 min. and make another - repeat until the battery is done. (time could varry between 3 and 30 min.)
> 2. Make a 10 min. phone call, send 10 SMS messages, take 10 pictures, play 10 videos - repeat until the battery is done.
> 3. Test Battery life with other applications - Browser, FM Radio, FB, etc.
>
> It would be great to know how many amps an applications consumes during normal usage.
>
> -Michael
>
>
> ----- Original Message -----
>
> From: "Malini Das" <md...@mozilla.com>
> To: mozilla...@lists.mozilla.org
> Sent: Thursday, August 23, 2012 1:46:20 PM
> Subject: Re: [b2g] Benchmark ideas
>

[Malini Das]

On 12-08-24 9:15 PM, Alex Keybl wrote:
>>> What kind of benchmarks would you like to see?
>
> Do you have a list of benchmarks that are already planned? Things that immediately come to mind (and you've probably already though of): startup/shutdown time, cold app launch time, warm app launch time (?), WebGL tech demo framerates, JS perf tests, dropped frames in video playback, memory usage all the while, and app eviction frequency (?).
>

For eideticker tests, we'll be doing the same tests that are being run
against fennec, ie: scroll and zoom fps and this can be used as benchmarks.

JS perf tests can be written now, but haven't been set up yet. I'm
prioritizing getting the eideticker tests done before I hook up the JS
perf tests. These JS tests can do things like measure app start up time
or anything Gaia facing.

>>> How do you envision comparing multiple phones?
>
> That's a really interesting question if I'm understanding you correctly. Will we ever be comparing B2G app performance to the same apps running in mobile browsers, or native apps? I'm not sure if we'd compare a WebGL demo to native OpenGL demos on Blackberry (for instance) or to the same WebGL demo in the Blackberry browser.
>

My question is more along the lines of "What comparisons would you like
to see between B2G and other available phones?" We have android phones
available for testing, so we can do things like load a webgl page in
both and check performance, but I'd like to get some suggestions on what
kind of tests would be valuable.

> -Alex
>
> On Aug 23, 2012, at 1:46 PM, Malini Das <md...@mozilla.com> wrote:
>
>> On 12-08-23 4:12 PM, Malini Das wrote:
>>> Hey all,
>>>
>>> We (the automation and tools team) are interested in getting an idea of
>>> the kinds of benchmarks you'd like to measure for B2G.
>>>
>>> We currently have efforts to get performance metrics
>>> (https://bugzilla.mozilla.org/show_bug.cgi?id=744902, but data gathering
>>> has yet to be hooked up) and eideticker support for b2g
>>> (http://www.youtube.com/watch?v=lrmIkdO4s0o) so we can do video analysis
>>> and get things like fps, but we'd like to know what else you'd like to
>>> measure, and what kind of tests you'd like to see so we can tackle these
>>> problems early on.
>>>
>>> What kind of benchmarks would you like to see? How do you envision
>>> comparing multiple phones?
>>>
>>
>> Hah, wrong link for eideticker tracking bug: https://bugzilla.mozilla.org/show_bug.cgi?id=781178
>>
>> Well, at least you now know one of my preferred genres.
>>
>> --
>> - Malini Das
>> _______________________________________________
>> dev-b2g mailing list
>> dev...@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-b2g
>

--
- Malini Das

[Malini Das]

On 12-08-23 4:12 PM, Malini Das wrote:

FYI, I'm gathering benchmark ideas here:
https://wiki.mozilla.org/Auto-tools/Projects/B2G_Test_And_Performance#Benchmarks

--
- Malini Das

[Malini Das]

Interesting! I'll take a look at using this / discuss this at the next
b2g automation meeting.

--
- Malini Das

[Rob Hawkes]

>From a games perspective, I'd love for the ability to pull the FPS count (from the built-in pink meter we already have) using JavaScript.

Right now, I have to either write down the FPS count as seen by eye (which is long and hard when you're trying to 'play' at the same time), or build my own FPS counter (which is a little hard when you're testing games you didn't make).

Rob

[Malini Das]

On 12-08-29 5:30 AM, Rob Hawkes wrote:
>>From a games perspective, I'd love for the ability to pull the FPS count (from the built-in pink meter we already have) using JavaScript.
>
> Right now, I have to either write down the FPS count as seen by eye (which is long and hard when you're trying to 'play' at the same time), or build my own FPS counter (which is a little hard when you're testing games you didn't make).
>
> Rob
>

Actually, project eideticker will help with that. We get the actual fps
from video capture. I'm working on getting support for it on a b2g
pandaboard. You can follow
https://bugzilla.mozilla.org/show_bug.cgi?id=781178 for progress.

Do you have a particular game in mind, or test in mind? Right now, we're
running the same tests as eideticker for fennec, but I'd like to have an
idea of what tests you'd like to see for b2g specifically.

>
> On Thursday, 23 August 2012 21:12:51 UTC+1, mdas wrote:
>> Hey all,
>>
>>
>>
>> We (the automation and tools team) are interested in getting an idea of
>>
>> the kinds of benchmarks you'd like to measure for B2G.
>>
>>
>>
>> We currently have efforts to get performance metrics
>>
>> (https://bugzilla.mozilla.org/show_bug.cgi?id=744902, but data gathering
>>
>> has yet to be hooked up) and eideticker support for b2g
>>
>> (http://www.youtube.com/watch?v=lrmIkdO4s0o) so we can do video analysis
>>
>> and get things like fps, but we'd like to know what else you'd like to
>>
>> measure, and what kind of tests you'd like to see so we can tackle these
>>
>> problems early on.
>>
>>
>>
>> What kind of benchmarks would you like to see? How do you envision
>>
>> comparing multiple phones?
>>
>>
>>
>> --
>>
>> - Malini Das
>

--
- Malini Das