Project Candle bug triage (20th September 2015)

[Nicholas Nethercote]

Greetings,

This is Project Candle bug triage email. The process used is described at
https://wiki.mozilla.org/Performance/Project_Candle#Bug_triage_process.

At the time of writing there are 57 unprioritized Power bugs, viewable at
http://mzl.la/1JjDIMy. In this thread I am discussing 15 of them: the most
recent 10, and 5 older ones that I feel are interesting and worth discussing
now.


* https://bugzilla.mozilla.org/show_bug.cgi?id=915725
* Baseline power draw measurements of Firefox for Android and Chrome on Nexus 4

Measuring power consumption on Android is both important and difficult.

I suggest P1, though I don't know if Nexus 4 is still the right choice. We
should probably also have bugs for regular power measurements on other
platforms to detect regressions.


* https://bugzilla.mozilla.org/show_bug.cgi?id=962573
* (power) [meta] Firefox uses significant energy

An old meta-bug for rvitillo's power investigations last year.

I suggest closing it, because the "[Power]" whiteboard annotations have
subsumed it.


* https://bugzilla.mozilla.org/show_bug.cgi?id=968123
* Hidden CSS background gif (completely covered by opaque child <img>)
activates the refresh driver

The difference with Chrome and Safari on the attached test case isn't that
high.

I suggest P3.


* https://bugzilla.mozilla.org/show_bug.cgi?id=971269
* High power consumption when using the HTML5 player on Youtube.

We have bug 1195790 open for YouTube on Mac; it's marked as Power:P1. This bug
seems to be about all platforms and has data for Windows and Mac.

I'm inclined to dup this bug to the other or vice versa, but I'm not certain.


* https://bugzilla.mozilla.org/show_bug.cgi?id=979119
* (powah) [meta][project] Firefox for Android power consumption

This is similar to meta-bug 962573 (above) but is focused just on Android. Some
of the blockers are in Android-specific components, but some are in core. So
both closing it and keeping it open would be reasonable.

mfinkle, rnewman: thoughts?


* https://bugzilla.mozilla.org/show_bug.cgi?id=1038752
* High energy usage on Huffington Post (edit)

Comment 3 has measurements on Mac showing we're a lot worse than Safari and
Chrome; comment 4 suggests it's due to painting.

I suggest P1 because HuffPost is the #33 site in the USA according to Alexa.


* https://bugzilla.mozilla.org/show_bug.cgi?id=1052137
* [Keyboard] About 5% higher CPU load on Flame device when keyboard is shown

I suggest P3 because the report is vague.


* https://bugzilla.mozilla.org/show_bug.cgi?id=1052142
* [Clock] About 50% CPU load on Flame device when using the stopwatch

Comments 4 and 5 have some explanation and a link to a fix for a similar bug.

I suggest P2.


* https://bugzilla.mozilla.org/show_bug.cgi?id=1053943
* Firefox consumes more CPU than Chrome(?)

This one is very vague; there's no particular workload, and it was filed in
response to a blog post.

Although doing ok vs. Chrome is obviously important, I suggest we close this as
INCOMPLETE because it is too vague and broad to be useful. It's more useful to
have bugs about specific workloads.


* https://bugzilla.mozilla.org/show_bug.cgi?id=1085554
* Fennec logs nonstop messages about "IdleService: DailyCallback running"

This one is a bit of a mess. It seems to be bad when it happens, but it looks
like it might only happen while charging the phone?

I suggest P2, but mfinkle or rnewman may have more to say about it.


* https://bugzilla.mozilla.org/show_bug.cgi?id=1105509
* OMTA-able animations not throttled for offscreen elements

This seems reasonably important, but may end up being subsumed by bug 1166500.

I suggest P2.


* https://bugzilla.mozilla.org/show_bug.cgi?id=1112701
* CSS animations cause many wakeups in Diablo sub-reddit

I suggest P2, based on not very much at all.


* https://bugzilla.mozilla.org/show_bug.cgi?id=1124325
* simple colorflashing benchmark causes many wakeups

I closed this last week because it's a synthetic benchmark and I don't trust
synthetic benchmarks. But bgirard disagreed and reopened it.

So I suggest P3 for the abovementioned reason, but I'll be interested to hear
what others think.


* https://bugzilla.mozilla.org/show_bug.cgi?id=1166500
* [Power] Invisible CSS animations can still sometimes keep the refresh driver
active

This is a general optimization that will subsume at least three other Power
bugs, one of which is a P1. hiro is working on it.

Definitely a P1.


* https://bugzilla.mozilla.org/show_bug.cgi?id=1179535
* mouse movement during off-main-thread (OMT) animation causes unnecessary
repainting

I suggest P2 if it's only during mouse movement, but dbaron and mstange's
uncertain comments in the bug have me a little worried.

[Jonathan Hylands]

On Sun, Sep 20, 2015 at 10:59 PM, Nicholas Nethercote <n.neth...@gmail.com> wrote:

Greetings,

This is Project Candle bug triage email. The process used is described at
https://wiki.mozilla.org/Performance/Project_Candle#Bug_triage_process.

At the time of writing there are 57 unprioritized Power bugs, viewable at
http://mzl.la/1JjDIMy. In this thread I am discussing 15 of them: the most
recent 10, and 5 older ones that I feel are interesting and worth discussing
now.


* https://bugzilla.mozilla.org/show_bug.cgi?id=915725
* Baseline power draw measurements of Firefox for Android and Chrome on Nexus 4

Measuring power consumption on Android is both important and difficult.

I suggest P1, though I don't know if Nexus 4 is still the right choice. We
should probably also have bugs for regular power measurements on other
platforms to detect regressions.

I definitely agree that Nexus 4 is not the right choice. We can use the Z3C for this now, since it runs both, and I in fact have a harness on two Z3C's, one running Android, and one running FxOS.

Note that I have an automated system running on a Jenkins server, doing power tests on a Flame device every day using the latest nightly build. You can see the results here:

https://raptor.mozilla.org/dashboard/script/power?var-device=flame-kk&var-memory=1024&var-branch=master

It is very difficult to use any other phone for automated power testing, since the Flame is the only one we have that allows the USB charging to be software-switched. I have a new ammeter that might allow the Z3C to be used, but I have been waiting for the automation/QA people to get it working under Jenkins first before I can test it.

- Jon

[Richard Newman]

* https://bugzilla.mozilla.org/show_bug.cgi?id=915725
* Baseline power draw measurements of Firefox for Android and Chrome on Nexus 4

Measuring power consumption on Android is both important and difficult.

I suggest P1, though I don't know if Nexus 4 is still the right choice. We
should probably also have bugs for regular power measurements on other
platforms to detect regressions.

IIRC the Nexus 4 was chosen because it was the only hardware we were able to measure on. I think we'd all love to be able to measure on a modern device! mfinkle knows more than I do about this.

 

* https://bugzilla.mozilla.org/show_bug.cgi?id=979119
* (powah) [meta][project] Firefox for Android power consumption

This is similar to meta-bug 962573 (above) but is focused just on Android. Some
of the blockers are in Android-specific components, but some are in core. So
both closing it and keeping it open would be reasonable.

mfinkle, rnewman: thoughts?

I know we've had a lot of partner interest in our power consumption on Android, and we have services and implementations that don't exist on desktop, so I'd be happy to keep this open in order to track stuff that's particularly relevant. 

 

Although doing ok vs. Chrome is obviously important, I suggest we close this as
INCOMPLETE because it is too vague and broad to be useful. It's more useful to
have bugs about specific workloads.

+1 to that general approach.

 

* https://bugzilla.mozilla.org/show_bug.cgi?id=1085554
* Fennec logs nonstop messages about "IdleService: DailyCallback running"

This one is a bit of a mess. It seems to be bad when it happens, but it looks
like it might only happen while charging the phone?

I suggest P2, but mfinkle or rnewman may have more to say about it.

There's some history there that implies this being seen in the wild (Bug 762620). It's also possible that this'll either (a) stop your device from charging because it's drawing too much current (which I've seen!), or (b) cause the horrific data usage that we've seen in Bug 1022569, if that IdleService callback ends up touching the network.

This seems like the loose end of a fairly serious thread, so I think this is important enough to at least move towards a diagnosis before retriaging. P1, IMO.

[Nicholas Nethercote]

Hi,

I got feedback from three people (two on the list, one privately)
which was better than last week, though still not amazing.

On Sun, Sep 20, 2015 at 7:59 PM, Nicholas Nethercote
<n.neth...@gmail.com> wrote:
>
> * https://bugzilla.mozilla.org/show_bug.cgi?id=915725
> * Baseline power draw measurements of Firefox for Android and Chrome on Nexus 4
>
> Measuring power consumption on Android is both important and difficult.
>
> I suggest P1, though I don't know if Nexus 4 is still the right choice. We
> should probably also have bugs for regular power measurements on other
> platforms to detect regressions.

I did P1, and added some notes summarizing the discussion below.

> * https://bugzilla.mozilla.org/show_bug.cgi?id=962573
> * (power) [meta] Firefox uses significant energy
>
> An old meta-bug for rvitillo's power investigations last year.
>
> I suggest closing it, because the "[Power]" whiteboard annotations have
> subsumed it.

Closed.

> * https://bugzilla.mozilla.org/show_bug.cgi?id=968123
> * Hidden CSS background gif (completely covered by opaque child <img>)
> activates the refresh driver
>
> The difference with Chrome and Safari on the attached test case isn't that
> high.
>
> I suggest P3.

I did P3.

> * https://bugzilla.mozilla.org/show_bug.cgi?id=971269
> * High power consumption when using the HTML5 player on Youtube.
>
> We have bug 1195790 open for YouTube on Mac; it's marked as Power:P1. This bug
> seems to be about all platforms and has data for Windows and Mac.
>
> I'm inclined to dup this bug to the other or vice versa, but I'm not certain.

I made it Power:meta and made bug 1195790 a blocker. If we have bugs
for YouTube on other platforms they can also block this one.

> * https://bugzilla.mozilla.org/show_bug.cgi?id=979119
> * (powah) [meta][project] Firefox for Android power consumption
>
> This is similar to meta-bug 962573 (above) but is focused just on Android. Some
> of the blockers are in Android-specific components, but some are in core. So
> both closing it and keeping it open would be reasonable.
>
> mfinkle, rnewman: thoughts?

rnewman wanted to keep it open, so I made it Power:meta.

> * https://bugzilla.mozilla.org/show_bug.cgi?id=1038752
> * High energy usage on Huffington Post (edit)
>
> Comment 3 has measurements on Mac showing we're a lot worse than Safari and
> Chrome; comment 4 suggests it's due to painting.
>
> I suggest P1 because HuffPost is the #33 site in the USA according to Alexa.

I did P1.

> * https://bugzilla.mozilla.org/show_bug.cgi?id=1052137
> * [Keyboard] About 5% higher CPU load on Flame device when keyboard is shown
>
> I suggest P3 because the report is vague.

I did P3.

> * https://bugzilla.mozilla.org/show_bug.cgi?id=1052142
> * [Clock] About 50% CPU load on Flame device when using the stopwatch
>
> Comments 4 and 5 have some explanation and a link to a fix for a similar bug.
>
> I suggest P2.

I did P2.

> * https://bugzilla.mozilla.org/show_bug.cgi?id=1053943
> * Firefox consumes more CPU than Chrome(?)
>
> This one is very vague; there's no particular workload, and it was filed in
> response to a blog post.
>

> Although doing ok vs. Chrome is obviously important, I suggest we close this as
> INCOMPLETE because it is too vague and broad to be useful. It's more useful to
> have bugs about specific workloads.

I closed it.

> * https://bugzilla.mozilla.org/show_bug.cgi?id=1085554
> * Fennec logs nonstop messages about "IdleService: DailyCallback running"
>
> This one is a bit of a mess. It seems to be bad when it happens, but it looks
> like it might only happen while charging the phone?
>
> I suggest P2, but mfinkle or rnewman may have more to say about it.

I did P1 on rnewman's advice, and put some text from him into the bug.

> * https://bugzilla.mozilla.org/show_bug.cgi?id=1105509
> * OMTA-able animations not throttled for offscreen elements
>
> This seems reasonably important, but may end up being subsumed by bug 1166500.
>
> I suggest P2.

I did P2.

> * https://bugzilla.mozilla.org/show_bug.cgi?id=1112701
> * CSS animations cause many wakeups in Diablo sub-reddit
>
> I suggest P2, based on not very much at all.

I did P2.

> * https://bugzilla.mozilla.org/show_bug.cgi?id=1124325
> * simple colorflashing benchmark causes many wakeups
>
> I closed this last week because it's a synthetic benchmark and I don't trust
> synthetic benchmarks. But bgirard disagreed and reopened it.
>
> So I suggest P3 for the abovementioned reason, but I'll be interested to hear
> what others think.

I did P3.

> * https://bugzilla.mozilla.org/show_bug.cgi?id=1166500
> * [Power] Invisible CSS animations can still sometimes keep the refresh driver
> active
>
> This is a general optimization that will subsume at least three other Power
> bugs, one of which is a P1. hiro is working on it.
>
> Definitely a P1.

I did P1.

> * https://bugzilla.mozilla.org/show_bug.cgi?id=1179535
> * mouse movement during off-main-thread (OMT) animation causes unnecessary
> repainting
>
> I suggest P2 if it's only during mouse movement, but dbaron and mstange's
> uncertain comments in the bug have me a little worried.

I did P2.

[Benjamin Smedberg]

On 9/20/15 7:59 PM, Nicholas Nethercote wrote:
> I suggest P1, though I don't know if Nexus 4 is still the right choice. We
> should probably also have bugs for regular power measurements on other
> platforms to detect regressions.

Nexus 4 is not the right choice.

Galaxy S4 is what we should use for this. We have standardized on this
device at the suggestion of the FxAndroid product team as the most
popular and representative device in the Android market for the next
year at least. It's what we're using for the content perf project, and
we have ordered these devices and they should already be available in
the major offices. Talk to Vladan or Erin Lancaster if you need access
to these or want to order more.

>
> *https://bugzilla.mozilla.org/show_bug.cgi?id=971269

> * High power consumption when using the HTML5 player on Youtube.
>
> We have bug 1195790 open for YouTube on Mac; it's marked as Power:P1. This bug
> seems to be about all platforms and has data for Windows and Mac.

Windows is far more important than Mac in terms of product priority.

I don't think we should necessarily dup this until we know which
subsystems are causing the problem. If this is a generic cross-platform
problem clearly should be a dup, but if this is caused by graphics or
widget subsystems, it could well be platform-specific.

>
> *https://bugzilla.mozilla.org/show_bug.cgi?id=1038752

> * High energy usage on Huffington Post (edit)
>
> Comment 3 has measurements on Mac showing we're a lot worse than Safari and
> Chrome; comment 4 suggests it's due to painting.
>
> I suggest P1 because HuffPost is the #33 site in the USA according to Alexa.

Is there test engineering/QA support for this project yet? Can we as a
general rule try to reproduce each of these mac bugs on Windows?

>
> *https://bugzilla.mozilla.org/show_bug.cgi?id=1124325

> * simple colorflashing benchmark causes many wakeups
>
> I closed this last week because it's a synthetic benchmark and I don't trust
> synthetic benchmarks. But bgirard disagreed and reopened it.
>
> So I suggest P3 for the abovementioned reason, but I'll be interested to hear
> what others think.

Is there a specific circumstance we can detect? If fixing this is hard,
would measuring it with telemetry (as a use-counter?) be easy and give
us a sense of it?

--BDS

[L. David Baron]

On Sunday 2015-09-20 19:59 -0700, Nicholas Nethercote wrote:
> * https://bugzilla.mozilla.org/show_bug.cgi?id=1166500
> * [Power] Invisible CSS animations can still sometimes keep the refresh driver
> active
>
> This is a general optimization that will subsume at least three other Power
> bugs, one of which is a P1. hiro is working on it.
>
> Definitely a P1.

For what it's worth, I just updated the summary of this bug to
reflect something that's fixable in one bug. Reducing the refresh
driver rate, rather than the amount of work we're doing per tick, is
something that belongs in a separate bug (which I filed as 1207014).

New summaries are:

https://bugzilla.mozilla.org/show_bug.cgi?id=1166500
[Power] Invisible Paint-only CSS animations are still restyled every refresh driver tick

https://bugzilla.mozilla.org/show_bug.cgi?id=1207014
refresh driver should use longer timers when its observers don't need frequent updates

> * https://bugzilla.mozilla.org/show_bug.cgi?id=1179535
> * mouse movement during off-main-thread (OMT) animation causes unnecessary
> repainting
>
> I suggest P2 if it's only during mouse movement, but dbaron and mstange's
> uncertain comments in the bug have me a little worried.

I think that's probably reasonable for now, but if we keep seeing
similar issues with the layer activity timer come up, then I think
redesigning something becomes higher priority.

-David

--
𝄞 L. David Baron http://dbaron.org/ 𝄂
𝄢 Mozilla https://www.mozilla.org/ 𝄂
Before I built a wall I'd ask to know
What I was walling in or walling out,
And to whom I was like to give offense.
- Robert Frost, Mending Wall (1914)

[Ehsan Akhgari]

On 2015-09-25 1:23 AM, Benjamin Smedberg wrote:
> *https://bugzilla.mozilla.org/show_bug.cgi?id=1038752
>> * High energy usage on Huffington Post (edit)
>>
>> Comment 3 has measurements on Mac showing we're a lot worse than
>> Safari and
>> Chrome; comment 4 suggests it's due to painting.
>>
>> I suggest P1 because HuffPost is the #33 site in the USA according to
>> Alexa.
>
> Is there test engineering/QA support for this project yet? Can we as a
> general rule try to reproduce each of these mac bugs on Windows?

It would also be nice if we could use rapl on Windows as well.
<https://developer.mozilla.org/en-US/docs/Mozilla/Performance/tools_power_rapl>
says that this is not possible. Is there any way around that issue? Or
are we just stuck with only using Intel Power Gadget? Can we somehow
use the API it exposes in rapl?

[Aaron Klotz]

On 9/25/2015 7:59 AM, Ehsan Akhgari wrote:
>
> It would also be nice if we could use rapl on Windows as well.
> <https://developer.mozilla.org/en-US/docs/Mozilla/Performance/tools_power_rapl>
> says that this is not possible. Is there any way around that issue?
>

We could write a device driver to expose the MSRs to user mode, but we'd
need releng to sign the binaries.

[Thomas Daede]

On 09/24/2015 10:23 PM, Benjamin Smedberg wrote:
> Galaxy S4 is what we should use for this. We have standardized on this
> device at the suggestion of the FxAndroid product team as the most
> popular and representative device in the Android market for the next
> year at least. It's what we're using for the content perf project, and
> we have ordered these devices and they should already be available in
> the major offices. Talk to Vladan or Erin Lancaster if you need access
> to these or want to order more.

The device also has a removable battery, which makes it easier to get
reliable power measurements.

But it also has an AMOLED screen, which means that power consumption
will depend somewhat on screen contents, which is kind of annoying.

Also, it's an early big.LITTLE device, which means it has rather odd CPU
governor behavior, and a very large hump in its power consumption.

[Benjamin Smedberg]

On 9/25/2015 1:25 PM, Thomas Daede wrote:
>
> Also, it's an early big.LITTLE device, which means it has rather odd CPU
> governor behavior, and a very large hump in its power consumption.

I don't know what this means, but I'm curious now! Do you have any links
to help educate on what this means?

From a product perspective, one of the reasons we picked the S4 is that
we expect the next generation of "cheaper" smartphones will be similar
in spec to the S4 (memory, CPU speed, network chipset/capabilities, and
battery/device size). Do you expect this is also true of "big.LITTLE"?
If so, that's probably an indication that we should be measuring this
type of config.

--BDS

[Thomas Daede]

On 09/25/2015 10:47 AM, Benjamin Smedberg wrote:
> I don't know what this means, but I'm curious now! Do you have any links
> to help educate on what this means?

LWN has a large number of articles on it, especially from the kernel
side of view:

https://lwn.net/Articles/539840/

Older article that's more introductory and has info about the old
hypervisor solution:
https://lwn.net/Articles/481055/

The S4 probably either uses the hypervisor or maybe IKS (it's barely new
enough).

> From a product perspective, one of the reasons we picked the S4 is that
> we expect the next generation of "cheaper" smartphones will be similar
> in spec to the S4 (memory, CPU speed, network chipset/capabilities, and
> battery/device size). Do you expect this is also true of "big.LITTLE"?
> If so, that's probably an indication that we should be measuring this
> type of config.

big.LITTLE makes sense if you have a big, power hungry CPU, but if
you're on the low end, you're usually struggling to get enough
performance, so adding a way to reduce your performance is
counterproductive. I don't expect to see it on lower end phones for a
little while, though farther out it's anyone's guess. I have a new low
cost HiSilicon chip with 8(!) Cortex-A53 cores, which are all the
"LITTLE" in-order cores.

The AMOLED screen is also quite expensive and has supply problems, and
is basically exclusively Samsung at this point. Luckily you can pretty
easily work around that difference.

[Nicholas Nethercote]

On Fri, Sep 25, 2015 at 3:23 PM, Benjamin Smedberg
<benj...@smedbergs.us> wrote:
>>
> Windows is far more important than Mac in terms of product priority.

That's true in general, but power consumption is much more important
for laptops than desktop machines. Do we have data about the
laptop/desktop split for Windows and Mac? I suspect the fraction of
laptop users is much higher on Mac.

> Is there test engineering/QA support for this project yet? Can we as a
> general rule try to reproduce each of these mac bugs on Windows?

I have a Windows machine ordered. (I currently have a Windows VM but
that's hopeless for measuring power usage.)

Nick

[Nicholas Nethercote]

Or try to reuse an existing driver, such as the one that comes with
Open Hardware Monitor (http://openhardwaremonitor.org/).

I'm planning to attempt a Windows rapl port for one of my Q4 goals.
Having a non-VM Windows machine (which I have on order; see above) is
necessary for this because VirtualBox and VMWare don't virtualize the
model-specific registers that contain the power estimates that rapl
uses!

Nick