Steady performance regression of master on datazilla (flame-319mb)

[Zibi Braniecki]

Over last week, datazilla is showing a steady performance regression on all apps on flame-319mb.

The regression is within 100-200ms range and seems to affect all apps. There's also no single regression point, but rather what looks like a trendline.

It makes me suspicious over wherever it really is a regression from a patch or some sort of accruing measurement error.

Anyone has any idea where it may be coming from?

zb.

[Kevin Grandon]

Sounds like it might be a system regression? I'm not sure if it's entirely related yet, and there may be more issues, but one bug we're looking at is this right now: https://bugzilla.mozilla.org/show_bug.cgi?id=1050510

Best,
Kevin

_______________________________________________
dev-gaia mailing list
dev-...@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-gaia

[Gabriele Svelto]

> Over last week, datazilla is showing a steady performance regression on all apps on flame-319mb.
>
> The regression is within 100-200ms range and seems to affect all apps. There's also no single regression point, but rather what looks like a trendline.
>
> It makes me suspicious over wherever it really is a regression from a patch or some sort of accruing measurement error.
>
> Anyone has any idea where it may be coming from?

Could be a side effect of the new generational garbage collector?

https://groups.google.com/d/msg/mozilla.dev.platform/h889H0EATJ4/Gs0vhzfLwQMJ

Somehow it doesn't sound like it but it might be worth checking since it
was enabled on B2G only recently.

Gabriele

[William Lachance]

FWIW Eideticker is noticing no such regressions:

E.g.:

http://eideticker.mozilla.org/#/b2g/flame-319/2.0/b2g-contacts-startup/timetostableframe/30

http://eideticker.mozilla.org/#/b2g/flame-319/2.0/b2g-dialer-startup/timetostableframe/30

http://eideticker.mozilla.org/#/b2g/flame-319/2.0/b2g-settings-startup/timetostableframe/30

Will

[Zibi Braniecki]

Can someone confirm my observations?

If you look at each of the apps separately here: https://datazilla.mozilla.org/b2g/ - in basically every one of them you'll see a performance regression between 11th and today.

if you look here: https://datazilla.mozilla.org/?start=1410453936&stop=1411058736&product=B2G&repository=master&test=settings&page=startup_%3E_moz-app-visually-complete&project=b2g - you'll also see the same regression (I guess it makes sense since it's the same data, right?)

Examples:

- Settings from 2600-2700 range to 2800-2900
- SMS from ~1600 to 1700-1800
- Video from 950-100 to 1000-1050
- Contacts from 900-1000 to 1100-1200
- Dialer from ~700 to 750-800
- Music from 1100-1200 to 1400

I think it's worth raising the alarm, filing a bug and getting people to track it down on its own, because it's a pretty serious and massive regression, but I feel like it may be very well my own lack of skill in reading the charts since there's a lot of noise overall (std is soo high :( )

When I try to read the trendline out of the graphs for two most telling apps - Settings and Music, I feel like there are two "jumps"
- one on 15th when in both, Settings and Music, we loose 100ms
- second on 16th when in both we loose another 100ms

I'm trying to be careful not to read too much from revisions in the perf graphs because they may be misleading, but trying to overlap what I see as dipping points and commit ranges my first attempt would be to assume that:

- the first regression might be https://github.com/mozilla-b2g/gaia/commit/f4e4858733e19ccfc278b7fe7e01dd7a16fb456b
- the second regression might be between:
- https://github.com/mozilla-b2g/gaia/commit/ebb3f6c3de5e1e6b1642a167866b47081db7d263
- https://github.com/mozilla-b2g/gaia/commit/1503e1f9f7846a3813c057ce95c35ac22031027f

Those two regression ranges match my local machine b2gperf results for them.

Can someone with more knowledge and experience look into this and tell me if I'm tracking a real thing or my own imagination and if it's worth filing two bugs?

zb.

[eper...@mozilla.com]

In my opinion these are real regressions and should be tracked as such. I don't see any indications that these are the result of some sort of accruing error. If I had to guess I would say the source of these regressions is either from the System app or within Gecko, but it will take some testing to determine. If you would like to file bugs and cc me on them, I'd be happy to get the ball rolling.

Thanks,

Eli Perelman
Software Engineer - Firefox OS

[eper...@mozilla.com]

Hey Will,

It appears you shared links for numbers based on the 2.0 branch, but when I look at the numbers for b2g-inbound, it appears the line trends upwards. Can you confirm this is the case since this should be against master?

Also with v2.1 being branched now, maybe you can create another branch in Eideticker for it?

Thanks!

Eli Perelman
Software Engineer - Firefox OS

[Zibi Braniecki]

Ok, filed:
- https://bugzilla.mozilla.org/show_bug.cgi?id=1069450 for 15th
- https://bugzilla.mozilla.org/show_bug.cgi?id=1069452 for 16th

zb.

[William Lachance]

Oops, Eli pointed out that I accidentally linked to the 2.0 branch.
These regressions are actually visible if you look at b2g-inbound:

http://eideticker.mozilla.org/#/b2g/flame-319/b2g-inbound/b2g-contacts-startup/timetostableframe/30

http://eideticker.mozilla.org/#/b2g/flame-319/b2g-inbound/b2g-dialer-startup/timetostableframe/30

http://eideticker.mozilla.org/#/b2g/flame-319/b2g-inbound/b2g-settings-startup/timetostableframe/30


Sorry about the confusion. It doesn't appear that this is indeed a
problem (and not just an issue with b2gperf).

Will

[William Lachance]

On 14-09-18 01:21 PM, moz...@eliperelman.com wrote:
>
> It appears you shared links for numbers based on the 2.0 branch, but
> when I look at the numbers for b2g-inbound, it appears the line trends
> upwards. Can you confirm this is the case since this should be against
> master?

Yup, noted my error. :)

> Also with v2.1 being branched now, maybe you can create another branch
> in Eideticker for it?

Filed: https://bugzilla.mozilla.org/show_bug.cgi?id=1069477

Will