Proposal: Unified Bootstrap Stages for Gecko

[zbran...@mozilla.com]

Gecko has a pretty substantial number of metrics used for measuring startup performance.

We have probes in Telemetry [0], StartupTimeline [1], various uses of MozAfterPaint, both in talos tests [2] and in production code [3][4].
We also have first paint in compositor [5], before-first-paint [6], timeToNonBlankPaint [7] which should not be confused with firstNonBlankPaint [8] and probably a number of other probes that register different timestamps and are used all around to mean different things. Some measure layout, others composition. Many of them are misused by capturing the timestamp in the callback to event listener fired asynchronously post-event.
We end up seeing huge swings in some "first-paint" [9] that are not reproducible in another "first-paint" [10], and we know that things like WebRender may not affect some of our "first-paint" because it measures only part of paint that WebRender doesn't affect[11].

It doesn't help that some of them are chrome-only while others are available in content.

I believe that, while we can recognize the complexity of the systems in play and how this complexity explains why different probes ended up being constituted, this situation is counter productive to our ability to understand the performance implication of our changes in product.

I'd like to suggest establishing a single set of timestamps with unique names to represent various stages of the product launch.
Those timestamps would be described based on the user-perceived results and as such serve us as best-effort approximations of the impact of any change on the user-perceived experience.

In particular, my proposal is based on WICG Paint Timing proposal [12] and establishes the 5 major timestamps to be executed at the latest event that contributes to the user-perceived outcome.
For example, when selecting when to mark "xPaint" event, we will use the consumer notion of the term "paint" and mark it after *all* operations required for the paint to happen are done - layout, composition, rendering and paint.

My proposal is also based on the work on negotiated performance milestones established for Firefox OS project [13].

The proposed milestones are:

1) firstPaint

This milestone happens when the first paint that is influenced by the data for the measured object is completed by the engine (and likely submitted to the graphic driver).

In the context of an HTML document, the first paint that is affected by the document's background or title is completed.

2) firstContentfulPaint

The first contentful paint of browser.xul happens when the first paint that includes layout of DOM data from browser.xul is completed.

3) visuallyCompletedPaint

This milestones is achieved after the first paint with the above-the-fold part of the DOM ready is submitted. This event may require the document to inform the engine that all the items in the visual field are ready, and the next paint captures the timestamp.

4) chromeInteractive

This milestone is achieved when the app reports the UI to be ready to be interacted with. The definition of what constitutes of the UI being ready is up to the app, and may just include the URL bar being ready to receive URL input, or may wait for all core parts of the product to have event handlers attached (urlbar, tabbar, main menu etc.)
This milestone may be reached before (3), but not after (5).

5) fullyLoaded

This milestone also may require data from the document and it should mark the timestamp when all startup operations are completed.
This should include delayed startup operations that may have waited for previous stages to be achieved, but should not wait for non-startup delayed operations like periodic updates of data from AMO etc.

The last milestone is a good moment to reliably measure memory consumption (possibly after performing GC), take a screenshot for any tests that compare UI between starts and so on.

Generally speaking, (5) is when we naively can say "the app is fully launched".


==================

This system would require us to provide a way for each document to inform the engine when some of the later stages are reached. Then the engine would take the next full paint and capture the timestamp.
Such timestamp list would be available for chrome and behind a flag for content, to be used by tests.

The value of such milestone approach lies not only in unification of reading, but also easier hooking of code into the bootstrap process. Having such a consistent multi-stage bootstrap allows developers to decide at which stage their code has to be executed to delay only what has to be affected by it and in result developing a culture of adding code that doesn't affect early stages of the bootstrap unnecessarily.

Lastly of course, the value comes in our ability to say that all telemetry probes, talos tests, tp6 tests etc. and automation can now rely on a single set of timestamps which would increase developers ability to understand and verify how their patches affect the user perceived experience.

I do not expect us to place all of them at once, but I believe that there's a value in having a conversation and establishing the list of marks we want to unify around and then start moving toward them.
In the end, the bootstrap timestamps of browser.xul should be read in a very similar fashion to how we read bootstrap timestamps of amazon.com with the difference being that later stages like "visuallyLoaded" or "interactive" may be triggered by browser.xul's JS code, while for amazon, we'll either not capture them, or use some custom code in testing machinery to decide when the hero elements are interactive.

zb.


[0] http://searchfox.org/mozilla-central/source/toolkit/components/startup/StartupTimeline.cpp#51
[1] http://searchfox.org/mozilla-central/source/toolkit/components/startup/StartupTimeline.h#15
[2] http://searchfox.org/mozilla-central/source/testing/talos/talos/tests/tpaint/file_tpaint_win.html#4
[3] http://searchfox.org/mozilla-central/source/browser/base/content/browser.js#1388
[4] http://searchfox.org/mozilla-central/source/browser/base/content/tabbrowser.xml#1124
[5] http://searchfox.org/mozilla-central/source/gfx/layers/ipc/UiCompositorControllerMessageTypes.h#24
[6] http://searchfox.org/mozilla-central/source/dom/ipc/TabChild.cpp#2847
[7] http://searchfox.org/mozilla-central/source/dom/performance/PerformanceTiming.h#249
[8] http://searchfox.org/mozilla-central/source/layout/base/nsPresContext.h#1430
[9] https://telemetry.mozilla.org/new-pipeline/evo.html#!aggregates=75th-percentile!95th-percentile&cumulative=0&end_date=null&keys=&max_channel_version=nightly%252F57&measure=SIMPLE_MEASURES_FIRSTPAINT&min_channel_version=nightly%252F54&processType=*&product=Firefox&sanitize=1&sort_keys=submissions&start_date=null&trim=1&use_submission_date=0
[10] https://treeherder.mozilla.org/perf.html#/graphs?series=mozilla-central,1517547,1,1&series=mozilla-central,1535589,1,1&series=mozilla-central,1535590,1,1&series=mozilla-central,1535587,1,1&series=mozilla-central,1535588,1,1
[11] https://bugzilla.mozilla.org/show_bug.cgi?id=1388157#c15
[12] https://w3c.github.io/paint-timing/
[13] https://developer.mozilla.org/en-US/docs/Archive/B2G_OS/Developing_Gaia/Raptor/Responsiveness_guidelines

[Mike Conley]

Hey gandalf,

Using WICG for browser start-up makes sense to me.

We should also consider adding a milestone for the "hero element" for
the browser. There's some discussion fragments in bug 1369417 about this
(which I see you're already involved in! Great!), captured from a very
caffeinated discussion with jmaher, florian and a few other folks in SF
this past all-hands.

At any rate, this all sounds strictly better than what ts_paint
currently captures. We just need to ensure that we measure paint times
when they're presented to the user (so after composite), and using the
timestamps on the MozAfterPaint events themselves[1] (and not taking a
timestamp at the event-servicing time, as this adds noise and padding to
the timestamp).

So, uh, thumbs up from me. :)

-Mike

[1]:
http://searchfox.org/mozilla-central/rev/f2a1911ad310bf8651f342d719e4f4ca0a7b9bfb/dom/webidl/NotifyPaintEvent.webidl#36

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>

[zbran...@mozilla.com]

On Tuesday, September 5, 2017 at 8:18:40 AM UTC-7, Mike Conley wrote:
> We should also consider adding a milestone for the "hero element" for
> the browser. There's some discussion fragments in bug 1369417 about this
> (which I see you're already involved in! Great!), captured from a very
> caffeinated discussion with jmaher, florian and a few other folks in SF
> this past all-hands.

I think we can dissect it in several ways:

1) We can separate firstContentfulPaint (any DOM is painted) from firstMeaningfulPaint (the paint that contains the selected hero element(s)).

I would be happy to see it as a progression. Once again, this way we can instrument our own code *and* our tp6 suite by marking which elements are "meaningful" and mark the timestamp only when those were accounted for in a paint that was completed.

The difference between firstPaint and firstContentfulPaint is the time when the page was "blank" but something (from <head> or manifest, so title or background) indicated that the load is successful.

The difference between firstContentfulPaint and firstMeaningfulPaint would be the time when the document was being reflowed. I expect that in most cases, and definitely in case of browser.xul, those two (actually, those three) will stay the same.
But you can imagine that we could then, say, switch URL bar to be injected from JS, and since it's marked as meaningful, we'd mark the timestamp only when this element was accounted for in the paint.

2) There's a distinction between visible and interactive.

In theory, we could paint, say the url bar, but until its JS is ready and hooked, it's not interactive.

I was thinking about defining the "uiInteractive" mark to be set when the JS required for making the meaningful UI pieces is ready. That could be up to Firefox UI owners discretion - maybe URL bar and tabbar is enough, maybe we want to wait for something more.

>
> At any rate, this all sounds strictly better than what ts_paint
> currently captures. We just need to ensure that we measure paint times
> when they're presented to the user (so after composite), and using the
> timestamps on the MozAfterPaint events themselves[1] (and not taking a
> timestamp at the event-servicing time, as this adds noise and padding to
> the timestamp).

Yeah. That seems the hardest part for me, since the required instrumentation goes far beyond my skills.
In bug 1388157 comment 5 [0] Markus described what has to happen for us to make us register full paints (including composition) timestamps involving given requirements.

> So, uh, thumbs up from me. :)

In comment 2 of the same bug, Markus pointed out that we'd need a product / metrics decision on it.

I'd be happy to formalize the proposal into a document and build a plan for implementing it (starting with a new firstPaint that will unify all firstpaints!). But I believe before I invest that time, I'd like to get consensus among platform, metrics, and graphics engineers to know that this proposal is something they'd be willing to work toward.

zb.

[0] https://bugzilla.mozilla.org/show_bug.cgi?id=1388157