Client side Speed Index using paint invalidations

[Charles Harrison]

Hi paint team,

After a discussion with Chris at BlinkOn, we're planning on implementing the Speed Index metric in chrome and reporting it to UMA.

Brief background: Speed Index tries to calculate the average time that elements are visible in the viewport. This entails calculating the integral of % visual complete over time.

So far, we're using invalidation rects here, we're seeing good numbers compared to calculations using video capture. However, we'd like to improve things even more.

1. PaintInvalidationReason looks really helpful to us. Are there Reasons that we should discard as not affecting user perception of loading? Scroll and Selection look like good candidates for this. Are there some reasons that should be weighted more?

2. Can we annotate the invalidations with more information? I'm not sure the extent of information the layout objects have, but it would be great to somehow know if the invalidation was "contentful" i.e. text/image. I realize this might not be possible.

3. Is the place we are instrumenting (GraphicsLayer::invalidateDisplayItemClient) the right place? Chris mentioned PaintController is the place to do this, but so far that's the only caller (and it's where the rest of our metrics are instrumented). Will this change as SlimmingPaint progresses?

A draft CL is up here, and aside from potential layering issues, it seems to get the job done reasonably.

[Jeremy Roman]

Yes, we eventually intend to remove GraphicsLayer, at which point there will be one PaintController (possibly one per local-root in OOPIF), and layers won't be updated until a phase after paint invalidation.

A draft CL is up here, and aside from potential layering issues, it seems to get the job done reasonably.

--
You received this message because you are subscribed to the Google Groups "paint-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to paint-dev+...@chromium.org.
To post to this group, send email to pain...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/paint-dev/CADjAqN6_otrkdXuB7pdEj%3Dx83fLq_9ns73PVeVMZGz%2Bd0QmazA%40mail.gmail.com.

[Charles Harrison]

Thanks Jeremy. In that case we should probably migrate all the paint metrics instrumentation up to PaintController.

[Nat Duca]

Would you be open to writing a design doc on this? I think it would be useful so that those on the outskirts can understand how this works in practice...

--

[Charles Harrison]

I wrote a design doc prior to coding it up, should have linked it earlier:

https://docs.google.com/a/chromium.org/document/d/1463ubiYUNrLlI_41c7Yb0qkAdVhS1pxWavqWafA6lUo/edit?usp=sharing

Some caveats because the doc is a little stale:

The "true" speed index that mimics the original blink implementation actually scored worse than a hybrid algorithm that I'm using now.

The current algorithm keeps a HashMap of Rect => # occurrences, and contributes Area / # occurrences for each rect. This entails only keeping O(unique rects) memory, and it performs better.

[Xianzhu Wang]

On Tue, Dec 8, 2015 at 8:03 AM, 'Charles Harrison' via paint-dev <pain...@chromium.org> wrote:

Hi paint team,

After a discussion with Chris at BlinkOn, we're planning on implementing the Speed Index metric in chrome and reporting it to UMA.

Brief background: Speed Index tries to calculate the average time that elements are visible in the viewport. This entails calculating the integral of % visual complete over time.

So far, we're using invalidation rects here, we're seeing good numbers compared to calculations using video capture. However, we'd like to improve things even more.

1. PaintInvalidationReason looks really helpful to us. Are there Reasons that we should discard as not affecting user perception of loading? Scroll and Selection look like good candidates for this. Are there some reasons that should be weighted more?

I think this depends on how you count the contribution of different types of document updates to the final speed index result. For example, an invalidation may be caused by:

- a newly appearing object (LocationChange, BoundsChange, BecomeVisible);

- style change (StyleChange);

- layout/layering/compositing change of existing objects because of newly appearing objects or script execution during load (other reasons);

etc.

Currently PaintInvaildationReasons is designed mainly for human debugging. Each invalidation can have only one reason, so sometimes a reason that is more important for speed index may be hidden by another reason. For example, PaintInvalidationOutline may hide PaintInvalidationBecomeVisible if the object having outline is appearing. If needed, we can change PaintInvalidationReason to bitfields, or try to make them reflect the most important reasons for speed index.

2. Can we annotate the invalidations with more information? I'm not sure the extent of information the layout objects have, but it would be great to somehow know if the invalidation was "contentful" i.e. text/image. I realize this might not be possible.

I guess measuring the complexity of the SkPictures of repainted display items would work. The simplest method might be to get the sum of SkPicture::approximateOpCount() of all repainted display items during each painting.

Using this method, perhaps we can avoid depending on PaintInvalidationReasons.

3. Is the place we are instrumenting (GraphicsLayer::invalidateDisplayItemClient) the right place? Chris mentioned PaintController is the place to do this, but so far that's the only caller (and it's where the rest of our metrics are instrumented). Will this change as SlimmingPaint progresses?

A draft CL is up here, and aside from potential layering issues, it seems to get the job done reasonably.

--

[Charles Harrison]

Thanks for the info! What you suggest (Using SkPicture::approximateOpCount()) sounds really interesting. Unfortunately, my understanding of the painting pipeline is rather limited.

What is the relationship between SkPictures and invalidation rects? My understanding is that the invalidation rects are tied to layout objects (display item clients), while SkPictures are the actual commands to paint which are generated later. Would moving to analyzing SkPictures replace the invalidation rect approach, or could it work in tandem (i.e. record invalidation rects, then weight them as SkPictures are created)?

[Xianzhu Wang]

On Tue, Dec 8, 2015 at 10:25 AM, Charles Harrison <cshar...@google.com> wrote:

Thanks for the info! What you suggest (Using SkPicture::approximateOpCount()) sounds really interesting. Unfortunately, my understanding of the painting pipeline is rather limited.

What is the relationship between SkPictures and invalidation rects? My understanding is that the invalidation rects are tied to layout objects (display item clients), while SkPictures are the actual commands to paint which are generated later. Would moving to analyzing SkPictures replace the invalidation rect approach, or could it work in tandem (i.e. record invalidation rects, then weight them as SkPictures are created)?

In PaintController, each drawing display item has a SkPicture and each display item client has a invalidation rect. After a paint, we can get a list of (invalidationRect, approximateOpCount) for all repainted (invalidated before painting) display items. I guess this data will be useful to calculate the loading progress and we can avoid using PaintInvalidationReasons.

[Charles Harrison]

That sounds perfect. It seems like every DisplayItem has a client, but I don't see how every DisplayItemClient has an invalidation rect. There's a snippet in PaintController::invalidate

    if (visualRect) {
        // TODO(wkorman): cache visualRect for the client.
    }

Is caching the visualRect a prerequisite for PaintController to have a mapping from DisplayItemClient -> invalidation rect?

[Walter Korman]

I'm adding logic right now to build and maintain the mapping, see http://crrev.com/1484163002. I'm working to get majority of this logic in this week in a series of smaller changes, so unless you get something written up sooner, it should be present in some form for you to work with. 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/paint-dev/CADjAqN4Aw8aDB4b0MfZuYwX2nndnaC84PHVP2brPK0%2BfqossoQ%40mail.gmail.com.

[Charles Harrison]

Great! I have a working algorithm right now that just logs every invalidation as it comes in, but I can sit on that until your changes start landing. I'm hopeful that this approach will be cleaner and more accurate than my current version.

[Walter Korman]

Sounds good, I will add you to http://crbug.com/529938 which I'm using as the tracking bug for the involved changes.

[Elliott Sprehn]

Is this meant to run on all page loads or just ones where tracing or some instrumentation is turned on?

On Dec 8, 2015 1:48 PM, "'Charles Harrison' via paint-dev" <pain...@chromium.org> wrote:

Great! I have a working algorithm right now that just logs every invalidation as it comes in, but I can sit on that until your changes start landing. I'm hopeful that this approach will be cleaner and more accurate than my current version.

--

You received this message because you are subscribed to the Google Groups "paint-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to paint-dev+...@chromium.org.
To post to this group, send email to pain...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/paint-dev/CADjAqN7%3DfVYw3XNJqUgbg9aqxjyAJgUEcBDEkJANzVuQUxNteg%40mail.gmail.com.

[Charles Harrison]

This is ideally meant to run on all page loads where the user has opted into UMA. If that's still too much (depending on how the implementation lands wrt memory/cpu), we can think about sampling.

[Yoav Weiss]

I'm very excited about that work! :) Are there any plans to eventually expose this to RUM?

You received this message because you are subscribed to the Google Groups "Chromium Loading Performance" group.
To unsubscribe from this group and stop receiving emails from it, send an email to loading-dev...@chromium.org.
To post to this group, send email to loadi...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/loading-dev/CADjAqN6FB0JcGUrqNYX%2BknK_jiAmfF-QTcie7Gg6ZgxDeWOLQg%40mail.gmail.com.

[Charles Harrison]

Not yet that I know of, but there has been some conversation about that wrt newer metrics like firstTextPaint / firstImagePaint.

I think if this proves to be accurate enough it would be great to expose.

As a reference, there is an implementation that Pat wrote for RUM that uses external resource loads.

https://github.com/WPO-Foundation/RUM-SpeedIndex/blob/master/src/rum-speedindex.js

[Ojan Vafai]

Question for paint team: Are you ready to commit to keeping around invalidation rects indefinitely? My recollection is that this is still an option question.

For Charles, some questions you might want to tackle in your design doc: 

1. What is the larger goal here?
   
1. Expose more reliable UMA?
2. Expose SpeedIndex in tracing and/or devtools?
3. Expose SpeedIndex to RUM-style measurement, i.e. so developers can measure their performance in the wild.
2. From your patch it seems you only care about above the fold (which makes sense). What should happen if the user scrolls before above the fold has stabilized?

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/paint-dev/CADjAqN6b9EtT%2BXXERxB%3DK1q-nBP09OFyxhcDJLFN5Z5%3D0u48mw%40mail.gmail.com.

[Chris Harrelson]

On Mon, Dec 14, 2015 at 12:52 PM Ojan Vafai <oj...@chromium.org> wrote:

Question for paint team: Are you ready to commit to keeping around invalidation rects indefinitely? My recollection is that this is still an option question.

The concept of an invalidation region will remain, since there are other required use cases such as raster invalidations and display item bounds, so I think continuing to explore this use case is fine.

[Charles Harrison]

Ojan, great questions.

1. The original goal was to expose more reliable UMA. I think that will be the litmus test for further exposure. Hopefully in the future this could get exposed both in tracing/devtools as well as RUM/js if the metric proves reliable and accurate.

The only reason I can think of that would prevent exposure would be

 - The metric is unreliable (see below).

 - We cannot afford to calculate the metric for every page load (memory, etc.)

2. Yeah, the video capture SpeedIndex in WPT only captures ATF content, as it tries to measure user perception. If the user scrolls we'll probably end up invalidating the calculation. Anything else I imagine would be more noisy and less correct. This fact makes me wary about reporting this statistic to RUM, as we may not want to report a value that isn't collected every page load.

Alternatively, it might make sense to calculate SpeedIndex looking at the rect that is visible to the user. Thus, when the user scrolls, our reference viewport rect scrolls as well. I think this would be much more technically challenging, as weighting invalidations (or whatever we instrument) during a scroll could be difficult. 

[Ojan Vafai]

On Mon, Dec 14, 2015 at 1:11 PM Charles Harrison <cshar...@google.com> wrote:

Ojan, great questions.

1. The original goal was to expose more reliable UMA. I think that will be the litmus test for further exposure. Hopefully in the future this could get exposed both in tracing/devtools as well as RUM/js if the metric proves reliable and accurate.

The only reason I can think of that would prevent exposure would be

 - The metric is unreliable (see below).

 - We cannot afford to calculate the metric for every page load (memory, etc.)

Just to put some other things on your radar about exposing this:

-If we're exposing this in tracing/devtools, using paint invalidation rects might be pretty surprising. There will likely be weird edge cases that will be confusing to developers because they are not built for this purpose (i.e. we'll over invalidate in confusing cases). That can make for a frustrating and untrustworthy developer experience. You need to be able to explain this to authors in a way that isn't understandable and not too dependent on quirks of our implementation.

-If we started exposing a RUM/js API, then you will need to think about how other browser vendors can make an interoperable implemention. So, depending on the details of how we do paint invalidation rects would almost certainly be a non-starter.

I think if we expose this to devtools and/or RUM/js, we might need a more direct approach in the code that we could explain to web developers and that other browsers could implement, which might then affect the performance characteristics.

Nat has expressed concerns about exposing metrics only to UMA. I don't know if they apply here. Nat?

 

2. Yeah, the video capture SpeedIndex in WPT only captures ATF content, as it tries to measure user perception. If the user scrolls we'll probably end up invalidating the calculation. Anything else I imagine would be more noisy and less correct.

For the sake of UMA, what would we track? SpeedIndex of pages that hit the SpeedIndex value without a scroll having happened? That might be problematic. For example, what if we are doing a finch experiment on something that happens to delay SpeedIndex in some scenarios such that users often scroll first. In that case, our sample size would change and make apples-to-apples comparisons hard. Technically, we should be able to deduce that from the data, but it'd be very hard.

 

This fact makes me wary about reporting this statistic to RUM, as we may not want to report a value that isn't collected every page load.

Alternatively, it might make sense to calculate SpeedIndex looking at the rect that is visible to the user. Thus, when the user scrolls, our reference viewport rect scrolls as well. I think this would be much more technically challenging, as weighting invalidations (or whatever we instrument) during a scroll could be difficult. 

FWIW, I think there are reasonable solutions to this problem if we can figure out what we should do for UMA. For example, we could expose to authors that the user scrolled before speedindex was done. Then they could filter as they prefer. Definitely something we can figure out in the future once we've proven to ourselves there is a good metric here.

To be clear, my goal here is not to block this idea, but to help you refine it. Don't take any of my comments as a request to stall your efforts.

[Paul Irish]

DevTools currently reported DOMContentLoaded and window load times to the user, but as we all know, these metrics are really unrepresentative of perceived load.

We're very interested in showing a Speed Index figure in DevTools.  

Also.. worth pointing out: once this metrics is available (either through a JS API or DevTools), it will be trivial for us to compare our new SI numbers vs WebPageTest's SI methodology. Pretty handy we can evaluate the figures so easily.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/loading-dev/CANMdWTvwn3NH-5VzwAd80bPNtv_cNdihvJtLEKmtjmA63JLPnA%40mail.gmail.com.

[Charles Harrison]

Ojan:

No, I really appreciate the feedback. You raise a lot of good points that we need to think about.

I'd need a bit more input from the paint team whether or not a more direct approach in the code could work. It was my understanding that after the move to cc/, what is actually painted is too coarse grained for Speed Index to be reliable. Another approach would be to use the same algorithm as WPT (histogram of pixel colors), but that requires sampling the pixels at a constant rate, which would hurt performance.

I'd guess that making this general enough that we could write up a spec for it and rope in other browsers will be very difficult.

RE: tracing: I think tracing could easily expose this metric. As long as there's a trace point where we collect invalidation rects, they can re-run the algorithm as a TBMv2 mapper. That sidesteps the question of whether to expose it in devtools / RUM however.

Paul:

Pat recently expose blink.user_timing traces to WPT, which makes comparing these figures very easy if it's traced with that category.  These get collected for every test by default I believe.

[Charles Harrison]

Ojan and I just synced about this. Some notes:

- We probably want to expose this to developers, but we need something that isn't flaky or inconsistent. This is a tough problem.

- Moving forward, we should keep in mind what a spec would look like, and a place to instrument that code that will support it.

- We want some notion of "stable above the fold" or "mostly loaded." How to implement this remains to be seen. Can we calculate something like "75% of ATF pixels painted a non-background color?" Ojan mentioned Slimming paint v2 could possibly perform this kind of introspection. This event would be the spiritual successor of onload, and could serve to be the stopping point of Speed Index.

Open Questions

 - Are we instrumenting the right layer? Layout might be a better place than paint for simplicity / spec work.

 - How to deal with ThrottledRendering / Backgrounding / Scrolling. Possible solution: Send a boolean to devs if we're not confident with results / they were impaired. For a layout-based approach, we could still look at layouts in the original ATF viewport if the user scrolled down. Actually using scrolled-to content in SpeedIndex seems like a non-starter for reliable results (devs metrics busted if their users start scrolling more?).

Next steps

I'll schedule a vc where some people on the paint / layout / loading teams can chat about this. I'll also update the doc to be a little more clear.

[Nat Duca]

Should the code that is used for computing client side speed index relate to the code that computes first meaningful paint? We seem to be approaching both at once, from two different continents.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/loading-dev/CADjAqN48M4T3cKABYYrQZzQBE-EAfNzDTyxpo8Fwn-f5YLNRBA%40mail.gmail.com.

[Charles Harrison]

I don't think so. The point metrics in order will be:

first paint -----first meaningful paint-----------------above the fold stable---------------------------onload

I think speed index should integrate loading between first paint and above the fold stable. First meaningful paint will be a better point metric than first paint, but I still think first paint should be included in the speed index measure.

[Charles Harrison]

Sorry I misread your email. Yes I think the code should relate :)

[Charles Harrison]

Although, it might end up that we spec speed index by layout instead of paint, in which case it might not make sense to relate them or probe them in the same place.

[Nat Duca]

Yeah the main thing I'm poking at is that right now, it looks like speed index  might derive from an after-layout pass, whereas first-meaningful-paint is derived from paint pass data. Maybe thats right in the end, maybe its not. I'm just not sure all the different parties are talking to each other yet :)

[Kinuko Yasuda]

Yep.  They are different metrics that could have different characteristics and implementation details, and the speed index implementation is in an experimental stage which helps us to have a better idea in this problem space.  Currently both are on our prioritized list, we have some conversation in between, but the two efforts are being done mostly independently.  We should be reviewing and setting our metrics strategy including the newly explored space for the next coming year soonish-- I assume.

Also: the questions asked on this and some other threads look to be a good starting point for us to build the common ground for all loading/paint metrics:

• All the signals we'll be using for metrics should be (ideally) able to be explained in a clear text without needing to touch browser-specific implementation details.
• When thinking about new metrics, we should also consider who/what are the target users/scenarios and where we're exposing the signals.  Ojan's question list seems to be a great start for these, to rephrase (and adding some):
• Are we exposing it on UMA?
• Are we exposing it on tracing and/or devtools?
• Are we exposing it to RUM-style measurement for developers?
• Related, is it going to be speccable?
• Also: is it aiming to replace similar approaches we've done before and are doing?

First paint effort is also trying to get more signals from paint layer, and we might end up with different signals, but we should keep asking the same questions & keep looking if there are things we could converge.

[Charles Harrison]

+layout-dev

Hi folks,

In preparation for the holidays, I put together an updated document on Speed Index implementations. We are receptive about speccing it out if that turns out to be a feasible path.

Please feel free to comment! My sections on layout are especially bare because I have limited experience. I'd appreciate any insights the layout team can give (e.g. where to best instrument these metrics).

I wanted to get this out before we schedule a meeting to see if we can resolve any open questions beforehand.

Happy holidays!

Charlie

[Charles Harrison]

Bumping this thread to request post-holiday comments. 

[Charles Harrison]

New document! https://docs.google.com/document/d/1uTkuOu1iEwAjDj8Yc-nGRtRAVm4FhATjzrGoSHb-9fQ/edit?usp=sharing

This describes the general game plan for getting a trace-based speed index up and running, as well as a brainstorming section for additional time-aware metrics. The brief sum up from the meeting was

1. This is hard for a single user (non lab conditions, interaction)

2. This is harder for all UMA opt-in page loads (performance, noise)

3 This is even harder for all page loads for a timing spec (PERFORMANCE)

So, we're going to work on (1) with tracing for now. Additionally, we're continuing to brainstorm additional metrics.