First meaningful paint?

[Kunihiko Sakamoto]

Hi loading-dev,

Last quarter I added a set of simple first-X-paint metrics. Now I'm thinking about "first meaningful paint" - something closer to user-perceived page load time.

To study how to define such a metric, I created a catalog page of WPT screenshots:

https://googledrive.com/host/0B_IpB7WXmT8iQlJYRjhJbGJnV2c/

I'm wondering what a useful definition of first-meaningful paint would be. It seems not as obvious as I thought...

For example, loading of wordpress.com (the first filmstrip of the above catalog) progresses like the following:

• 2.758s: First contentful paint (logo)
  
• 2.953s: Significant part of the background image is visible
  
• 3.201s: Some text become visible
  
• 3.434s: The background image completes
  
• 3.618s: All fonts ready

Which is first meaningful paint? I think 2.953s (user sees something nontrivial) or 3.618s (button text becomes visible -> user is able to interact with the page) could be, but others may think differently.

Whatever the definition, we'll need some programmatic approximation of the metric, to make it measureable.

Random ideas that could be (components of) first-meaningful-paint criteria:

• Painted area
  
• Size of painted image, number of characters painted
• Key user moments
  
• All text is visible
• Input form is usable
• Last layout jank

Any ideas / comments will be appreciated.

Thanks!

[Patrick Meenan]

For me, most of the pages only become meaningful when the text is available.  How about something like defaulting it to the first visible text paint time and if no text is drawn then fall back to the first image paint (and largely ignore backgrounds)?  Unfortunately in the fallback case the decision would only be able to be made after the page finished loading (to know that no text painted) but I think that would work well enough in the general case.

It wouldn't be perfect because there's no guarantee that the first text painted is meaningful either but I think it handles the general case well.

The YouTube experience (and gmail) where there is a loading progress indicator first is more complicated though.  It would be great if we could ignore the loading progress indicator and only count the actual content but I can't think of a good way to do that.

--
You received this message because you are subscribed to the Google Groups "Chromium Loading Performance" group.
To unsubscribe from this group and stop receiving emails from it, send an email to loading-dev...@chromium.org.
To post to this group, send email to loadi...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/loading-dev/CAO5vZCiacbe%2BhahYhb-BHZjdaA17%2B5Zetc5e9UL0-PP7zP%2BVnw%40mail.gmail.com.

[Ilya Grigorik]

+1 to (any) visible text. It has its edge cases, but it's a reasonable user-utility proxy + reasonably sane (stable) to reason about, both as a site and browser developer.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/loading-dev/CACPgMqXdhXmU95j52%2BE-cE3PR0L1xWWNhJhX3oigcTDTPg4CLQ%40mail.gmail.com.

[Kunihiko Sakamoto]

Thanks -- sounds like first-text-paint is a good starting point.

I've added "Load/Save selections" buttons to the catalog page (https://goo.gl/UIkVpg). You can see the timing of first-text-paint by loading attached json file using the top-right "Load Selections" button.

I can see two patterns where naive firstTextPaint doesn't work (heuristics needed):

• First text is non-content. Page header, loading indicator, etc.  Example: 5. http://youtube.com, 32. http://bbc.co.uk
  
• First text is not visible. Hidden by other layers, or out of initial viewport?  Example: 18. http://flipkart.com, 71. http://msn.com
  

Btw, currently my catalog only have domain top pages (from Alexa top 200 sites). Is there any recommendation for an URL list that includes leaf pages too?

[Charles Harrison]

Try the alexa pages that Pat curated: https://github.com/WPO-Foundation/webpagetest/blob/master/bulktest/alexa-curated.txt

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/loading-dev/CAO5vZCgs2cv7QZB1zxx-uizT0iWJw3cyAggZ7pckHZDkzPFXBA%40mail.gmail.com.

[Kunihiko Sakamoto]

Nice! Created screenshot page: https://googledrive.com/host/0B_IpB7WXmT8iQlJYRjhJbGJnV2c/alexa-curated.html

and first-text-paint JSON: https://googledrive.com/host/0B_IpB7WXmT8iQlJYRjhJbGJnV2c/alexa-curated-tttp.json

[Kunihiko Sakamoto]

I did a manual evaluation for "meaningful-ness" of FirstTextPaint using this data.

2016-02-03 18:09 GMT+09:00 Kunihiko Sakamoto <ksak...@chromium.org>:

Nice! Created screenshot page: https://googledrive.com/host/0B_IpB7WXmT8iQlJYRjhJbGJnV2c/alexa-curated.html

and first-text-paint JSON: https://googledrive.com/host/0B_IpB7WXmT8iQlJYRjhJbGJnV2c/alexa-curated-tttp.json

https://docs.google.com/spreadsheets/d/1qQidtZwhKu-kf-7wnWqQhXQzbE1p18v0c7hRLhuBilI/edit?usp=sharing

Summary:

• 146 (73%) - Good, FirstTextPaint was meaningful.
  
• 8 (4%) - Splash screen / spinner
  
• I'm not sure whether we should exclude these from first meaningful paint. Page author chose to load contents in background.
• 29 (15%) - Page header only
  
• Detectable by page height?
• 11 (6%) - Not meaningful, (almost) background only
  
• In most cases, first text was not visible, i.e. FirstTextPaint was not working well.
• 5 (3%) - N/A (FirstTextPaint not recorded for some reason)

Patrick's suggestion was to use FirstImagePaint if no text is drawn, but within these 199 pages there's no such case.

[Kinuko Yasuda]

Thanks for looking into this.

On Mon, Feb 8, 2016 at 4:45 PM, Kunihiko Sakamoto <ksak...@chromium.org> wrote:

I did a manual evaluation for "meaningful-ness" of FirstTextPaint using this data.

2016-02-03 18:09 GMT+09:00 Kunihiko Sakamoto <ksak...@chromium.org>:

Nice! Created screenshot page: https://googledrive.com/host/0B_IpB7WXmT8iQlJYRjhJbGJnV2c/alexa-curated.html

and first-text-paint JSON: https://googledrive.com/host/0B_IpB7WXmT8iQlJYRjhJbGJnV2c/alexa-curated-tttp.json

https://docs.google.com/spreadsheets/d/1qQidtZwhKu-kf-7wnWqQhXQzbE1p18v0c7hRLhuBilI/edit?usp=sharing

Summary:

• 146 (73%) - Good, FirstTextPaint was meaningful.
  
• 8 (4%) - Splash screen / spinner
  
• I'm not sure whether we should exclude these from first meaningful paint. Page author chose to load contents in background.

Looking into these questions I have an unclear feeling that we don't have a good definition / base agreement of what kind of new metric(s) we want to add/define for measuring real-world loading performance yet.

For this particular one I can imagine one could argue either this is meaningful paint or not, while I can also imagine that having the time-to-splash-text/spinner could be useful regardless of whether it's considered 'meaningful' or not.  But it alone wouldn't tell all what we might want to know--- showing splash text/spinner would probably improve user experience some sense but if loading the body contents takes forever after that it'd still make the overall loading experience bad.  Then how should we track those timings?  Might we want to record multiple timings or could somehow combine them?  If we want to record multiple values/timings (I think we'll likely end up doing this) how many we'd need to add (I think we want them to be distinct / minimum)?

I think maybe you/we could first build some hypotheses (with rationals) on what / how many values we'd want to add, and then discuss how we could approximate the timings if we were to add them as UMA/RUM-type values?  (I'd also want existing metrics to be reviewed in that context too-- why they're not enough or how many we want to keep etc)

• 29 (15%) - Page header only
  
• Detectable by page height?
• 11 (6%) - Not meaningful, (almost) background only
  
• In most cases, first text was not visible, i.e. FirstTextPaint was not working well.

I think for this one many have mentioned that we'd want to only include the visible text paints.  Would it be easy to fix this?  Would we need some heuristics here?

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/loading-dev/CAO5vZCiE-6bS9NTYuBmjO3_DBGpH9vDegr1G2Sf23KhPocAyJA%40mail.gmail.com.

[Bryan McQuade]

Thank you Kunihiko for this great analysis. Would it be possible for you to put together a doc that has ~5 representative screenshots for each of the example cases you cite (first paint screenshot vs first meaningful paint screenshot vs final page screenshot)? That would help us to quickly scan and understand the challenges with each of these cases?

I agree with Kinuko that it may be hard to detect whether a spinner counts as meaningful. Assuming that detecting this will be very difficult, I think we can mostly ignore that case for now and keep it in mind for future work. That may also be a case that we decide is out of scope for first meaningful paint and is instead better covered by speed index.

I do wonder if we can detect some other cases, e.g. header only or background only, by looking at the dom/layout tree at the time of the paint and determining that a significant amount of content is not yet paintable due to e.g. pending fonts being loaded. For those cases, I think it may be worth investing further. How much work would it be for us to determine that contentful content in the DOM is not yet paintable due to pending resource loads? I imagine this information must already be computed during the layout phase - perhaps we could piggyback on those computations? Might be good to talk to the layout or paint folks to get their guidance on when we compute this information and whether it's recorded for us to look at when we're deciding whether first meaningful paint has happened yet.

-bryan

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/loading-dev/CAMWgRNYdnUdAf%2B_pTxvoKwcmzd2KtXmbrwjvtvquSFkgT10aoQ%40mail.gmail.com.

[Bryan McQuade]

+Ben Greenstein who is looking for a first meaningful paint metric.

[Kunihiko Sakamoto]

Thanks Kinuko and Bryan.

I put together a document: https://docs.google.com/document/d/1eMA9YWTLFCJomyiKUO4Be4PeVIOBFMfXMdE8bpudL_o/edit?usp=sharing

Comments and suggestions welcome!

[Ben Greenstein]

This sounds great! Would a first contentful paint metric that is based on first text wait until there are no more reflows of that first text? I'm interested in when the critical content (maybe just the text) on a page is available to the user, and I worry that until the pixels stabilize the content might not really available to be consumed even if it has been painted. Ideally, if a page has text, a bunch of images, and a bunch of ads, a first contentful paint metric would tell me when the text is stable and can be consumed. This may depend on images being loaded in to stabilize the layout, and might even depend on ads appearing.

[Kenji Baheux]

My take is that we need metrics for the following key user moments:

1. “is it happening?” (i.e. time to first paint)

2. “is it useful?” (i.e. time to first meaningful paint)

3.  “is it usable?” (i.e. time to stable layout)

Beyond loading:
4. "is it delightful?" (i.e. metrics for the RAI of RAIL)

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/loading-dev/CADPZbiAYLMDO_dsoyYMxcaP195MJ1gQCT_MBPRCYv4iRKx7h%3DQ%40mail.gmail.com.

[Jonathan Garbee]

Thinking from the perspective of a user (as RAIL advises) first meaningful paint is certainly *not* a loading spinner. It is the content, or at least the applications layout being shown.

I think Kenji presented a good flow to think by. However, I'd put Usable before Useful with the examples provided. A stable layout is where the site becomes useful, since you aren't worried about reflow happening while trying to scroll and having content jump around. Then useful is when the content comes in to fill in the layout. 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/loading-dev/CADWWn7VHeeWtnU3_L0REUJhxooYPcx%3DUW%3D7uW%3D6XuOYD8oQNOw%40mail.gmail.com.

[Kenji Baheux]

I agree with Jonathan that a spinner is not meaningful and that eventually it would be good to address these cases as well. In the meantime, a metric that works for most cases is already a major improvement over what we've been using so far.

Regarding the ordering of useful and usable, the reason I have them in that order is because as a user I can't know beforehand if a website is usable by the time it's useful: I would start reading and scrolling as soon as I see something useful on the screen, only to later realize that the site isn't quite yet usable (e.g. a reflow triggers while reading).

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/loading-dev/CA%2BJdOKWjzdfJnzkQL5HCvM%3DSnoBLC2Lc8b4K%3D6Q9LoF%2BdsYJRA%40mail.gmail.com.

[Paul Irish]

Kenji, I really like this user moment breakdown. 

1. “is it happening?” (i.e. time to first paint)

Splash screen, loading indicator, basically some feedback that communicates your navigation has started.  

2. “is it useful?” (i.e. time to first meaningful paint)

I agree with the earlier definitions on the thread. Generally focusing on the paints of above-the-fold body & headline text. And sometimes a large hero image is primary content.

3.  “is it usable?” (i.e. time to stable layout)

This one captures the key aspect of "the page feels ready to interact with".  

There's probably a few technical considerations to identify this moment:

• User-agent loading indicator is probably done
  
• Any almost-primary images/text has been painted.
• Above-the-fold layout has stabilized

But, I suppose there's a bit more on layout stability that isn't captured here.…

Ideally, there is very little change in the layout of links and actionable elements between the user moments 2 and 3. Worst case: every click target completely moves and is replaced with a different click target that navigates somewhere else.

So there may be a metric around layout stability that timestamps can't offer. Something that captures the movement of the layout during the entire load--from moment 2 to moment 3 to ten seconds after, when the page should be totally done.

I had just started a thread on lighthouse-core exploring if the Speed Index metric takes this into account. If not, I can imagine a standalone metric for layout stability to complement these others.

All-in-all, I do like breaking apart loading into these 3 key user moments. Then adding in measures of layout stability and main thread availability complete the picture for me.

[Colin Blundell]

A related question: Is there a meaningful distinction between "is it usable?" and "is it complete?" Kenji's categorization (which I find extremely insightful) is not explicitly tracking the latter.

--
You received this message because you are subscribed to the Google Groups "Chromium Loading Performance" group.
To unsubscribe from this group and stop receiving emails from it, send an email to loading-dev...@chromium.org.
To post to this group, send email to loadi...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/loading-dev/CAK-PPf1yHqNtoGq-xJfQVOCP2Dgr7%2Bmd_Ps_mHV7%3Dr0ACs%2B5KA%40mail.gmail.com.

[Nat Duca]

+yi who has been poking at this space too

On Fri, Feb 19, 2016 at 1:50 AM, Colin Blundell <blun...@chromium.org> wrote:

A related question: Is there a meaningful distinction between "is it usable?" and "is it complete?" Kenji's categorization (which I find extremely insightful) is not explicitly tracking the latter.

On Fri, Feb 19, 2016 at 3:36 AM Paul Irish <paul...@chromium.org> wrote:

Kenji, I really like this user moment breakdown. 

1. “is it happening?” (i.e. time to first paint)

Splash screen, loading indicator, basically some feedback that communicates your navigation has started.  

2. “is it useful?” (i.e. time to first meaningful paint)

I agree with the earlier definitions on the thread. Generally focusing on the paints of above-the-fold body & headline text. And sometimes a large hero image is primary content.

3.  “is it usable?” (i.e. time to stable layout)

This one captures the key aspect of "the page feels ready to interact with".  

There's probably a few technical considerations to identify this moment:

• User-agent loading indicator is probably done
  
• Any almost-primary images/text has been painted.
• Above-the-fold layout has stabilized

But, I suppose there's a bit more on layout stability that isn't captured here.…

Ideally, there is very little change in the layout of links and actionable elements between the user moments 2 and 3. Worst case: every click target completely moves and is replaced with a different click target that navigates somewhere else.

So there may be a metric around layout stability that timestamps can't offer. Something that captures the movement of the layout during the entire load--from moment 2 to moment 3 to ten seconds after, when the page should be totally done.

I had just started a thread on lighthouse-core exploring if the Speed Index metric takes this into account. If not, I can imagine a standalone metric for layout stability to complement these others.

All-in-all, I do like breaking apart loading into these 3 key user moments. Then adding in measures of layout stability and main thread availability complete the picture for me.

--
You received this message because you are subscribed to the Google Groups "Chromium Loading Performance" group.

To unsubscribe from this group and stop receiving emails from it, send an email to loading-dev+unsubscribe@chromium.org.

To post to this group, send email to loadi...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/loading-dev/CAK-PPf1yHqNtoGq-xJfQVOCP2Dgr7%2Bmd_Ps_mHV7%3Dr0ACs%2B5KA%40mail.gmail.com.

--

You received this message because you are subscribed to the Google Groups "Chromium Loading Performance" group.

To unsubscribe from this group and stop receiving emails from it, send an email to loading-dev+unsubscribe@chromium.org.

To post to this group, send email to loadi...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/loading-dev/CAMGE5NHeq7F_As1uR6Y35QBb4ZiiV3iYbCO8zTjegNSXytJ0pw%40mail.gmail.com.

[Kunihiko Sakamoto]

Thanks all for the comments.

I agree we should focus on these three key metrics.

The moment 1 is what we already have as time-to-first-paint.

The moment 2 is basically what I was trying to define in my document. It should detect paints of above-the-fold body & headline text (sometimes image).

Probably it should take into account the number of painted characters (per total characters in final state?), so that trivial text like this will be ignored. Of course it shouldn't count text painted out of initial viewport, or painted below the splash screen layer. I'll ask for a advice from paint team.

The moment 3 is a layout-based metric. There's some existing layout-related timings in WebMeaningfulLayout, although what this captures is somewhat different from what we want.

As Paul mentioned we may end up with a non-timestamp metric, but let's start with very simple one. A first-order approximation would be the time of the last layout change before WPT visual complete. I'll test it in some pages and see how well that works / how can it be improved.

2016-02-20 4:52 GMT+09:00 Nat Duca <nd...@chromium.org>:

+yi who has been poking at this space too

On Fri, Feb 19, 2016 at 1:50 AM, Colin Blundell <blun...@chromium.org> wrote:

A related question: Is there a meaningful distinction between "is it usable?" and "is it complete?" Kenji's categorization (which I find extremely insightful) is not explicitly tracking the latter.

On Fri, Feb 19, 2016 at 3:36 AM Paul Irish <paul...@chromium.org> wrote:

Kenji, I really like this user moment breakdown. 

1. “is it happening?” (i.e. time to first paint)

Splash screen, loading indicator, basically some feedback that communicates your navigation has started.  

2. “is it useful?” (i.e. time to first meaningful paint)

I agree with the earlier definitions on the thread. Generally focusing on the paints of above-the-fold body & headline text. And sometimes a large hero image is primary content.

3.  “is it usable?” (i.e. time to stable layout)

This one captures the key aspect of "the page feels ready to interact with".  

There's probably a few technical considerations to identify this moment:

• User-agent loading indicator is probably done
  
• Any almost-primary images/text has been painted.
• Above-the-fold layout has stabilized

But, I suppose there's a bit more on layout stability that isn't captured here.…

Ideally, there is very little change in the layout of links and actionable elements between the user moments 2 and 3. Worst case: every click target completely moves and is replaced with a different click target that navigates somewhere else.

So there may be a metric around layout stability that timestamps can't offer. Something that captures the movement of the layout during the entire load--from moment 2 to moment 3 to ten seconds after, when the page should be totally done.

I had just started a thread on lighthouse-core exploring if the Speed Index metric takes this into account. If not, I can imagine a standalone metric for layout stability to complement these others.

All-in-all, I do like breaking apart loading into these 3 key user moments. Then adding in measures of layout stability and main thread availability complete the picture for me.

--
You received this message because you are subscribed to the Google Groups "Chromium Loading Performance" group.

To unsubscribe from this group and stop receiving emails from it, send an email to loading-dev...@chromium.org.

To post to this group, send email to loadi...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/loading-dev/CAK-PPf1yHqNtoGq-xJfQVOCP2Dgr7%2Bmd_Ps_mHV7%3Dr0ACs%2B5KA%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "Chromium Loading Performance" group.

To unsubscribe from this group and stop receiving emails from it, send an email to loading-dev...@chromium.org.

To post to this group, send email to loadi...@chromium.org.