Understand the metrics of Chrome UX Report API vs web-vitals JavaScript library

[Tobias Willmann]

Hi I want to understand the CrUX metrics better. 

We are using the JavaScript library to to track an event with every pageview. Which I understand. I have X pageviews and Y% of the X pageviews had e.g. a poor FID

But what do I get in Chrome UX Report API. Users? 

The documentation says here https://developers.google.com/web/tools/chrome-user-experience-report/api/reference#metric

Metrics are expressed in a histogram, which represents the percent of users that experienced a metric with that value proportionally to all.

A simple three bin histogram for metric "ExampleMetric" looks like this.

{
  "histogram": [
    {
      "start": 0,
      "end": 1000,
      "density": 0.38179089544772343
    },
    {
      "start": 1000,
      "end": 3000,
      "density": 0.49904952476238057
    },
    {
      "start": 3000,
      "density": 0.11915957978989571
    }
  ],
}

This data indicates that 38.2% of users experience an ExampleMetric value between 0ms and 1000ms. The units of ExampleMetric are not contained in this histogram, in this case we will assume milliseconds.

But what about a user who experienced a good and a poor FID score a few hours later? I guess these users are not listed in both buckets? 

Here by the way https://developers.google.com/web/tools/chrome-user-experience-report#data-format it's about page loads. This sounds similar to the JavaScript tracking. 

For example, the above shows a sample record from the Chrome User Experience Report, which indicates that 12.3% of page loads had a “first paint time” measurement in the range of 1000-1200 milliseconds when loading “http://example.com” on a “phone” device over a ”4G”-like connection.

It's BigQuery here. I'm more curious about the API for now. 

Do I miss some important part in the documentation? How are the shares of users calculated if a user can have poor scores and good scores with another session. 

Thanks

[Rick Viscomi]

Hi Tobias,

All UX data in CrUX are aggregated per page load, not users.

We will correct the API docs you cited. Thanks for pointing that out!

Rick

[Aaron Alter]

Hi Rick,

Hope it's okey if I "hijack" this topic to ask a somewhat related question that just came up in our organisation: Does CrUX collect data for each page load or only for certain scenarios (e.g. entrance from Google)? To me it seems logical that every single page load is measured even if they happen consecutively in a series (such as navigations within a page), is that assumption correct? 

 

Asking because our properties run on React and the inital navigation to a page is always slower than subsequent ones. Until we start tracking WebVitals ourselves, we're relying on CrUX data and need to be sure it's as accurate representation of the users experience as possible. :)

Cheers 

Aaron

[Tobias Willmann]

That's a good question. I'm wondering here why our own javascript tracking with Tag Manager differs from the CrUX numbers. If it's not each page load this could be an explanation. Best Tobi

--
You received this message because you are subscribed to a topic in the Google Groups "chrome-ux-report" group.
To unsubscribe from this topic, visit https://groups.google.com/a/chromium.org/d/topic/chrome-ux-report/WPns-HwDjac/unsubscribe.
To unsubscribe from this group and all its topics, send an email to chrome-ux-repo...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chrome-ux-report/090250de-cfb2-46a9-85d9-c7bb8c23d1a7n%40chromium.org.

--

Tobias Willmann
Head of SEO

Ringier AG
Blick-Gruppe
Dufourstrasse 23
8008 Zürich
Schweiz

M +41 79 438 43 70

tobias....@ringier.ch

www.blick.ch | www.ringier.com

    

[Rick Viscomi]

Does CrUX collect data for each page load or only for certain scenarios (e.g. entrance from Google)? To me it seems logical that every single page load is measured even if they happen consecutively in a series (such as navigations within a page), is that assumption correct? 

The problem I think you're describing has to do with SPAs in which "soft navigations" between pages are not counted by CrUX. This is a known limitation not only in CrUX but all RUM tools and web platform APIs. For example, there is no second onload event after a soft navigation because the platform APIs already consider the page to be loaded from the first navigation. Because we lack a standardized way of determining when a soft navigation occurred and how to reset the performance timeline, unfortunately CrUX is blind to these experiences.

If your question is not specific to SPAs, then yes all page loads are counted equally, regardless of referrer or anything.

Rick

To unsubscribe from this group and all its topics, send an email to chrome-ux-report+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chrome-ux-report/090250de-cfb2-46a9-85d9-c7bb8c23d1a7n%40chromium.org.

[Tobias Willmann]

Thanks for me this explains the difference. We use React... :-)

To unsubscribe from this group and all its topics, send an email to chrome-ux-repo...@chromium.org.

[Aaron Alter]

Thanks for the detailed answer Rick, Exactly what we needed! :)

[Dave Smart]

Sorry for kinda barging in to an old thread!

The problem I think you're describing has to do with SPAs in which "soft navigations" between pages are not counted by CrUX.  

Does this mean that given the scenario, a  top navigation occurs to example.com/foo, then a soft navigation occurs to example.com/bar CrUX could:

• Get data for the top navigation to /foo, but be blind to what happens after, i.e. the soft nav. to /bar
• Get 'hybrid' data for /foo & /bar, ascribed to /foo. In terms of vitals I'm guessing LCP & FID would be accurate to /foo, but CLS could be /foo & /bar combined
  
• Be blind to /foo & /bar totally?

[Rick Viscomi]

It's the hybrid scenario. As far as Chrome and CrUX are aware, SPA usage is one long page view, even though to the user they've visited two or more pages. Here's what that means for each metric:

FID reflects the first user input, but I can't imagine a scenario where a page has a soft navigation before any user input, so it should only apply to the first page experience.

LCP is updated throughout the page view, but only until the first user interaction, so it should also only apply to the first page experience.

CLS is accumulated throughout the page view, but there is no hard stop, so it can continue to grow indefinitely across soft navigations.

Also consider infinite-scrolling web apps like Reddit or Twitter. You can use these apps and never have a soft navigation, but still have a similarly long-lived session. CLS would still continue to grow indefinitely as you scroll. But since you can scroll without triggering FID, it may happen very late in the session or not at all. LCP would only apply to the initial viewport state before scrolling.

Core Web Vitals are updated annually, so I look forward to future iterations better capturing user experiences in all the ways developers build apps.

[Kristian Sköld]

Since this topics has moved into the realm of SPA I’d like to comment on that topic as well. I know it has been discussed a number of times before. 

@Rick: I beg to differ regarding  "This is a known limitation not only in CrUX but all RUM tools”. Various RUM solutions like mPulse and also our own have solved this by hooking into the history api. In basically all SPA frameworks this API can be used to identify a beginning of a soft navigation (yes, it’s not always 100% precise) and using various tricks we can do a pretty good estimation of when a soft navigation ends. Nic Jansma has done some write up on hat topic (e.g. as a primer these slides https://nicj.net/talks-files/measuring-the-performance-of-single-page-applications/presentation/). So shouldn’t it be even easier for the Chrome team, having access to a lot more internal details of what happens in the browser, to identify such soft navigations?

Then measuring the Web Vitals should also be fairly easy I assume, since LCP, FID CLS could all be measured from that soft navigation starting point. 

It would be extremely useful and make the sites which are using SPAs in a good way look much less as abysmal as they typically do today in the CrUX data. 

I very often have to explain why sites with very fast user experiences have such bad CrUX results: because they are SPAs.

Would love to see some progress regarding SPA performance API in Chrome and of course in CrUX as well.

Kristian

You received this message because you are subscribed to the Google Groups "Chrome UX Report (Discussions)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chrome-ux-repo...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chrome-ux-report/579a5dbd-d638-478f-9ddc-6fac5307dbc6n%40chromium.org.

[Rockey Nebhwani]

@rick - Thanks for explaining this. So metric which is problematic in case of SPAs is CLS. Can I clarify a scenario about sites with AMP?

Let's take an example where a site is on www.example.com and they have AMP version on amp.example.com. Customer see AMP page from Google but when they click on any click, they are transitioned to www.example.com. Am I correct I seeing in that all core web vitals (including CLS) should be pretty accurate for origin amp.example.com as there is always only one page in user journey on that origin. 

[Dave Smart]

@rick, thank you for your excellent, throughout reply!

[Rick Viscomi]

Kristian, I'd recommend watching the June 4th meeting of the Web Performance Working Group in which Ilya Grigorik and Michal Mocny described some of the challenges of standardizing SPA-compatible performance APIs and how a possible solution might work. One of the guiding principles of the CrUX dataset is that it's grounded in standardized APIs, so we're eagerly looking to the WebPerfWG for progress on this issue.

Rockey, are cross-origin soft navigations possible? If so that'd be an interesting edge case and I'm not sure how CLS would be counted.

[Ishan Anand]

Rick, thanks for the deep links. I had heard Ilya's part of that video before but not Michal's and have reached out to him. I appreciate the challenge of the problem but it's critical that it be addressed given the SEO impact of CrUX. 

To add to the discussion, two other things to note:

• CLS (as Rick has pointed out) gives a leeway on shifts within 500ms of a user tap so if the SPA can transition in that time it helps.
• One side effect is that if your traffic is a mix of SPA and MPA pages, your SPA pages will appear to shrink in traffic relative to your MPA pages in CrUX. Our team discussed this in a blog post https://www.moovweb.com/post/google-penalizes-pwas-spas

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chrome-ux-report/c15faf30-28ba-4c2e-b6af-278f40224481n%40chromium.org.

--

Ishan Anand

Moovweb

340 Pine Street, Suite 400

San Francisco, CA 94104

Cell: +1-415-335-6094 (sms/texts are welcome)

Calendar: http://bit.ly/2jv5t0v