Discrepancies between the Chrome UX Report API and PGSI

[François-Xavier Brottier]

Hi,

While using the Chrome UX Report API and comparing it with the PGSI RUM data, I came across discrepancies when querying some URLs (not origin).

For example, when looking at this page on desktop I have the following percentile results for the strategy/formFactor desktop :

• FCP
  • pgsi 1734
  • crux report 1384
• LCP
  • pgsi 4030
    
  • crux report 3275
• FID
  • pgsi 34
  • crux report 25
• CLS
  • pgsi 15 (0.15)
    
  • crux report 0.14 (14)

Here is what I am running for PGSI: https://www.googleapis.com/pagespeedonline/v5/runPagespeed?category=performance&strategy=desktop&url=https://www.cdiscount.com/telephonie/telephone-mobile/apple-iphone-11-noir-128-go/f-144043124-iphone11128gonoi.html

For the chrome ux report api the body data is as follow :

{
  "formFactor": "DESKTOP",
   "url": "https://www.cdiscount.com/telephonie/telephone-mobile/apple-iphone-11-noir-128-go/f-144043124-iphone11128gonoi.html"
}

Given what I understand about both APIs, I did not expect these differences. Can you help me understand where they could come from?

I am also coming across another unexpected result, maybe related to the previous point: for this page PGSI won't return field data anymore (non-origin data I mean) for mobile and desktop while the chrome ux report api is still able to do so for both phone and desktop.

Could both discrepancies be related? What did I miss about these APIs?

Thanks for your help,

Best regards,

François-Xavier

[François-Xavier Brottier]

I am doing a little more digging so I ended up with a dashboard to compare both APIs

You will find below the results for another page on phone/mobile, and also the results for an amazon url with less discrepancies, maybe it has to do with popularity?

The trends appear to be similar which is great :)

Best regards

FX

[Rick Viscomi]

FX, thanks for the additional info! While some variation between APIs can be expected (the data sources are not identical, which we hope to do soon) this magnitude of discrepancy is surprising. I'll forward this info to the relevant teams to investigate and I'll get back to you. Meanwhile, if you had to pick, I'd recommend relying on the CrUX API data as the source of truth.

By the way, that dashboard looks really nice! Is that something other CrUX users could use?

Rick

[François-Xavier Brottier]

Thanks Rick for the details, we will focus on the data from the crux api report api.

As for the dashboard I will ask my team if it's possible to share it, if so I'll open a separate thread or send it to the crux twitter account.

FX

[Max Prin]

Thanks for the post, FX, and for the details, Rick, as I was also looking for an explanation. I wanted to add that I'm also seeing discrepancies (smaller though) when looking at the Origin data, not just URLs.

Based on your comment, Rick, we can expect the metrics in PSI to eventually match those of the CrUX API - and not the other way around, correct?

Thanks again,
Max 

[Rick Viscomi]

Hi Max,

That's correct, the plan is to make the CrUX API the data source for field data in PSI. No ETA on that yet.

Rick

[Max Prin]

Sounds good, thanks, Rick!

[Rockey Nebhwani]

Rick,

We have been monitoring FCP from CRUX APIs for Origin https://direct.asda.com and form factor 'PHONE' and this is how progression looks like for FCP (in chronological order.. reading taken at regular intervals)

As you can see for today 4th Dec, this matches with Addy's tool

But this is not what we see in CRUX Dashboard for this origin

CRUX dashboard is showing FCP 20% ahead of CRUX API. Now I know that technically 71.52% in CRUX dashboard for Oct-2020 is for the month of Oct-2020 where as CRUX API is 28 day rolling average but 2 nearest readings around 31st Oct which I have are on 27th Oct (47.06%) and 4th Nov (46.99%). 

Why CRUX API doesn't co-relate with what we see in CRUX Dashboard for FCP?

My team has readings for same dates for LCP/FID/CLS and those seem to match with CRUX dashboard?

Any idea?

Thanks,

Rockey

[Rockey Nebhwani]

I am not sure why screenshots attached by me are not displaying in previous message so trying again

Rick,

We have been monitoring FCP from CRUX APIs for Origin https://direct.asda.com and form factor 'PHONE' and this is how progression looks like for FCP (in chronological order.. reading taken at regular intervals)

As you can see for today 4th Dec, this matches with Addy's tool 

But this is not what we see in CRUX Dashboard for this origin 

CRUX dashboard is showing FCP 20% ahead of CRUX API. Now I know that technically 71.52% in CRUX dashboard for Oct-2020 is for the month of Oct-2020 where as CRUX API is 28 day rolling average but 2 nearest readings around 31st Oct which I have are on 27th Oct (47.06%) and 4th Nov (46.99%). 

Why CRUX API doesn't co-relate with what we see in CRUX Dashboard for FCP?

My team has readings for same dates for LCP/FID/CLS and those seem to match with CRUX dashboard?

Any idea?

Thanks,

Rockey

[Rockey Nebhwani]

Rick,

We have been monitoring FCP from CRUX APIs for Origin https://direct.asda.com and form factor 'PHONE' and this is how progression looks like for FCP (in chronological order.. reading taken at regular intervals)

As you can see for today 4th Dec, this matches with Addy's tool 

But this is not what we see in CRUX Dashboard for this origin

CRUX dashboard is showing FCP 20% ahead of CRUX API. Now I know that technically 71.52% in CRUX dashboard for Oct-2020 is for the month of Oct-2020 where as CRUX API is 28 day rolling average but 2 nearest readings around 31st Oct which I have are on 27th Oct (47.06%) and 4th Nov (46.99%). 

Why CRUX API doesn't co-relate with what we see in CRUX Dashboard for FCP?

My team has readings for same dates for LCP/FID/CLS and those seem to match with CRUX dashboard?

Any idea?

Thanks,

Rockey

[Rick Viscomi]

The API uses thresholds of 1 and 3 seconds for good/poor FCP:

      "first_contentful_paint": {

        "histogram": [

          {

            "start": 0,

            "end": 1000,

            "density": 0.52068097350437681

          },

          {

            "start": 1000,

            "end": 3000,

            "density": 0.42704711665268014

          },

          {

            "start": 3000,

            "density": 0.052271909842944511

          }

        ]

The CrUX Dashboard is based on the `chrome-ux-report.materialized.device_summary` table, which uses 1.5 and 2.5 seconds as the good/poor FCP thresholds, as announced earlier in the year.

This misalignment is a suboptimal experience and we'll be taking a look at getting these thresholds in sync. Thanks for raising this issue, Rockey.

[Rockey Nebhwani]

Thanks Rick. This explains and I will keep this in mind till these are synced.