testmysite tool supposed to use CrUX data - doesn't make sense

[Kristian Sköld]

Hi, 

I'm trying to understand how the CrUX data is actually used in the testmysite tool.

According to the English documentation the site speed is the 75th percentile of the LCP based on CrUX data of the last month (currently Nov 2020). 

However, every site I ran does show quite different results in testmysite site speed compared to the actual CrUX data as it for example is reported in the origin summary on pagespeed insights (which is the 4 weeks rolling window) or calculated directly from the data in BigQuery.

I specifically chose german sites, since I know the traffic is primarily from german users and then using the country_de CrUX table compared to the Country selection Germany in testmysite should match.

site testmysite CrUX BigQuery PSI

galeria.de 4.1 3.7 3.7

cyberport.de 4.6 4.8 5.1

Tchibo.de 4.1 3.9 3.8

Saturn.de 3.3 2.9 3

thomann.de 2.2 2.5 2.5

lidl.de 2.5 2.9 2.9

As you can see BQ and PSI are always very close, but testmysite varies +-400ms - and that's quite a lot especially when you want to use the ROI calculator in testmysite.

Curiously the German documentation for testmysite still states that the the site speed is determined by the 90th percentile of the FCP taken from CrUX data Nov 2020 !?

But they don't match the testmysite results at all.

Any idea, where the testmysites results are actually coming from?

It's really confusing, since our own RUM data matches the CrUX data very nicely, but trying to use the ROI calculator is pretty useless, if the initial input site speed is already off. 

I can use a "hack" by simply entering a none existent domain, since the calculator then offers me to enter both before and after speed values - but that seems a bit crude.

Would really love to get a better understanding around this, since all those tools can be really powerful - but also mighty confusing.

Thanks,

Kristian

[Rick Viscomi]

Hi Kristian,

Expand the "Understanding the speed tools" section of the Test my Site documentation for more info about the benchmarking tool:

"Why does the rank look out of order in the benchmarking tool? Why is the benchmarking tool not showing a tie even though two sites have the same site speed? The site speed metric for each domain shows 75th percentile Largest Contentful Paint (LCP). However, rank is based on a combination of three metrics — LCP, DOMContentLoaded, and onload — not just LCP alone. As a result, sites with slower LCP times may rank higher than sites with the same or faster LCP times."

Hope that helps,

Rick

[Kristian Sköld]

Thank you Rick, 

but that section is referring to the benchmark ranking.

The data I used and listed above from testmysite is the speed shown in the ROI calculator.

I just listed a number of different results from different sites I ran individually and did not use the benchmarking tool of test my site.

And under the "understanding the speed tools" section it says:

How is revenue uplift calculated?

The potential increase in annual revenue is estimated using a formula based on data from Google Analytics accounts that have chosen to share anonymous aggregated data with Google for benchmarking purposes. The speed metric shown is 75th percentile Largest Contentful Paint (LCP)....

So I would expect to then also get the 75pctl speed reported in the ROI calculator and it to either match the number I get out of the CrUX database for the same month or the rolling window of 28days as it's being reported in the PSI data.

But the number matches neither.

Also when I just reran testmysite with the same parameters I'm getting different results compared to my run on Dec 18th  - although again according to the documentation the data should be based on the CrUX data Nov 2020.

(well actually, it says Nov 2021 now in the documentation, but I take that as a glitch :) )

e.g. always using Germany and 4G

lidl.de: Dec 18 - 2.5s  Jan 5 - 3s

galleria.de: Dec 18 - 4.1s. Jan 5 - 3.8s