Site is not generating enough CRUX data - intermittent issue... what am I doing wrong?!

[Joel]

Basics covered:

1. Not a new website (5+ year old e-commerce site)

2. Lots of users and user activity (in 2022 we had 308K views with 127K unique visitors

      - On our slowest month we still had 1,000 unique visitors with 6,400 page views.        Super high 30-45% sales conversion rate. Average order size is $100+.)

3. Crawlable/Indexable (All pages are indexed except for cart, checkout and account pages. Robots.txt file allows entire site to be crawled. I tried to disallow the wp-admin pages for a month to see if maybe I needed to try and increase the crawl budget but it didn't help. Have everything submitted in search console, sitemaps include all pages and URL's are accurate.)

4. PageSpeed/Performance (Desktop has always been nearly 100 on all 4 metrics, never under 90. Mobile speeds have been 90+ on all 3 metrics except for performance on product pages, which has stayed between 40-60. That is due to product images not being able to pre-load or avoid lazy load in Storefront theme.  Has always been consistent in 2022 and early 2023... so wouldn't explain any intermittent issues being included in CRUX report. We didn't have any decrease in 'good' pages in search console, and barely ever have any 'bad' pages in the past year. If there are it was 1-2 pages and they were fixed within a couple days.) A few weeks ago I got all of the scores up to 90-100 on mobile, and still haven't been included in the report.

5. Good pages/mobile experience/https on search console (Core web vitals, mobile usability, and HTTPS was actually lower in the months that we were included in the CRUX data report. It has improved a lot in March and April and still not included in the report. More details: We had roughly 30% good mobile page experience in Jan/Feb when we were included in the CRUX report, in March and April I was able to get it up to 95% and still not generating CRUX data. Desktop was 40-60% in Jan/Feb when we were included in Crux report, in March and April the data disappeared and we weren't generating any report info. Speeds for desktop have improved too). 

6. Intrusive Interstitials (The only items we use are the 'trustedsite' badge in the lower right corner, google customer reviews star badge in the lower left corner, and Yotpo product reviews tab in the lower middle screen. The Yotpo and Google customer reviews have been used for several years, but the trustedsite badge we added around December).

Issue:

Our website is only occasionally making the CRUX reports, and seems to be with no rhyme or reason. In the past year we were only included in the core web vitals/CRUX data in October, January and February. FID and INP only showed in October and January. In 2022 we had 308K views and 127K visitors... with nothing remarkable happening around October.. Our number of impressions and page views stay the same every month since it's always just return customers. There weren't any changes made to the site during the months that we were successfully included in the CRUX data report. In late February and early March I was working on the site and had used the 'purge cache' option a lot, and was concerned maybe that slowed the site down enough for users to hurt web vitals, but from what I read, clearing the cache wouldn't cause any changes. And logically that wouldn't explain why we were excluded so many months in 2022 when we were generating equal (or more) user data (without any purges). In 2022, I made very few edits/purges on the site.  It seems like when we do get included in the CRUX report, our Google rankings are up 800% higher than the months when we don't, so it's a huge impact.

- Host/Server:  Siteground w/ CDN, Optimizer plugin w/ caching, premium CDN. They are on PHP 7.4.33 and say it is the latest 'stable version'. But they have the option to update it to 8.2.4 if anyone thinks that could be the cause?

If anyone has any advice/info, I would be extremely, extremely grateful...   and thanks in advance to anyone who responds!

[Joe]

Anyone have any ideas on things that might be hurting visibility in Crux data?  Would clearing cookies too often or issues with caching be a possible cause? Or possibly an issue with headers?

[Barry Pollard]

The eligibility criteria are listed here: https://developer.chrome.com/docs/crux/methodology/#eligibility

As you are intermittently showing, that suggests you're meeting all the criteria except for the "Sufficiently Popular" criteria. We don't make the exact levels for this public, but do remember it only applies to certain Chrome users: https://developer.chrome.com/docs/crux/methodology/#user-eligibility. So if you get a lot of visitors from Safari or Firefox, or more privacy conscious users, then you may not meet the "sufficiently popular" criteria.

It is also worth checking out the CrUX BigQuery and API calls without specifying a form factor (DESKTOP or PHONE). We may not have sufficient popularity to split by those, but may still have it at an overall loevel but many tools (including PageSpeed Insights and the CrUX Dashboard) only work based on DESKTOP or PHONE data.

[Joel]

Thanks for the reply!

Does it hurt at all to have too many pages indexed? For example,
having some pages indexed that are new and not gaining a lot of views?

Would indexing the my-account, and cart pages help increase the amount
of Crux data?

On 5/5/23, Barry Pollard <barryp...@google.com> wrote:
> The eligibility criteria are listed
> here: https://developer.chrome.com/docs/crux/methodology/#eligibility
>
> As you are intermittently showing, that suggests you're meeting all the
> criteria except for the "Sufficiently Popular" criteria. We don't make the
> exact levels for this public, but do remember it only applies to certain
> Chrome
> users: https://developer.chrome.com/docs/crux/methodology/#user-eligibility.
>
> So if you get a lot of visitors from Safari or Firefox, or more privacy
> conscious users, then you may not meet the "sufficiently popular" criteria.
>
> It is also worth checking out the CrUX BigQuery and API calls without
> specifying a form factor (DESKTOP or PHONE). We may not have sufficient
> popularity to split by those, but may still have it at an overall loevel
> but many tools (including PageSpeed Insights and the CrUX Dashboard) only
> work based on DESKTOP or PHONE data.
>
> On Friday, May 5, 2023 at 2:39:46 PM UTC+1 Joe wrote:
>
>> Anyone have any ideas on things that might be hurting visibility in Crux
>> data? Would clearing cookies too often or issues with caching be a
>> possible cause? Or possibly an issue with headers?
>>
>> On Sunday, April 23, 2023 at 4:00:18 AM UTC-4 Joe wrote:
>>

>>> *Basics covered:*
>>> 1. *Not a new website* (5+ year old e-commerce site)
>>>
>>> 2. *Lots of users and user activity* (in 2022 we had 308K views with

>>> 127K unique visitors
>>> - On our slowest month we still had 1,000 unique visitors with
>>> 6,400 page views. Super high 30-45% sales conversion rate. Average
>>>
>>> order size is $100+.)
>>>

>>> 3. *Crawlable/Indexable* (All pages are indexed except for cart,

>>> checkout and account pages. Robots.txt file allows entire site to be
>>> crawled. I tried to disallow the wp-admin pages for a month to see if
>>> maybe
>>> I needed to try and increase the crawl budget but it didn't help. Have
>>> everything submitted in search console, sitemaps include all pages and
>>> URL's are accurate.)
>>>

>>> 4. *PageSpeed/Performance* (Desktop has always been nearly 100 on all 4

>>> metrics, never under 90. Mobile speeds have been 90+ on all 3 metrics
>>> except for performance on product pages, which has stayed between 40-60.
>>>
>>> That is due to product images not being able to pre-load or avoid lazy
>>> load
>>> in Storefront theme. Has always been consistent in 2022 and early
>>> 2023...
>>> so wouldn't explain any intermittent issues being included in CRUX
>>> report.
>>> We didn't have any decrease in 'good' pages in search console, and barely
>>>
>>> ever have any 'bad' pages in the past year. If there are it was 1-2 pages
>>>
>>> and they were fixed within a couple days.) A few weeks ago I got all of
>>> the
>>> scores up to 90-100 on mobile, and still haven't been included in the
>>> report.
>>>

>>> 5. *Good pages/mobile experience/https on search console* (Core web

>>> vitals, mobile usability, and HTTPS was actually lower in the months that
>>>
>>> we were included in the CRUX data report. It has improved a lot in March
>>>
>>> and April and still not included in the report. More details: We had
>>> roughly 30% good mobile page experience in Jan/Feb when we were included
>>> in
>>> the CRUX report, in March and April I was able to get it up to 95% and
>>> still not generating CRUX data. Desktop was 40-60% in Jan/Feb when we
>>> were
>>> included in Crux report, in March and April the data disappeared and we
>>> weren't generating any report info. Speeds for desktop have improved
>>> too).
>>>

>>> 6.* Intrusive Interstitials* (The only items we use are the

>>> 'trustedsite' badge in the lower right corner, google customer reviews
>>> star
>>> badge in the lower left corner, and Yotpo product reviews tab in the
>>> lower
>>> middle screen. The Yotpo and Google customer reviews have been used for
>>> several years, but the trustedsite badge we added around December).
>>>
>>>

>>> *Issue:*

[Barry Pollard]

CrUX does not show data for non-indexed pages, but has no concerns over having too many indexed pages.

Would indexing the my-account, and cart pages help increase the amount of Crux data?

No it would not.

[FID Racing]

Hi Barry,

My website fid-racing.com has been online for more than a year. Its monthly traffic is around 1000 users. It's indexed and crawlable. Page Speed Insight score is not bad. But there is no Crux report at all. Could you help look into this? Thanks

I took several screenshots for your viewing. 

[❄ Johannes Henkel]

This origin is in the CrUX dataset, however, there's no phone or desktop specific data for most months - only 'overall' data.

Reason is a bit convoluted, but basically, not quite enough data to offer the breakdown into these dimensions (https://developer.chrome.com/docs/crux/methodology/#filtering).

Unfortunately, our user interfaces don't usually show this overall data.

Ways to get this data:

1) You can use BigQuery, I think this query is fairly convenient.

select * from `chrome-ux-report`.materialized.metrics_summary where origin = 'https://www.fid-racing.com'

You can also get full histograms, e.g. by querying `chrome-ux-report`.all.202307 or by querying experimental.global.

select * from `chrome-ux-report`.all.202307 where origin = 'https://www.fid-racing.com'

This third party (not Google) site is a nice example for visualizing the data:

https://treo.sh/sitespeed/www.fid-racing.com

2) CrUX API and CrUX History API; REST APIs.

https://developer.chrome.com/docs/crux/methodology/#tool-crux-api

https://developer.chrome.com/docs/crux/methodology/#tool-crux-history-api

For the history API, we have a Colab which can make simple graphs, this is easy to run (and if you like Python you can start programming / scripting from there :-)), the main hurdle is to get an API key - free and fairly painless if you have a Google Cloud project or willing to set one up.

When running the colab, be sure to select the form_factor ALL in the form.

https://colab.sandbox.google.com/github/GoogleChrome/CrUX/blob/main/colab/crux-history-api.ipynb

This third party (not Google) site is a nice example for visualizing the data:

https://tamethebots.com/tools/cwv-history (make sure you select device type 'all')

--
You received this message because you are subscribed to the Google Groups "Chrome UX Report (Discussions)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chrome-ux-repo...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chrome-ux-report/bfe147cf-63e4-41d0-8dfc-57d9829bb151n%40chromium.org.

--

johannes :-)

[FID Racing]

Thanks Johannes

It's very helpful.

I have a concern now. Will this Crux result affect the ranking of my site?

Thanks.

[❄ Johannes Henkel]

Hi!

Unfortunately, this team can't answer any questions on the Search side, so you'd need to contact that team to understand that.

I think this would be the right place to ask the question: https://support.google.com/webmasters/community?hl=en

Good wishes!

--

johannes :-)