How to run CRuX API query in batch?

[E. O.]

Could somebody give me an example of batch query of CRuX API? I tried to use examples from documentation page but had no luck with them .

I've tried it like on the screenshot - but got only 404 and nothing comparable with response from the documentation:

[Rick Viscomi]

Hi,

I've replied to your question on Stack Overflow. Linking here in case anyone else finds it useful: https://stackoverflow.com/a/67477488/1022333

Rick

--
You received this message because you are subscribed to the Google Groups "Chrome UX Report (Discussions)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chrome-ux-repo...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chrome-ux-report/7b5c534c-3807-4298-89d6-4c5163662c0dn%40chromium.org.

[Rick Viscomi]

We have a little JavaScript utility on GitHub to assist with using the CrUX API, so I've added a `batch` method to alleviate some of the challenging parts of making batch requests and parsing batch responses. See it here: https://github.com/GoogleChrome/CrUX/blob/f08abc674aea1d835423370a9c7bcea2e2666cd0/js/crux-api-util.js#L26-L72

I've also updated my Stack Overflow answer to provide an example of using the util method and its response. Hopefully that helps!

Rick

[Aleksey Kulikov]

There's also a tiny battle-tested crux-api library that supports batch requests, handles errors (429 & 404), and provides types.

import { createBatch } from 'crux-api/batch'

const batch = createBatch({ key: CRUX_API_KEY })

const records = await batch([

  { url: 'https://github.com/', formFactor: 'MOBILE', effectiveConnectionType: '4G' },

  { url: 'https://github.com/marketplace', formFactor: 'DESKTOP' },

  { url: 'https://www.github.com/explore', formFactor: 'TABLET' },

  // ... up to 1000 records.

])

[Ryan Siddle]

You cannot batch 1000 requests as that would require 1000 queries. The rate limiter is set to 150 queries per 60 seconds (as shown on your project Quotas page). The batch API simply wraps all of the HTTP requests that would normally be sent across the network individually, as one request. The downside of batching this way is that you need additional logic to extract all of the results and resend any failures in a retry queue.  Batching can also be inefficient depending on your business workflow. If your script goes over the rate limit by one request, then the entire batch receives a 429 error. We’re checking a large amount of requests each day for our clients and opted for single HTTP requests over batch.

 

Ryan Siddle / Digital Director
ryan....@merj.com

Merj
+44 (0) 2033 222 660 / Ext: #101
9th Floor, 1 Ropemaker Street, London, EC2Y 9HT
merj.com

--
You received this message because you are subscribed to the Google Groups "Chrome UX Report (Discussions)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chrome-ux-repo...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chrome-ux-report/7b5c534c-3807-4298-89d6-4c5163662c0dn%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "Chrome UX Report (Discussions)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chrome-ux-repo...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chrome-ux-report/6fb918e2-fb38-48dd-a991-3b1d289de507n%40chromium.org.

[Aleksey Kulikov]

Hi Ryan, you actually can batch 1000 requests without hitting the rate limit. I've just tested two batches with 312 and 720 records and got no issues (script: https://github.com/treosh/crux-api/blob/main/script/batch-limits.js). I understand this is a subject for change since recently CrUX API changed the limits from 25k req/day to 150 req/minute. But now it works as stated in the documentation.

Building the crux-api library, we had the same objections as you – the library keeps requests order and handles 429. 

I agree that you need to keep in mind the API rate limits because it's easy to exhaust them and get blocked for an extended period.

[E. O.]

Thank you, this helped me a lot - i've got an expected response using CURL. Could you help me to sort this cURL query?

curl -d '

--Batch

POST /v1/records:queryRecord?key=my_key HTTP/1.1

Content-ID: item2

Content-Type: application/json

{"url":"https://www.example.com", "metrics": ["first_contentful_paint"]}

--Batch--

' -H 'Content-Type: multipart/mixed; boundary="Batch"' -X POST -s https://chromeuxreport.googleapis.com/batch/ -v

POST request URL: 

https://chromeuxreport.googleapis.com/batch/

Headers: 

Content-ID: item2

Content-Type: application/json

Content-Type: multipart/mixed; boundary="Batch"

POST body:

{"url":"https://www.example.com", "metrics": ["first_contentful_paint"]}

Q: What means the line "POST /v1/records:queryRecord?key=my_key HTTP/1.1"? Is here maybe any parameter missed, like -u? How should i handle this line in terms of request, headers, body?

[Rick Viscomi]

> Q: What means the line "POST /v1/records:queryRecord?key=my_key HTTP/1.1"? Is here maybe any parameter missed, like -u? How should i handle this line in terms of request, headers, body?

The Format of a batch request section of the CrUX API docs explains that the body of the batch request contains nested HTTP requests that the server will parse using the `boundary` delimiter:

"A batch request is a single standard HTTP request containing multiple Chrome UX Report API calls, using the multipart/mixed content type. Within that main HTTP request, each of the parts contains a nested HTTP request."

So there are no extra cURL flags needed for the nested requests, only the outer one.

Rick

[E. O.]

I'm still can't get it work. If I use as POST body 

POST /v1/records:queryRecord?key=my_key

{"url":"https://www.allianz.de", "metrics": ["first_contentful_paint"]}

I get an error "invalid POST body".

If I combine a request url like https://chromeuxreport.googleapis.com/batch//v1/records:queryRecord?key=my_key - I get just 404.

[Rick Viscomi]

Your cURL command should look something like this:

curl https://chromeuxreport.googleapis.com/batch/ -H 'Content-Type: multipart/mixed; boundary="BATCH_BOUNDARY"' -d '--BATCH_BOUNDARY

POST /v1/records:queryRecord?key=YOUR_API_KEY

{"url":"https://www.allianz.de","metrics":["first_contentful_paint"]}

--BATCH_BOUNDARY--'

I've tried to simplify the command as much as possible while still returning a valid response. The whitespace before and after the POST is required.

[vivek shekhar singh]

Can somebody suggest overall how many different or multiple batch calls of 1000 requests each  i can make ? Is there a daily quota . My use case is for a huge website with aprrox 80-90 K pages . I wish to use the batch api to output an offline excel report of web vitals . Also is the batch support removed from this  https://github.com/treosh/crux-api library . If yes can you help me figure out how can i use the same to meet the above use case. I am currently adding 1000 requests as 1 batch and adding a timeout of 1 minute b/w each batch . Seems to be working fine for 3000 URLs . Please advise.

[Rick Viscomi]

You can get more info about your quota (current usage and limits) at https://console.cloud.google.com/apis/api/chromeuxreport.googleapis.com/quotas.

I'd recommend filtering down your 80,000 pages to only the ones with the most traffic that are likely to have data in CrUX. So for example if your quota is 150 queries per minute, maybe choose the top 1,500 most popular pages and spread out those queries over 10 minutes. You may also want to filter it down even more by page types, like home/search/product pages, and choose a few popular URLs to represent each type. In theory, pages that are built similarly also have similar user experiences, so it may be redundant to track thousands of the same page type. Search Console can also be used to find CWV issues in groups of pages.

[vivek shekhar singh]

Thanks Rick , i did see this   https://console.cloud.google.com/apis/api/chromeuxreport.googleapis.com/quotas. yesterday but it doesn't have data on the key or project i created for the batch api request. I saw this note " You're limited to 1000 calls in a single batch request. If you need to make more calls than that, use multiple batch requests."  on this link https://developers.google.com/web/tools/chrome-user-experience-report/api/guides/batch . Is this different from 150 queries per minute quota . I have also noticed if i make concurrent multiple batch requests of 1000 each they start returning 429 at some point not sure if there is a daily overall quota on batch which i still need to know. By the way to be really frank Search Console is a bit weird , i am not sure what is the algorithm behind grouping similar pages , what are those KPIs and factors . I see there are many pages bucketed in needs improvement category as similar pages but they are actually not 1 has a form and other doesn't . One has a totally different header/footer , one has a completely different set of layout from the other and worse part is that some of them are passing core web vitals but bucketed in a group that needs improvement , which becomes very difficult for us to benchmark search console . This is the reason i want to do an offline report and then filter and do a granular level analysis . Let me know your thoughts.

[Rick Viscomi]

Sorry, the CrUX API batch documentation is auto-generated for generic APIs that may have more quota, so the "1000 calls" part doesn't necessarily apply here. The CrUX API quota is 150 qpm, and you shouldn't be able to issue more API calls than that in a single batch request. To make more than 150 qpm, you would need to rate limit your requests to stay under the quota. I'll look into updating our docs to make this clearer.

Also, thanks for your feedback about Search Console, it helps us improve our tools.

[vivek shekhar singh]

Thanks Rick so if i get it correct i should be batching 150 URLs per batch request separated by a minute or two correct me if i am wrong . Having said that just FYI i am actually able to batch 1000  URLs per batch request per minute for across first 7 minutes , post which i get one 429 and from the 9th minute it again returns 200 . If 150 URLs per batch is what it is then it should fail me in my first batch request of 1000 URLs isn't it ? Sorry if i am bothering you with too many questions here but just too curious.

[Rick Viscomi]

Based on my understanding of the batch API, I would expect it to return a 429 error when attempting to request more than 150 URLs at a time. If you're able to make more requests than the documented limit, proceed cautiously knowing that the incorrect behavior might be fixed at any time. Ultimately, I'd recommend implementing a solution that doesn't exceed the documented limit of 150 qpm to avoid future disruptions.

[vivek shekhar singh]

Thanks a lot Rick that helps i will proceed with 150 qpm . Final question this 150qpm limit is per user per IP address correct . So parallely two or more different users on different systems can make 150 calls per minute , correct ?

[Rick Viscomi]

The limit should be per API key. So if one user is consuming 100% of the quota, no other user should be able to query the API with the same key.

[vivek shekhar singh]

Interesting thanks 

[vivek shekhar singh]

Any idea on how can i use Crux API to show a monthly trend report of Core Web Vitals ?

[Rick Viscomi]

This is getting slightly off-topic from the batch question but you can follow a process like the one I wrote about a few years ago that uses Google Sheets and Apps Script to ping the API on a daily basis and log the results. From there you can graph the results to see the change over time. I also shared an updated Apps Script snippet on GitHub that uses the CrUX API (as opposed to the old PSI API).

Rick

[vivek shekhar singh]

Thanks Rick . Apologies for  asking this question here . I have some more questions around this , how do you propose i should ask them if not on this conversation.

Let me know if you want me to trigger a new conversation with a different topic.