User Query Quota Limits

[William Chen]

Hi All,

I have been using BigQuery to obtain the measurement-lab.ndt.ndt7 data for a project. However, since returning to the project a couple weeks ago, I have found that the amount of data I am allowed to access per day has shrunk significantly. Whereas I was able to obtain about a year's worth of data for a given US city per day earlier this year, I am only able to obtain around 2 month's worth of data per day now. Would anybody know why?

Thank you.  

[Pavlos Sermpezis]

Hello, 

We have implemented limits in usage of BiqQuery (BQ) since last year to avoid abuse of the platform, which comes at a high cost for M-Lab.

The current quota per user per day is 10TiB.

We can recommend you a couple ways for avoiding hitting your quota:

(1) Optimize your queries so that you effectively request only the information you need in each query. Small differences in queries can make a huge difference in how much data any given query will process.
For example, for a use case similar to yours, the following query would process 41.67 TB (i.e., it would hit the quota and not run)

SELECT * FROM `measurement-lab.ndt.ndt7`

WHERE client.Geo.City = 'New York'

AND date > '2024-07-01'

but a similar query that requests only the data you may need (here, I omit the raw field) would request < 1 TB (i.e., will NOT hit the quota):

SELECT date, a, server, client FROM `measurement-lab.ndt.ndt7`

WHERE client.Geo.City = 'New York'

AND date > '2024-07-01'

Btw,  it is also very helpful, to use the BigQuery Studio to draft queries, where it will estimate how much data is going to be processed before you even run the query. That way, you can tweak your queries to find how to process less data without ever having to actually run the query to find out. See the screenshot below

(2) the second way would be to split your query into smaller time intervals, and query over multiple days (every day the quota refreshes). We would still recommend the first way.

Best regards,

Pavlos

--

Pavlos Sermpezis
Director, Tech Lead

Measurement Lab

Code for Science and Society