Randomly reaching QPS quota

[maranz hiberus]

Hi team,

We have implemented in our app parallelization of queries and we have set up in place a rate limiter so we keep the threshold in order to not get rejections from the API. Unfortunately, after some testing done and donwsizing progressively our rate of queries, we have reach a point that it shouldn't be possible to have a burst of queries that exceeds the quota.

Our last attempt was with a rate of 2 queries every 255ms - which effectively reduces the maximum available queries in a burst to 10 queries in a random period of 1021ms. This shouldn't trigger the exception, but we seem to get it nevertheless.

Thus, either there is a rogue query that we are sending that we are not aware of and slips unnoticed, or there is something we are not understanding about how the quota threshold is detected from your side, raising the blocking of further requests. We would like to know if there is a way to know what are you receiving, in a way like the change log  (which unfortunately only shows creation / update operations). We would need to see all of them instead...

Also, we would like to know how much time should we hold our requests once we get an error like the following in order to safely resume operations (in order to implement a Circuit Breaker so we do not block ourselves further until we can get a successful configuration):

{

  "code" : 429,

  "errors" : [ {

    "domain" : "global",

    "message" : "Quota exceeded for quota metric 'API Queries' and limit 'API Queries per minute per user' of service 'dfareporting.googleapis.com' for consumer 'project_number:XXXXXXXXXX'.",

    "reason" : "rateLimitExceeded"

  } ],

  "message" : "Quota exceeded for quota metric 'API Queries' and limit 'API Queries per minute per user' of service 'dfareporting.googleapis.com' for consumer 'project_number:XXXXXXXXXXXXX'.",

  "status" : "RESOURCE_EXHAUSTED"

}

Thanks in advance, regards,

Miguel Angel

[DCM API (DoubleClick Campaign Mgr)]

Hi Miguel,

 

Thank you for reaching out to us.

 

We do have documentation with regards to quota limits to help you understand why you are encountering this errors.

 

Some things that can be highlighted from that document are the following:

• Queries per minute per user is set to 60 by default. This limit can be increased as needed, up to a maximum of 600 (10 QPS).
• We only have 50,000 requests per project per day limit.

 

These two are most probably the cause of the issue on your end. If you think are you indeed exceeding this limitations, we would recommend reaching out to your account manager for assistance. On the other hand, you could also request for increase for additional quota by following the steps provided in the following guide:

• Request for additional quota

 

Let us know if you have any other concern.

Regards,

Michael Angelo Legaspi DCM API Team

ref:_00D1U1174p._5004Q2R8SHK:ref

[maranz hiberus]

Hello Michael,

Thanks for the links, though as you can deduce from my email, we already know the quota that is the cause of the issue us the queries per minute per user - as was clearly visible in the error message I provided

("message" : "Quota exceeded for quota metric 'API Queries' and limit 'API Queries per minute per user'). We have no problem with the daily quota; we are are now barely scrapping 10k queries per day.

We have that value already set to the maximum (600), and with our rate limiter it shouldn't be able to reach this number (2 queries every 255ms means that 60.000ms/255 we can get up to 470,58 QPM; far less than the maximum). That's why we reached out to you to help us find out:

• How can we know what queries are you receiving (a log if possible), so we can trace the offending ones that seem to slip our rate limiter, and;
• How much time does our app need to hold up if we detect that this issue appears, in order to dampen the impact.  

If we refer to your documentation, our case would be the las one exposed in https://developers.google.com/doubleclick-advertisers/quotas#exceeding_limits (userRateLimitExceeded). Unfortunately, we have already backoff and jittering in place, but it does not seem to be enough. Eventually the API starts responding, but the process that initially encounters this issue finishes in error as it is unable to recover before depleting all the retry attempts. New processes, as there is usually a few minutes spread between them, work fine. And as I pointed out above, theorically we are already limiting our requests. That's why we need assistance to know what's going wrong.

You  recommended reaching out to our account manager for assistance; but are you sure they the right contact to provide assistance to resolve said points? Because there is no need to change configuration or increasing quotas; what we need to know  is why are we breaking the threshold with our current configuration.

Regards,

Miguel Angel

[DCM API (DoubleClick Campaign Mgr)]

Hi Miguel,

 

Thank you for clarifying some information.

 

Could you please share the following information with us privately via "Reply To Author" button?

• DCM Account ID
• Project number
• Profile ID

 

We will need this to further check the issue.

Regards,

[DCM API (DoubleClick Campaign Mgr)]

Hi Miguel,

 

We have received all the information you shared privately. That said, I have raised this issue to the rest of the team for further investigation. We will provide more information once we have the result of the investigation.

Regards,

[DCM API (DoubleClick Campaign Mgr)]

Hi Miguel,

 

Hope you are doing fine.

 

Our team responded back to us and told us that the issue seems to be related to the general questions about patch consuming 3x quota. With that, we have increased the per minute quota for you from 600 to 1800 per minute since patch calls consume 3x the quota. Attached screenshot for reference.

Let us know should you have any other concern.

Regards,