API Issue - High Latency and deadline exceeded errors all of a sudden

[Haritha Devalla]

Hi Team,

We are facing high latency with the API calls and are also seeing deadline exceeded errors for most of the accounts for the past two hours.
Could you please look into this at the earliest and let us know if there is any issue with the API currently.

Thanks,
Haritha

[Jing CHEN]

The same situation.

[Radosław Pociecha]

We are seeing the same behaviour.

[Sergei Aleinik]

I confirm. Same problems

среда, 26 июня 2024 г. в 16:50:45 UTC+2, Radosław Pociecha:

[Zorion]

Same here. The API is unusable in those conditions.

[Richard Cutting]

Same here also seeing a lot of RESOURCE_EXHAUSTED errors and I know we haven't !

[Matt Dworkin]

We're also having the same issues. First it was Bad Gateways, now RESOURCE_EXHAUSTED. We supposedly have an unlimited quote so it's not clear why we would have these RESOURCE_EXHAUSTED issues

[Giacomo Dell'Omo]

Same situation, we are receiving a lot of random RESOURCE_EXHAUSTED error on multiple different accounts

[Matteo Medda]

We are seeing the same behaviour from today.

[ro...@searchkings.ca]

Same -- glad to know it's not just us!

[Levi Kline]

+1 Same issue

[Kevin Nguyen]

+1 Same issue

[GDZ]

We have an unlimited quota. Why all this RESOURCE_EXHAUSTED errors all of a sudden? We had to pause campaigns in hundreds of different accounts. Needs to be fixed ASAP!

[Mário Carvalho]

Same problem here

[Ferran Gil Cuesta]

Same issue. We see some "RESOURCE_EXHAUSTED" but also so many unexplained timeouts and delays everywhere.

[Alex Rybak]

We are experiencing the same issue

On Wednesday, June 26, 2024 at 5:13:57 PM UTC+2 Zorion wrote:

[Alex Black]

We're being significantly impacted too, I assume this is a result of https://ads-developers.googleblog.com/2024/05/google-ads-api-resource-usage-policy.html ?

It'd help to get a quota increase, even a temporary one!

On Wednesday, June 26, 2024 at 8:52:06 AM UTC-7 ro...@searchkings.ca wrote:

[John Schroeder]

+1 Same issue across many accounts. Current automated API jobs (for tasks like offline conversion upload) succeed for a few requests before timing out with RESOURCE_EXHAUSTED errors.

[Artem Zarutskiy]

We are having the same issue starting this morning: RESOURCE_EXHAUSTED on many requests. We have the Standard access level with unlimited quota. We have hundreds of campaigns pending important updates that we can't process because of this. 

[wang davy]

+1, is there  any issue?

[James Guiden]

+1

Experiencing similar difficulties. API requests made to searchStream are left hanging, with occasional 429 RESOURCE_EXHAUSTED error messages being returned (for a single API call, not exceeding our quota limit). 

[Kevin]

+1 same issue here

[Dumitru Cenusa]

Same issue. We see "Resource has been exhausted"

[Google Ads API and AdWords API Forum]

Hi,

Thank you for reaching out to the Google Ads API support team.

Allow me to check with the rest of our team and one of our colleagues will reach out to you once we have an update on your issue.

 

This message is in relation to case "{{{Case.Thread_Id}}}" (ADR-{{{Case.CaseNumber}}})

Thanks,



Google Ads API Team
Register for the upcoming workshop: Performance Max and the Google Ads API!

​​​​​​​

[AdWordsAPI-Gnr]

The same situation continues for me. However, I thought the problem was related to new query limits. I hope it is a general problem and gets solved.

26 Haziran 2024 Çarşamba tarihinde saat 20:27:48 UTC+3 itibarıyla Levi Kline şunları yazdı:

Interesting -- I was not aware of this breaking change. It also seems as though they haven't even added these new v18 errors to protobuf.

[Google Ads API Forum Advisor]

Jing:

There is a known problem in our stack that has been causing these errors. The root cause has been mitigated but the effects may linger. I don't have an ETA on when that will be.

In the interim, the best strategy on your side is to use an exponential backoff or reduce the size of your requests.

 

This message is in relation to case "ref:!00D1U01174p.!5004Q02tJTru:ref" (ADR-00244039)

Thanks,
 

Bob Hancock Google Ads API Team

[Matt Dworkin]

In the logs I'm seeing that a retry needs to wait 900 seconds, so I'm not sure exponential backoff will help. Could that actually end up hurting? Does the "retry time" reset if we send another request before 15 minutes has passed?

[Richard Cutting]

Working again for me now.

[Google Ads API Forum Advisor]

Richard:

I expect that you will start to see the normal behavior return as the backlog is cleared. If you receive the error, then retry. If you start receiving it consistently again, let us know in this thread.

[Levi Kline]

Thanks for the update Bob -- could you please clarify the new v18 search rate limits mentioned above (https://ads-developers.googleblog.com/2024/05/google-ads-api-resource-usage-policy.html)? After digging through available resources, I have a couple of concerns/questions:

1. What are the actual query limits? The attached resource only provides insights into typical usage (https://developers.google.com/google-ads/api/docs/productionize/manage-data-efficiently). Should we expect to see an increase in rate limits? Our entire infrastructure depends on the freshness of data, so would be helpful to know how critical it is for us to work on building caching mechanisms etc.
2. V18 protobuf definitions don't seem to be available. Are these scheduled for release?

Best,
Levi

[Google Ads API Forum Advisor]

Levi:

Did you mean v17? v18 won't be released until the last part of October.

To give more insight on the query limits I'm going transfer this case to my colleague Anash who is the expert on this subject.

[Levi Kline]

Appreciate it! Apologies for the confusion on the versioning -- the docs mentioned a v18 `QuotaError.EXCESSIVE_SHORT_TERM_QUERY_RESOURCE_CONSUMPTION,` so I assumed it was available.

[Google Ads API Forum Advisor]

Hi Levi,

1. We have no plans to publicly share the actual limits. However, if you share your API Manager account, I can look it up and let you know where your current usage is relative to the limits for this error. 

2. V18 is scheduled to release in October 2024.
 

This message is in relation to case "ref:!00D1U01174p.!5004Q02tJTru:ref" (ADR-00244039)

Thanks,
 

Anash Oommen Google Ads API Team

[Google Ads API Forum Advisor]

Hello Everyone!

You should start seeing improvements in your API requests now; I'm already seeing the improvements in our monitoring. We are actively working on fully resolving the issue. Please retry requests that you were unable to complete before. A big thank you to you all for providing feedback and sticking to one thread like this on the forum; it was quite helpful in our analysis to get to the solution.

All the best,

This message is in relation to case "ref:!00D1U01174p.!5004Q02tJTru:ref" (ADR-00244039)

 

Nadine Wang Google Ads API Team

[Levi Kline]

Hi Anash, just wanted to follow up -- I believe I privately replied with my manager account id / email. Google Groups doesn't seem to have a view for private replies, so just wanted to make sure it got to you. If not, please let me know if there is an email or other way to pass along the info. Thanks again for the help!

Best,
Levi