com.google.ads.googleads.v10.errors.GoogleAdsException: errors {
error_code {
internal_error: INTERNAL_ERROR
}
message: "An internal error has occurred."
}
request_id: "EDSPIdhbp3lSjX29k4INQg"
at com.google.ads.googleads.v10.errors.GoogleAdsException$Factory.createException(GoogleAdsException.java:42)
at com.google.ads.googleads.v10.errors.GoogleAdsException$Factory.createException(GoogleAdsException.java:34)
at com.google.ads.googleads.lib.stubs.exceptions.BaseGoogleAdsException$Factory.createGoogleAdsException(BaseGoogleAdsException.java:105)
at com.google.ads.googleads.lib.callables.GoogleAdsExceptionTransformation.transform(GoogleAdsExceptionTransformation.java:39)
at com.google.ads.googleads.lib.stubs.callables.ExceptionTransformingServerStreamingCallable$ExceptionTransformingStreamObserver.onError(ExceptionTransformingServerStreamingCallable.java:71)
at com.google.api.gax.tracing.TracedResponseObserver.onError(TracedResponseObserver.java:103)
at com.google.api.gax.rpc.RetryingServerStreamingCallable$1.onFailure(RetryingServerStreamingCallable.java:99)
at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)
at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1041)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1215)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771)
at com.google.api.gax.retrying.BasicRetryingFuture.handleAttempt(BasicRetryingFuture.java:200)
at com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.handle(CallbackChainRetryingFuture.java:135)
at com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.run(CallbackChainRetryingFuture.java:117)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1215)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771)
at com.google.api.core.AbstractApiFuture$InternalSettableFuture.setException(AbstractApiFuture.java:95)
at com.google.api.core.AbstractApiFuture.setException(AbstractApiFuture.java:77)
at com.google.api.core.SettableApiFuture.setException(SettableApiFuture.java:52)
at com.google.api.gax.rpc.ServerStreamingAttemptCallable.onAttemptError(ServerStreamingAttemptCallable.java:378)
at com.google.api.gax.rpc.ServerStreamingAttemptCallable.access$600(ServerStreamingAttemptCallable.java:97)
at com.google.api.gax.rpc.ServerStreamingAttemptCallable$2.onErrorImpl(ServerStreamingAttemptCallable.java:249)
at com.google.api.gax.rpc.StateCheckingResponseObserver.onError(StateCheckingResponseObserver.java:86)
at com.google.api.gax.rpc.Watchdog$WatchdogStream.onErrorImpl(Watchdog.java:284)
at com.google.api.gax.rpc.StateCheckingResponseObserver.onError(StateCheckingResponseObserver.java:86)
at com.google.api.gax.grpc.ExceptionResponseObserver.onErrorImpl(ExceptionResponseObserver.java:84)
at com.google.api.gax.rpc.StateCheckingResponseObserver.onError(StateCheckingResponseObserver.java:86)
at com.google.api.gax.grpc.GrpcDirectStreamController$ResponseObserverAdapter.onClose(GrpcDirectStreamController.java:149)
at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at com.google.ads.googleads.lib.logging.LoggingInterceptor$1$1.onClose(LoggingInterceptor.java:113)
at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:463)
at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:427)
at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:460)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:557)
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:69)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:738)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:717)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Suppressed: java.lang.RuntimeException: Asynchronous task failed
at com.google.api.gax.rpc.ServerStreamIterator.hasNext(ServerStreamIterator.java:105)
at java.base/java.lang.Iterable.forEach(Iterable.java:74)
|
||||||
date:2022-12-01T18:56:57.653+09:00
- ex_account_id: 5156847529date:2022-12-01T18:56:44.679+09:00
- ex_account_id: 4598968391
|
||||||
date:2022-12-14T18:02:42.133+09:00 thread:grpc-default-executor-3 track: level:WARN logger:com.google.ads.googleads.lib.request.summary message:FAILURE REQUEST SUMMARY. Method: google.ads.googleads.v12.services.GoogleAdsService/SearchStream, Endpoint: googleads.googleapis.com:443, CustomerID: 3258900403, RequestID: 89crmlX3fGKZSaQkxlJgRg, ResponseCode: RESOURCE_EXHAUSTED, Fault: Resource has been exhausted (e.g. check quota)..
date:2022-12-14T18:02:42.160+09:00 thread:grpc-default-executor-3 track: level:INFO logger:com.google.ads.googleads.lib.request.detail message:FAILURE REQUEST DETAIL.
Request
-------
MethodName: google.ads.googleads.v12.services.GoogleAdsService/SearchStream
Endpoint: googleads.googleapis.com:443
Headers: {developer-token=REDACTED, login-customer-id=5693729523, x-goog-api-client=gl-java/17.0.2 gccl/22.0.0 gapic/22.0.0 gax/2.12.2 grpc/1.44.0 pb/3.19.6}
Body: customer_id: "3258900403"
query: "SELECT customer.descriptive_name, campaign.name, customer.id, campaign.id, ad_group.id, ad_group.name, ad_group_ad.ad.id, ad_group_ad.ad.name, metrics.cost_micros, metrics.impressions, metrics.average_cpm, metrics.clicks, metrics.average_cpc, metrics.ctr, metrics.video_views, metrics.video_view_rate, metrics.video_quartile_p25_rate, metrics.video_quartile_p50_rate, metrics.video_quartile_p75_rate, metrics.video_quartile_p100_rate, metrics.conversions, metrics.conversions_from_interactions_rate, metrics.cost_per_conversion, metrics.conversions_value FROM ad_group_ad WHERE segments.date = \'2022-12-14\' ORDER BY campaign.id"
Response
--------
Headers: Metadata(content-type=application/grpc,request-id=89crmlX3fGKZSaQkxlJgRg,date=Wed, 14 Dec 2022 09:02:42 GMT,alt-svc=h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43")
Body: null
Failure message: errors {
error_code {
quota_error: RESOURCE_EXHAUSTED
}
message: "Too many requests. Retry in 900 seconds."
}
request_id: "89crmlX3fGKZSaQkxlJgRg"
Status: Status{code=RESOURCE_EXHAUSTED, description=Resource has been exhausted (e.g. check quota)., cause=null}.
date:2022-12-14T18:02:42.390+09:00 thread:main track: level:ERROR logger:com.rakuten.rmp.connect.sync.domain.service.googleads.GoogleAdsReaderServiceImpl message:Cannot get Google API Ads - ex_account_id: 3258900403
com.google.ads.googleads.v12.errors.GoogleAdsException: errors {
error_code {
quota_error: RESOURCE_EXHAUSTED
}
message: "Too many requests. Retry in 900 seconds."
}
Regards,
Aaron.
|
||||||
Hi Aaron,
I work with Nirmita. Thanks for getting back to us.
I understand that you have encountered an RESOURCE_EXHAUSTED error, and error message ‘Too many requests. Retry in 900 seconds“. This seems to be that API requests hitting the server aggressively. The server throws this error once it detects that your request has exceeded the system frequency limit. When you retry requests, you may continue using an exponential backoff policy by delaying 5-10 seconds. For example, if you first pause 5 seconds before the first retry, you could pause 10 seconds after the second and 20 seconds after the third retry. Exponential backoff helps ensure you are not calling the API too aggressively.
Also, you mentioned that you set your batch job hourly to process sync campaign data from Google Platform, but in the above email you provided error logs for the GoogleAdsService/SearchStream API request. So, there seems to be concurrent API requests making calls to Google Ads API at same time, and this is why you encountered said error.
That being said, we suggest setting a reasonable upper bound for the total number of concurrent tasks that are going to make requests (across all processes and machines), and adjusting upward to optimize your throughput without exceeding the rate limit is recommended. Otherwise, you may possibly experience the exceeding rate limits that the client app is spawning an excessive number of parallel tasks since we don't limit the number of parallel requests a client app can have, this can easily exceed the Requests Per Second limit at the developer token level. However, Throttles can be implemented at the consumer side by limiting the number of concurrent consumers, or implement rate limiters or throttlers for either the producers or consumers.For more information, you may refer to this throttling QPS from the client side (check out Throttling and rate limiters).
You may apply above mentioned points while sending requests to Google Ads API server, and let us know how this goes on.
Regards,
|
||||||
Request
-------
MethodName: google.ads.googleads.v12.services.GoogleAdsService/SearchStream
Endpoint: googleads.googleapis.com:443
Headers: {developer-token=REDACTED, login-customer-id=5693729523, x-goog-api-client=gl-java/17.0.2 gccl/22.0.0 gapic/22.0.0 gax/2.12.2 grpc/1.44.0 pb/3.19.6}
Body: customer_id: "1161207172"
query: "SELECT customer.id, campaign.id, campaign_budget.amount_micros FROM campaign WHERE segments.date = \'2022-12-21\' ORDER BY campaign.id"
Response
--------
Headers: Metadata(content-type=application/grpc,request-id=UprDhyB4aNklsoPcyhKigQ,date=Wed, 21 Dec 2022 06:01:05 GMT,alt-svc=h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43")
Body: null
Failure message: errors {
error_code {
internal_error: INTERNAL_ERROR
}
message: "An internal error has occurred."
}
request_id: "UprDhyB4aNklsoPcyhKigQ"
Status: Status{code=INTERNAL, description=Internal error encountered., cause=null}.
date:2022-12-21T15:01:05.574+09:00 thread:main track: level:ERROR logger:com.rakuten.rmp.connect.sync.domain.service.googleads.GoogleAdsReaderServiceImpl message:Cannot get Google API Ads - ex_account_id: 1161207172
com.google.ads.googleads.v12.errors.GoogleAdsException: errors {
error_code {
internal_error: INTERNAL_ERROR
}
message: "An internal error has occurred."
}
request_id: "UprDhyB4aNklsoPcyhKigQ"
at com.google.ads.googleads.v12.errors.GoogleAdsException$Factory.createException(GoogleAdsException.java:42)
at com.google.ads.googleads.v12.errors.GoogleAdsException$Factory.createException(GoogleAdsException.java:34)
at com.google.ads.googleads.lib.stubs.exceptions.BaseGoogleAdsException$Factory.createGoogleAdsException(BaseGoogleAdsException.java:105)
at com.google.ads.googleads.lib.callables.GoogleAdsExceptionTransformation.transform(GoogleAdsExceptionTransformation.java:39)
at com.google.ads.googleads.lib.stubs.callables.ExceptionTransformingServerStreamingCallable$ExceptionTransformingStreamObserver.onError(ExceptionTransformingServerStreamingCallable.java:71)
at com.google.api.gax.tracing.TracedResponseObserver.onError(TracedResponseObserver.java:103)
at com.google.api.gax.rpc.RetryingServerStreamingCallable$1.onFailure(RetryingServerStreamingCallable.java:99)
at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)
at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1100)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1277)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1038)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:808)
at com.google.api.gax.retrying.BasicRetryingFuture.handleAttempt(BasicRetryingFuture.java:200)
at com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.handle(CallbackChainRetryingFuture.java:135)
at com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.run(CallbackChainRetryingFuture.java:117)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1277)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1038)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:808)
at com.google.api.core.AbstractApiFuture$InternalSettableFuture.setException(AbstractApiFuture.java:95)
at com.google.api.core.AbstractApiFuture.setException(AbstractApiFuture.java:77)
at com.google.api.core.SettableApiFuture.setException(SettableApiFuture.java:52)
at com.google.api.gax.rpc.ServerStreamingAttemptCallable.onAttemptError(ServerStreamingAttemptCallable.java:378)
at com.google.api.gax.rpc.ServerStreamingAttemptCallable.access$600(ServerStreamingAttemptCallable.java:97)
at com.google.api.gax.rpc.ServerStreamingAttemptCallable$2.onErrorImpl(ServerStreamingAttemptCallable.java:249)
at com.google.api.gax.rpc.StateCheckingResponseObserver.onError(StateCheckingResponseObserver.java:86)
at com.google.api.gax.rpc.Watchdog$WatchdogStream.onErrorImpl(Watchdog.java:284)
~~~~~~~~~~~~~~~~~~~~~~
Request
-------
MethodName: google.ads.googleads.v12.services.GoogleAdsService/SearchStream
Endpoint: googleads.googleapis.com:443
Headers: {developer-token=REDACTED, login-customer-id=5693729523, x-goog-api-client=gl-java/17.0.2 gccl/22.0.0 gapic/22.0.0 gax/2.12.2 grpc/1.44.0 pb/3.19.6}
Body: customer_id: "1154526700"
query: "SELECT customer.descriptive_name, campaign.name, customer.id, campaign.id, ad_group.id, ad_group.name, ad_group_ad.ad.id, ad_group_ad.ad.name, metrics.cost_micros, metrics.impressions, metrics.average_cpm, metrics.clicks, metrics.average_cpc, metrics.ctr, metrics.video_views, metrics.video_view_rate, metrics.video_quartile_p25_rate, metrics.video_quartile_p50_rate, metrics.video_quartile_p75_rate, metrics.video_quartile_p100_rate, metrics.conversions, metrics.conversions_from_interactions_rate, metrics.cost_per_conversion, metrics.conversions_value FROM ad_group_ad WHERE segments.date = \'2022-12-20\' ORDER BY campaign.id"
Response
--------
Headers: Metadata(content-type=application/grpc,request-id=t-3-psKzVikj_CTAbb8_-g,date=Wed, 21 Dec 2022 06:01:06 GMT,alt-svc=h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43")
Body: null
Failure message: errors {
error_code {
internal_error: INTERNAL_ERROR
}
message: "An internal error has occurred."
}
request_id: "t-3-psKzVikj_CTAbb8_-g"
Status: Status{code=INTERNAL, description=Internal error encountered., cause=null}.
date:2022-12-21T15:01:06.710+09:00 thread:main track: level:ERROR logger:com.rakuten.rmp.connect.sync.domain.service.googleads.GoogleAdsReaderServiceImpl message:Cannot get Google API Ads - ex_account_id: 1154526700
com.google.ads.googleads.v12.errors.GoogleAdsException: errors {
error_code {
internal_error: INTERNAL_ERROR
}
message: "An internal error has occurred."
}
request_id: "t-3-psKzVikj_CTAbb8_-g"
at com.google.ads.googleads.v12.errors.GoogleAdsException$Factory.createException(GoogleAdsException.java:42)
at com.google.ads.googleads.v12.errors.GoogleAdsException$Factory.createException(GoogleAdsException.java:34)
at com.google.ads.googleads.lib.stubs.exceptions.BaseGoogleAdsException$Factory.createGoogleAdsException(BaseGoogleAdsException.java:105)
at com.google.ads.googleads.lib.callables.GoogleAdsExceptionTransformation.transform(GoogleAdsExceptionTransformation.java:39)
Thank you very much ~
Regards,
Aaron.
|
||||||
static final String CAMPAIGN_INFO = "SELECT" +
" customer.descriptive_name," +
" campaign.name," +
" customer.id," +
" campaign.id," +
" ad_group.id," +
" ad_group.name," +
" ad_group_ad.ad.id," +
" ad_group_ad.ad.name" +
" FROM ad_group_ad";
static final String CAMPAIGN_CONVERSION = "SELECT" +
" customer.descriptive_name," +
" campaign.name," +
" customer.id," +
" campaign.id," +
" ad_group.id," +
" ad_group.name," +
" ad_group_ad.ad.id," +
" ad_group_ad.ad.name," +
" metrics.cost_micros," +
" metrics.impressions," +
" metrics.average_cpm," +
" metrics.clicks," +
" metrics.average_cpc," +
" metrics.ctr," +
" metrics.video_views," +
" metrics.video_view_rate," +
" metrics.video_quartile_p25_rate," +
" metrics.video_quartile_p50_rate," +
" metrics.video_quartile_p75_rate," +
" metrics.video_quartile_p100_rate," +
" metrics.conversions," +
" metrics.conversions_from_interactions_rate," +
" metrics.cost_per_conversion," +
" metrics.conversions_value" +
" FROM ad_group_ad" +
" WHERE segments.date = '%s'" +
" ORDER BY campaign.id";
static final String CAMPAIGN_BUDGET_INFO = "SELECT" +
" customer.id," +
" campaign.id," +
" campaign_budget.amount_micros" +
" FROM campaign" +
" WHERE segments.date = '%s'" +
" ORDER BY campaign.id";
Hi Aaron,
Thanks for getting back to us.
Could you confirm if you again hit the RESOURCE_EXHAUSTED error after implementing the exponential back off policy and rate limiter? If yes, our team requires the most recent complet logs in order to investigate the issue further.
When receiving a RESOURCE_EXHAUSTED error, you must wait for the duration stated in the error description, or else the API will start increasing the wait time, which is likely why you see it up to 900 seconds. Continuing to retry before that time is going to cause many, many more errors, which will generally ruin your throughput.
Most likely what happened is you got one error, didn't wait, and then it escalated until all or most of your requests were rejected by not abiding by the rate limit.
Kindly see below responses to your questions:
a. If no data for API request return, does it still mean one API operation and count into maximum requests?
>> API operations are the total sum of get requests and mutate operations. If you have sent one API request for the search or searchStream and the API request did not return any response, then it will be considered as 1 operation consumed. A request that is not a Get, Mutate, Search, or SearchStream request counts as one operation against the user's daily operation quota. Requests that are rejected with a GoogleAdsFailure still count against the user's daily operation quota.
On the other hand, requests that fail but don't return a GoogleAdsFailure, such as from an error at the network level, won't count against the user's daily operation quota since the requests would never reach the service. An example of this is a network connectivity failure.
b. How to calculate the API requests total count from my side to check if hit the 15,000 API operations per day?
>> Kindly note that the API request and operation are different. 1 request can have many operations such as OfflineUserDataJobService.AddOfflineUserDataJobOperations can consume 10,000 operations in one request. However, using Google Ads API there is no way to calculate the consumed API operations count.
Regards,