performance for Google Ads shopping_performance_view

[Hao Long]

Hi Team,

we are facing performance issue when download data from shopping performance view. the case is we are downloading latest 45 days data from shopping performance report from Google Adwords API account by account, but it failed when we migrate to Google Ads API. could you please help identify the root cause?

Exception Message:

[2022-02-18 19:53:33,134] {bash.py:173} INFO - MethodName: google.ads.googleads.v9.services.GoogleAdsService/SearchStream
[2022-02-18 19:53:33,134] {bash.py:173} INFO - Endpoint: googleads.googleapis.com:443
[2022-02-18 19:53:33,134] {bash.py:173} INFO - Headers: {developer-token=REDACTED, login-customer-id=5749847778, x-goog-api-client=gl-java/1.8.0_275 gccl/0.0.0 gapic/ gax/2.6.1 grpc/}
[2022-02-18 19:53:33,134] {bash.py:173} INFO - Body: customer_id: "7740555995"
[2022-02-18 19:53:33,134] {bash.py:173} INFO - query: "select segments.date,segments.product_item_id,ad_group.id,campaign.id,segments.product_merchant_id,segments.product_custom_attribute0,segments.product_custom_attribute1,segments.product_custom_attribute2,segments.product_custom_attribute3,segments.product_custom_attribute4,segments.device,metrics.conversions_value from shopping_performance_view  where segments.date between \"2021-12-28\" and \"2022-02-11\" and metrics.impressions > 0"
[2022-02-18 19:53:33,134] {bash.py:173} INFO -
[2022-02-18 19:53:33,134] {bash.py:173} INFO -
[2022-02-18 19:53:33,134] {bash.py:173} INFO - Response
[2022-02-18 19:53:33,134] {bash.py:173} INFO - --------
[2022-02-18 19:53:33,134] {bash.py:173} INFO - Headers: null
[2022-02-18 19:53:33,134] {bash.py:173} INFO - Body: null
[2022-02-18 19:53:33,134] {bash.py:173} INFO - Failure message: null
[2022-02-18 19:53:33,134] {bash.py:173} INFO - Status: Status{code=DEADLINE_EXCEEDED, description=deadline exceeded after 3599.997897730s. [closed=[], open=[[buffered_nanos=207933631, buffered_nanos=166584391, remote_addr=*********]]], cause=null}.
[2022-02-18 19:53:33,134] {bash.py:173} INFO - 19:53:33.118 [pool-9-thread-3] ERROR c.e.m.d.c.g.s.GoogleAdsAPIReportDownloadHelper - Donwload got exception: com.google.api.gax.rpc.DeadlineExceededException: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 3599.997897730s. [closed=[], open=[[buffered_nanos=207933631, buffered_nanos=166584391, remote_addr=*********]]]
[2022-02-18 19:53:33,134] {bash.py:173} INFO - com.google.api.gax.rpc.DeadlineExceededException: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 3599.997897730s. [closed=[], open=[[buffered_nanos=207933631, buffered_nanos=166584391, remote_addr=*********]]]
[2022-02-18 19:53:33,134] {bash.py:173} INFO -  at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:51)
[2022-02-18 19:53:33,135] {bash.py:173} INFO -  at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72)
[2022-02-18 19:53:33,135] {bash.py:173} INFO -  at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60)
[2022-02-18 19:53:33,135] {bash.py:173} INFO -  at com.google.api.gax.grpc.ExceptionResponseObserver.onErrorImpl(ExceptionResponseObserver.java:82)
[2022-02-18 19:53:33,135] {bash.py:173} INFO -  at com.google.api.gax.rpc.StateCheckingResponseObserver.onError(StateCheckingResponseObserver.java:86)
[2022-02-18 19:53:33,135] {bash.py:173} INFO -  at com.google.api.gax.grpc.GrpcDirectStreamController$ResponseObserverAdapter.onClose(GrpcDirectStreamController.java:149)
[2022-02-18 19:53:33,135] {bash.py:173} INFO -  at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
[2022-02-18 19:53:33,135] {bash.py:173} INFO -  at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
[2022-02-18 19:53:33,135] {bash.py:173} INFO -  at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
[2022-02-18 19:53:33,135] {bash.py:173} INFO -  at com.google.ads.googleads.lib.logging.LoggingInterceptor$1$1.onClose(LoggingInterceptor.java:113)
[2022-02-18 19:53:33,135] {bash.py:173} INFO -  at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:463)
[2022-02-18 19:53:33,135] {bash.py:173} INFO -  at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:427)
[2022-02-18 19:53:33,135] {bash.py:173} INFO -  at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:460)
[2022-02-18 19:53:33,135] {bash.py:173} INFO -  at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:557)
[2022-02-18 19:53:33,135] {bash.py:173} INFO -  at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:69)
[2022-02-18 19:53:33,135] {bash.py:173} INFO -  at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:738)
[2022-02-18 19:53:33,135] {bash.py:173} INFO -  at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:717)
[2022-02-18 19:53:33,135] {bash.py:173} INFO -  at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
[2022-02-18 19:53:33,135] {bash.py:173} INFO -  at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
[2022-02-18 19:53:33,135] {bash.py:173} INFO -  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2022-02-18 19:53:33,135] {bash.py:173} INFO -  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2022-02-18 19:53:33,135] {bash.py:173} INFO -  at java.lang.Thread.run(Thread.java:748)
[2022-02-18 19:53:33,135] {bash.py:173} INFO -  Suppressed: java.lang.RuntimeException: Asynchronous task failed
[2022-02-18 19:53:33,135] {bash.py:173} INFO -   at com.google.api.gax.rpc.ServerStreamIterator.hasNext(ServerStreamIterator.java:105)
[2022-02-18 19:53:33,135] {bash.py:173} INFO -   at com.ebay.mkt.data.costdownload.google.service.GoogleAdsAPIReportDownloadHelper.writeCVRReponsesOnHdfs(GoogleAdsAPIReportDownloadHelper.java:383)
[2022-02-18 19:53:33,135] {bash.py:173} INFO -   at com.ebay.mkt.data.costdownload.google.service.GoogleAdsAPIReportDownloadHelper.downloadReportV4(GoogleAdsAPIReportDownloadHelper.java:313)
[2022-02-18 19:53:33,135] {bash.py:173} INFO -   at com.ebay.mkt.data.costdownload.google.job.DownloadGoogleRptV4Job$$anonfun$downloadTask$1$$anonfun$apply$1.apply(DownloadGoogleRptV4Job.scala:136)
[2022-02-18 19:53:33,135] {bash.py:173} INFO -   at com.ebay.mkt.data.costdownload.google.job.DownloadGoogleRptV4Job$$anonfun$downloadTask$1$$anonfun$apply$1.apply(DownloadGoogleRptV4Job.scala:119)
[2022-02-18 19:53:33,135] {bash.py:173} INFO -   at scala.util.Try$.apply(Try.scala:192)
[2022-02-18 19:53:33,135] {bash.py:173} INFO -   at com.ebay.mkt.data.common.utility.SafeTry$.apply(SafeTry.scala:8)
[2022-02-18 19:53:33,135] {bash.py:173} INFO -   at com.ebay.mkt.data.common.utility.SafeRetry$.retry(SafeRetry.scala:27)
[2022-02-18 19:53:33,135] {bash.py:173} INFO -   at com.ebay.mkt.data.common.utility.SafeRetry$.apply(SafeRetry.scala:17)
[2022-02-18 19:53:33,135] {bash.py:173} INFO -   at com.ebay.mkt.data.costdownload.google.job.DownloadGoogleRptV4Job$$anonfun$downloadTask$1.apply(DownloadGoogleRptV4Job.scala:119)
[2022-02-18 19:53:33,135] {bash.py:173} INFO -   at com.ebay.mkt.data.costdownload.google.job.DownloadGoogleRptV4Job$$anonfun$downloadTask$1.apply(DownloadGoogleRptV4Job.scala:117)
[2022-02-18 19:53:33,136] {bash.py:173} INFO -   at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
[2022-02-18 19:53:33,136] {bash.py:173} INFO -   at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
[2022-02-18 19:53:33,136] {bash.py:173} INFO -   ... 3 more
[2022-02-18 19:53:33,136] {bash.py:173} INFO - Caused by: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 3599.997897730s. [closed=[], open=[[buffered_nanos=207933631, buffered_nanos=166584391, remote_addr=*******]]]
[2022-02-18 19:53:33,136] {bash.py:173} INFO -  at io.grpc.Status.asRuntimeException(Status.java:535)
[2022-02-18 19:53:33,136] {bash.py:173} INFO -  ... 17 more

[Google Ads API Forum Advisor]

Hi Hao Long,

 

I suggest you implement exponential backoff pattern in your retry logic, if you haven't already tried this yet. Also, I suggest to split the report into smaller date ranges in order to improve the performance of the API. You can do some post-processing on your end to merge the reports into a single one once you have all the data you need.

 

If you are still consistently encountering issues after following my suggestions, please provide the complete request and response logs with request ID so we can further investigate. You may send the requested information via Reply privately to author option. If this option is not available, you may send the details directly to our googleadsa...@google.com alias instead.

 

Regards,

Kevin Gil Google Ads API Team

ref:_00D1U1174p._5004Q2WxQjP:ref

[Hao Long]

yes we have retry logic in our program(once failed it will retry after 10mins) but still get the same failed result.

i just confused we could one time download 45 days data but could not achieve this in new Google Ads API.

[Hao Long]

Thanks Kevin,

i will split the request into 9(5 days data per request).

Thanks

Hao

[Google Ads API Forum Advisor]

Hi Hao,

Thank you for your reply.

Regarding what you mentioned "..we could one time download 45 days data but could not achieve this in new Google Ads API.", could you also share your complete request and response logs, with the request-id, and login-customer-id used?

This is so that our team can also try and replicate your report and see whether we would also encounter the DEADLINE_EXCEEDED error.

In addition, could you also share the query you used when you were able to download the 45 days worth of data?

If you haven't yet, logging can be enabled for Java by referring to this guide. You may then send the requested information via the Reply privately to author option. If this option is not available, you may send the details directly to our googleadsa...@google.com alias instead.

Best regards,

Peter Laurence Google Ads API Team

ref:_00D1U1174p._5004Q2WxQjP:ref