CONCURRENT_MODIFICATION while populate new users to audience group

Konstantin

unread,

Dec 26, 2021, 12:03:23 AM12/26/21

to AdWords API and Google Ads API Forum

i try to create to populate users (about 20M) in group with addOfflineUserDataJobOperations method

I use batch to upload with size 100k

One thread, sleep about 4s between calls and
Also i check this error (CONCURRENT_MODIFICATION) and add retry for it with sleep 40s

No any success for 2 days (((

My error is:

errors { error_code { database_error: CONCURRENT_MODIFICATION } message: "Multiple requests were attempting to modify the same resource at once. Please retry the request." } request_id: "izSkpssg3niNwY-oUn9GTg"

Could you help me with any advice ?

Google Ads API Forum Advisor

unread,

Dec 27, 2021, 6:27:51 AM12/27/21

to mishe...@gmail.com, adwor...@googlegroups.com

Hi,

Thanks for reaching out to us.

In general, the CONCURRENT_MODIFICATION error occurs due to multiple processes that are trying to update the same entity at the same time. This also includes updating the entity in Google Ads UI while updating via the API as well. We recommend waiting for about 30 seconds, then retry the request.

That being said, you’ll need to ensure that only one operation is performed against an entity at each moment and slow down on the consolidated repeated mutates against the same entity into a single request to avoid this issue altogether. Please give this a try and let me know if you have any additional concerns.

Regards,

Yasar
Google Ads API Team

ref:_00D1U1174p._5004Q2TkoQ6:ref

Konstantin

unread,

Dec 28, 2021, 9:06:32 AM12/28/21

to AdWords API and Google Ads API Forum

Hi
Thanks for your relay

Yes! I find this answer already in this group

Now i have to wait about from 15 to 30 minutes to next success api call?

Is it normal ?

Google Ads API Forum Advisor

unread,

Dec 28, 2021, 2:53:49 PM12/28/21

to mishe...@gmail.com, adwor...@googlegroups.com

Hello there,

Thanks for replying back to us. We recommend waiting for about 30 seconds (not 30 mins) as suggested by another colleague of mine, then try another request via API to avoid the CONCURRENT_MODIFICATION error. This error occurs due to multiple requests attempting to modify the same resource at the same time.
Please retry the request. If the issue persists, could you please provide us with the complete request and response logs with request-id via the Reply privately to the author option while sharing the details.

Best regards,

Jakeia Sabrina
Google Ads API Team

ref:_00D1U1174p._5004Q2TkoQ6:ref

Message has been deleted

Google Ads API Forum Advisor

unread,

Dec 29, 2021, 1:05:53 AM12/29/21

to mishe...@gmail.com, adwor...@googlegroups.com

Hello,

Thank you for providing additional information, however, we need the complete request and response logs with request-id to further investigate. If you're using one of our client libraries, detailed logging can be enabled by navigating to the Client libraries > Your client library (ex. Java) > Logging documentation, which you can access from this link.

Please send the requested information via Reply privately to author option. If this option is not available, you may send the details directly to our googleadsa...@google.com alias instead. Also, I've deleted your previous post as it contains private information. Below is the copy of your post with scrubbed information.

For example... i get an error, wait about 2 minutes and get the error again

2021-12-29 05:37:49,977 WARN GoogleAdsProvider: Failed to perform action#1: com.google.ads.googleads.v7.errors.GoogleAdsException: errors {

error_code {

database_error: CONCURRENT_MODIFICATION

}

message: "Multiple requests were attempting to modify the same resource at once. Please retry the request."

}

request_id: "xxxxxxxxxxxxxxxxxxxxxx"

2021-12-29 05:39:31,536 WARN GoogleAdsProvider: Failed to perform action#2: com.google.ads.googleads.v7.errors.GoogleAdsException: errors {

error_code {

database_error: CONCURRENT_MODIFICATION

}

message: "Multiple requests were attempting to modify the same resource at once. Please retry the request."

}

request_id: "xxxxxxxxxxxxxxxxxxxxxx"

Regards,

Kevin Gil Soriano
Google Ads API Team

ref:_00D1U1174p._5004Q2TkoQ6:ref

Konstantin

unread,

Dec 29, 2021, 2:43:23 PM12/29/21

to AdWords API and Google Ads API Forum

Hi
For example
error in 2021-12-29 05:37:49,977 
first retry in 2021-12-29 05:39:31,536 retry with error
success in 2021-12-29 05:44:32,084 



2021-12-29 05:37:36,983 INFO  BazingaTaskQueueName_name_segment_filter_export__Worker_2#rid=segmentFilterExport/20211228T214652.579Z/r.y.alet.crm.core.googleads.GoogleAdsProvider: Created an offline user data job with resource name: customers/5687337496/offlineUserDataJobs/15828107152
2021-12-29 05:37:49,977 WARN  BazingaTaskQueueName_name_segment_filter_export__Worker_2#rid=segmentFilterExport/20211228T214652.579Z/r.y.alet.crm.core.googleads.GoogleAdsProvider: Failed to perform action#1: com.google.ads.googleads.v7.errors.GoogleAdsException: errors {

  error_code {
    database_error: CONCURRENT_MODIFICATION
  }
  message: "Multiple requests were attempting to modify the same resource at once. Please retry the request."
}

request_id: "xU1ZDYrvYnsmJ-oTtjXOWQ"

com.google.ads.googleads.v7.errors.GoogleAdsException: errors {
  error_code {
    database_error: CONCURRENT_MODIFICATION
  }
  message: "Multiple requests were attempting to modify the same resource at once. Please retry the request."
}

request_id: "xU1ZDYrvYnsmJ-oTtjXOWQ"

    at com.google.ads.googleads.v7.errors.GoogleAdsException$Factory.createException(GoogleAdsException.java:42) ~[google-ads-12.0.0.jar:?]
    at com.google.ads.googleads.v7.errors.GoogleAdsException$Factory.createException(GoogleAdsException.java:34) ~[google-ads-12.0.0.jar:?]
    at com.google.ads.googleads.lib.BaseGoogleAdsException$Factory.createGoogleAdsException(BaseGoogleAdsException.java:105) ~[google-ads-12.0.0.jar:?]
    at com.google.ads.googleads.lib.callables.GoogleAdsExceptionTransformation.transform(GoogleAdsExceptionTransformation.java:36) ~[google-ads-12.0.0.jar:?]
    at com.google.ads.googleads.lib.callables.ExceptionTransformingUnaryCallable$ExceptionTransformingFuture.onFailure(ExceptionTransformingUnaryCallable.java:82) ~[google-ads-12.0.0.jar:?]
    at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68) ~[api-common-1.10.1.jar:?]
    at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1052) ~[guava-guava-26.0-jre.jar:?]
    at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:398) ~[guava-guava-26.0-jre.jar:?]
    at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1029) ~[guava-guava-26.0-jre.jar:?]
    at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:871) ~[guava-guava-26.0-jre.jar:?]
    at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:716) ~[guava-guava-26.0-jre.jar:?]
    at com.google.api.gax.retrying.BasicRetryingFuture.handleAttempt(BasicRetryingFuture.java:200) ~[gax-1.63.0.jar:1.63.0]
    at com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.handle(CallbackChainRetryingFuture.java:135) ~[gax-1.63.0.jar:1.63.0]
    at com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.run(CallbackChainRetryingFuture.java:117) ~[gax-1.63.0.jar:1.63.0]
    at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:398) ~[guava-guava-26.0-jre.jar:?]
    at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1029) ~[guava-guava-26.0-jre.jar:?]
    at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:871) ~[guava-guava-26.0-jre.jar:?]
    at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:716) ~[guava-guava-26.0-jre.jar:?]
    at com.google.api.core.AbstractApiFuture$InternalSettableFuture.setException(AbstractApiFuture.java:95) ~[api-common-1.10.1.jar:?]
    at com.google.api.core.AbstractApiFuture.setException(AbstractApiFuture.java:77) ~[api-common-1.10.1.jar:?]
    at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97) ~[gax-grpc-1.63.0.jar:1.63.0]
    at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68) ~[api-common-1.10.1.jar:?]
    at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1052) ~[guava-guava-26.0-jre.jar:?]
    at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:398) ~[guava-guava-26.0-jre.jar:?]
    at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1029) ~[guava-guava-26.0-jre.jar:?]
    at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:871) ~[guava-guava-26.0-jre.jar:?]
    at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:716) ~[guava-guava-26.0-jre.jar:?]
    at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:522) ~[grpc-grpc-stub-1.26.0.jar:1.26.0]
    at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:497) ~[grpc-grpc-stub-1.26.0.jar:1.26.0]
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) ~[grpc-grpc-api-1.26.0.jar:1.26.0]
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[grpc-grpc-api-1.26.0.jar:1.26.0]
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[grpc-grpc-api-1.26.0.jar:1.26.0]
    at com.google.ads.googleads.lib.logging.LoggingInterceptor$1$1.onClose(LoggingInterceptor.java:113) ~[google-ads-12.0.0.jar:?]
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) ~[grpc-grpc-api-1.26.0.jar:1.26.0]
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[grpc-grpc-api-1.26.0.jar:1.26.0]
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[grpc-grpc-api-1.26.0.jar:1.26.0]
    at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:700) ~[grpc-core-1.26.0.jar:1.26.0]
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) ~[grpc-grpc-api-1.26.0.jar:1.26.0]
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[grpc-grpc-api-1.26.0.jar:1.26.0]
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[grpc-grpc-api-1.26.0.jar:1.26.0]
    at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:399) ~[grpc-core-1.26.0.jar:1.26.0]
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:521) ~[grpc-core-1.26.0.jar:1.26.0]
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:66) ~[grpc-core-1.26.0.jar:1.26.0]
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:641) ~[grpc-core-1.26.0.jar:1.26.0]
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:529) ~[grpc-core-1.26.0.jar:1.26.0]
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:703) ~[grpc-core-1.26.0.jar:1.26.0]
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:692) ~[grpc-core-1.26.0.jar:1.26.0]
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.26.0.jar:1.26.0]
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) ~[grpc-core-1.26.0.jar:1.26.0]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
    at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
    at java.lang.Thread.run(Thread.java:834) [?:?]
    Suppressed: com.google.api.gax.rpc.AsyncTaskException: Asynchronous task failed
        at com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:57) ~[gax-1.63.0.jar:1.63.0]
        at com.google.api.gax.rpc.UnaryCallable.call(UnaryCallable.java:112) ~[gax-1.63.0.jar:1.63.0]
        at com.google.ads.googleads.v7.services.OfflineUserDataJobServiceClient.addOfflineUserDataJobOperations(OfflineUserDataJobServiceClient.java:481) ~[google-ads-12.0.0.jar:?]
        ...
Caused by: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Request contains an invalid argument.
    at io.grpc.Status.asRuntimeException(Status.java:533) ~[grpc-grpc-api-1.26.0.jar:1.26.0]
    ... 27 more
2021-12-29 05:39:31,536 WARN  BazingaTaskQueueName_name_segment_filter_export__Worker_2#rid=segmentFilterExport/20211228T214652.579Z/r.y.alet.crm.core.googleads.GoogleAdsProvider: Failed to perform action#2: com.google.ads.googleads.v7.errors.GoogleAdsException: errors {

  error_code {
    database_error: CONCURRENT_MODIFICATION
  }
  message: "Multiple requests were attempting to modify the same resource at once. Please retry the request."
}

request_id: "8gHaQVPfYTFTi3lCV9HeIA"

com.google.ads.googleads.v7.errors.GoogleAdsException: errors {
  error_code {
    database_error: CONCURRENT_MODIFICATION
  }
  message: "Multiple requests were attempting to modify the same resource at once. Please retry the request."
}

request_id: "8gHaQVPfYTFTi3lCV9HeIA"

    at com.google.ads.googleads.v7.errors.GoogleAdsException$Factory.createException(GoogleAdsException.java:42) ~[google-ads-12.0.0.jar:?]
    at com.google.ads.googleads.v7.errors.GoogleAdsException$Factory.createException(GoogleAdsException.java:34) ~[google-ads-12.0.0.jar:?]
    at com.google.ads.googleads.lib.BaseGoogleAdsException$Factory.createGoogleAdsException(BaseGoogleAdsException.java:105) ~[google-ads-12.0.0.jar:?]
    at com.google.ads.googleads.lib.callables.GoogleAdsExceptionTransformation.transform(GoogleAdsExceptionTransformation.java:36) ~[google-ads-12.0.0.jar:?]
    at com.google.ads.googleads.lib.callables.ExceptionTransformingUnaryCallable$ExceptionTransformingFuture.onFailure(ExceptionTransformingUnaryCallable.java:82) ~[google-ads-12.0.0.jar:?]
    at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68) ~[api-common-1.10.1.jar:?]
    at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1052) ~[guava-guava-26.0-jre.jar:?]
    at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:398) ~[guava-guava-26.0-jre.jar:?]
    at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1029) ~[guava-guava-26.0-jre.jar:?]
    at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:871) ~[guava-guava-26.0-jre.jar:?]
    at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:716) ~[guava-guava-26.0-jre.jar:?]
    at com.google.api.gax.retrying.BasicRetryingFuture.handleAttempt(BasicRetryingFuture.java:200) ~[gax-1.63.0.jar:1.63.0]
    at com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.handle(CallbackChainRetryingFuture.java:135) ~[gax-1.63.0.jar:1.63.0]
    at com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.run(CallbackChainRetryingFuture.java:117) ~[gax-1.63.0.jar:1.63.0]
    at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:398) ~[guava-guava-26.0-jre.jar:?]
    at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1029) ~[guava-guava-26.0-jre.jar:?]
    at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:871) ~[guava-guava-26.0-jre.jar:?]
    at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:716) ~[guava-guava-26.0-jre.jar:?]
    at com.google.api.core.AbstractApiFuture$InternalSettableFuture.setException(AbstractApiFuture.java:95) ~[api-common-1.10.1.jar:?]
    at com.google.api.core.AbstractApiFuture.setException(AbstractApiFuture.java:77) ~[api-common-1.10.1.jar:?]
    at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97) ~[gax-grpc-1.63.0.jar:1.63.0]
    at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68) ~[api-common-1.10.1.jar:?]
    at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1052) ~[guava-guava-26.0-jre.jar:?]
    at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:398) ~[guava-guava-26.0-jre.jar:?]
    at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1029) ~[guava-guava-26.0-jre.jar:?]
    at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:871) ~[guava-guava-26.0-jre.jar:?]
    at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:716) ~[guava-guava-26.0-jre.jar:?]
    at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:522) ~[grpc-grpc-stub-1.26.0.jar:1.26.0]
    at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:497) ~[grpc-grpc-stub-1.26.0.jar:1.26.0]
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) ~[grpc-grpc-api-1.26.0.jar:1.26.0]
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[grpc-grpc-api-1.26.0.jar:1.26.0]
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[grpc-grpc-api-1.26.0.jar:1.26.0]
    at com.google.ads.googleads.lib.logging.LoggingInterceptor$1$1.onClose(LoggingInterceptor.java:113) ~[google-ads-12.0.0.jar:?]
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) ~[grpc-grpc-api-1.26.0.jar:1.26.0]
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[grpc-grpc-api-1.26.0.jar:1.26.0]
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[grpc-grpc-api-1.26.0.jar:1.26.0]
    at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:700) ~[grpc-core-1.26.0.jar:1.26.0]
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) ~[grpc-grpc-api-1.26.0.jar:1.26.0]
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[grpc-grpc-api-1.26.0.jar:1.26.0]
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[grpc-grpc-api-1.26.0.jar:1.26.0]
    at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:399) ~[grpc-core-1.26.0.jar:1.26.0]
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:521) ~[grpc-core-1.26.0.jar:1.26.0]
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:66) ~[grpc-core-1.26.0.jar:1.26.0]
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:641) ~[grpc-core-1.26.0.jar:1.26.0]
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:529) ~[grpc-core-1.26.0.jar:1.26.0]
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:703) ~[grpc-core-1.26.0.jar:1.26.0]
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:692) ~[grpc-core-1.26.0.jar:1.26.0]
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.26.0.jar:1.26.0]
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) ~[grpc-core-1.26.0.jar:1.26.0]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
    at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
    at java.lang.Thread.run(Thread.java:834) [?:?]
    Suppressed: com.google.api.gax.rpc.AsyncTaskException: Asynchronous task failed
        at com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:57) ~[gax-1.63.0.jar:1.63.0]
        at com.google.api.gax.rpc.UnaryCallable.call(UnaryCallable.java:112) ~[gax-1.63.0.jar:1.63.0]
        at com.google.ads.googleads.v7.services.OfflineUserDataJobServiceClient.createOfflineUserDataJob(OfflineUserDataJobServiceClient.java:229) ~[google-ads-12.0.0.jar:?]
        at com.google.ads.googleads.v7.services.OfflineUserDataJobServiceClient.createOfflineUserDataJob(OfflineUserDataJobServiceClient.java:197) ~[google-ads-12.0.0.jar:?]
        ...
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Request contains an invalid argument.
    at io.grpc.Status.asRuntimeException(Status.java:533) ~[grpc-grpc-api-1.26.0.jar:1.26.0]
    ... 27 more
2021-12-29 05:44:32,084 INFO  BazingaTaskQueueName_name_segment_filter_export__Worker_2#rid=segmentFilterExport/20211228T214652.579Z/r.y.alet.crm.core.googleads.GoogleAdsProvider: Created an offline user data job with resource name: customers/5687337496/offlineUserDataJobs/15827545742

Google Ads API Forum Advisor

unread,

Dec 30, 2021, 1:18:32 PM12/30/21

to mishe...@gmail.com, adwor...@googlegroups.com

Hi Konstantin,

Please allow me to share this with our team and we will provide an update shortly.

Aside: I see you are using v7 and the Java client library. If you are using Log4J to log, please update to the latest version due to a vulnerability found in the logging utility.

Regards,
Matt
Google Ads API Team

Reminder: Share your feedback about the Google Ads (AdWords) API! Take the 2021 Google Ads API and AdWords API Annual Survey

Matt
Google Ads API Team

ref:_00D1U1174p._5004Q2TkoQ6:ref

Google Ads API Forum Advisor

unread,

Dec 30, 2021, 6:26:00 PM12/30/21

to adwor...@googlegroups.com, mishe...@gmail.com

Greetings!

I did an analysis of your requests on our servers, and I found a pattern that appears to be causing these errors. I'll cover my observations followed by some best practices, so that these errors are not as frequent.

Observations

I noticed that there was a pattern in how the OfflineUserDataJobService was being called. The pattern was similar to this:

Create the offline data job for the user list.
Immediately add operations with one request.
Run the job.
Immediately create another data job for the same user list.
Immediately add operations with one request.
Run the job.
And so on...

When running offline user data jobs in this way, the concurrent modification exception is one I would expect. Heres's why. Every time a request is made in reference to an object like a user list, the user list resource is locked down so two threads don't accidentally cause a race condition. In the pattern I'm seeing, a new job is created for a user list before the previous job has completed. I saw repeatedly that the new job was created in the same second or the second after run job method was called on the same user list. This is where the concurrent modification exception comes in. Due to the user list still being locked down by the previous job adding data to it, the following requests can sometimes can locked out. I can give you some best practices next to explain how to structure your jobs to make that less of a chance.

Best practices

There are two best practices here that will help stop this error from happening.

Best Practice #1 Do not have multiple jobs running in parallel for the same user list. What would need to change in your code would be to wait until the job has finished running for a user list by polling the job periodically until it's done. Then, create a new job for that user list. What I've seen our community do is have multiple offline data jobs running, but each offline data job is updating a different user list. So, they have one thread per user list running with multiple threads in parallel. That way they can still get great throughput on multiple user lists being updated without causing the error.

Best Practice #2 Create larger jobs. What most people do is that for a single user list, they will create an offline job, send an add operations request, then another add operations request with more operations, then another operations create and so on, and then call run job. This creates a larger job. What this does is it means that you call create job and run job less often. A single job may take longer, but because you don't have to call create job and run job so often, you actually save time over a longer period. This is especially the case when you need to upload large amounts of data to a single user list.

Please give those best practices a try. That should clear up those errors that you're seeing.

Regards,

Nadine Wang
Google Ads API Team

ref:_00D1U1174p._5004Q2TkoQ6:ref

Reply all

Reply to author

Forward