Add Multiple Requests to the Same Job Facing CONCURRENT_MODIFICATION error.

[Online Engineering]

Hi,

We are trying to upload pii data to create the user lists in Google Ads manager and face the CONCURRENT_MODIFICATION error.

When we read the following paragraph from the doc, our understanding is that as long as we run one job, we should not run into the CONCURRENT_MODIFICATION issue. So we added the requests concurrently to the same job, and submitted the job for only once. However, we still saw this error.

 

• Avoid simultaneously running multiple OfflineUserDataJob processes that modify the same user list (that is, multiple jobs whose CustomerMatchUserListMetadata.user_list point to the same resource name). Doing so can result in a CONCURRENT_MODIFICATION error since multiple jobs are not permitted to operate on the same list at the same time. This error can also occur if attempting to simultaneously modify a list through the Google Ads UI and the Google Ads API. Note that this does not apply to adding operations to an existing job, which can be done at any time before the job is started.

 

Could you help us address following concerns?

1. Is adding requests to the same job concurrently supported without throwing out the CONCURRENT_MODIFICATION error?
2. If number 1 is supported, then is it a bug development team needs to trouble shoot?
3. If number 1 is not supported, could you suggest if there is any other way to upload the data asynchronously so that we can upload the data faster than the synchronous way?

 

Here I’m pasting how we tested the API again.

 

Following work flow is implemented to upload data to Google in parallel.

1. Created 'google_client' using access tokens

2. Created 'user_list' (we at Epsilon refer it as audience)

3. Created 'offline_user_data_job' using 'OfflineUserDataJobService' service

4. Create a function which takes the 'google_client', 'user_list', 'offline_user_data_job' and small portion of dataframe to be uploaded. This function is called in parallel by spark.

5. Within this function,

               5.1 Create request for 'AddOfflineUserDataJobOperations'

               5.2 Initialize the request with 'offline_user_data_job' ( This will tag a request to job.

               5.3 Create operations using the input data. and assign to request.operations

               5.4 submit the 'AddOfflineUserDataJobOperations' request

               5.5 Repeat step 5.1 to 5.4 till all data is submitted to Google.

6. On submitting all the 'AddOfflineUserDataJobOperations' request, finally call 'run_offline_user_data_job' which triggers the upload.

 

Issue details:

We are facing 'database_error: CONCURRENT_MODIFICATION' error when we are submitting "'AddOfflineUserDataJobOperations' request" in parallel. ( Step 5.4)

 

(<_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.INVALID_ARGUMENT\n\tdetails = "Request contains an invalid argument."\n\tdebug_error_string = "

{

    "created": "@1647967680.852767877",

    "description": "Error received from peer ipv4:172.253.122.95:443",

    "file": "src/core/lib/surface/call.cc",

    "file_line": 903,

    "grpc_message": "Request contains an invalid argument.",

    "grpc_status": 3

}

"\n>, <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.INVALID_ARGUMENT\n\tdetails = "Request contains an invalid argument."\n\tdebug_error_string = "

{

    "created": "@1647967680.852767877",

    "description": "Error received from peer ipv4:172.253.122.95:443",

    "file": "src/core/lib/surface/call.cc",

    "file_line": 903,

    "grpc_message": "Request contains an invalid argument.",

    "grpc_status": 3

}

"\n>, errors {\n  error_code {\n    database_error: CONCURRENT_MODIFICATION\n  }\n  message: "Multiple requests were attempting to modify the same resource at once. Please retry the request."\n}\nrequest_id: "jUsAGBf4dnPrg6XQX8TEbA"\n, \'jUsAGBf4dnPrg6XQX8TEbA\')',)

 

 

 

[Google Ads API Forum Advisor]

Hi,

Thank you for reaching out to the Google Ads API support team.

In general, the CONCURRENT_MODIFICATION error occurs due to multiple processes that are trying to update the same entity at the same time. This also includes updating the entity in Google Ads UI while updating via the API as well. We recommend waiting for about 30 seconds, then retry the request via API to avoid the CONCURRENT_MODIFICATION error. 

That being said, you’ll need to ensure that only one operation is performed against an entity at each moment and slow down on the consolidated repeated mutates against the same entity into a single request to avoid this issue altogether. Please give this a try and let me know if you have any additional concerns.

 ​
Thanks,

Nirmita Google Ads API Team

ref:_00D1U1174p._5004Q2Ysj9k:ref

[Sree kurukuri]

Hi Nirmita,

I'm facing the same issue. According to the documentation, we can send multiple add operations requests at the same time. But i still get the  CONCURRENT_MODIFICATION error. can you please help with this ?

Thanks,

Sree

[Matt Dworkin]

Hello,

We're having the same problem. It's not even clear that we're mutating the same entity. As far as we can tell we're not doing that, so we're a bit confused as to what's going on. If needed we can provide request ids via logs. Thanks!

Best,
Matt

[Google Ads API Forum Advisor]

Hi Matt,

Thank you for raising your concern to our team.

To confirm first, could you clarify if you are using other tools like Google Ads Editor or UI that make changes on the same resources at the same time? Could you confirm this on your end?

However, you can indeed provide the complete request and response logs with request ID and request header generated on your end so that we can further investigate.

You can provide it via Reply privately to author option. If this option is not available, then send it instead on this email address googleadsa...@google.com.

Regards,

Ernie John Google Ads API Team

ref:_00D1U1174p._5004Q2Ysj9k:ref