DatabaseError.CONCURRENT_MODIFICATION error

[Qian Huang]

Hi,

when I send mutateMembers requests, there are a lots DatabaseError.CONCURRENT_MODIFICATION error:

<soap:Body>

       <soap:Fault>

           <faultcode>soap:Client</faultcode>

           <faultstring>[DatabaseError.CONCURRENT_MODIFICATION @ com.google.ads.api.services.campaignmgmt.common.error.DatabaseError.&lt;init&gt;(Datab]</faultstring>

           <detail>

               <ns2:ApiExceptionFault xmlns="https://adwords.google.com/api/adwords/cm/v201809" xmlns:ns2="https://adwords.google.com/api/adwords/rm/v201809">

                   <message>[DatabaseError.CONCURRENT_MODIFICATION @ com.google.ads.api.services.campaignmgmt.common.error.DatabaseError.&lt;init&gt;(Datab]</message>

                   <ApplicationException.Type>ApiException</ApplicationException.Type>

                   <errors xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="DatabaseError">

                       <fieldPath/>

                       <trigger/>

                       <errorString>DatabaseError.CONCURRENT_MODIFICATION</errorString>

                       <ApiError.Type>DatabaseError</ApiError.Type>

                       <reason>CONCURRENT_MODIFICATION</reason>

                   </errors>

               </ns2:ApiExceptionFault>

           </detail>

       </soap:Fault>

   </soap:Body>

most of my requests only wrap hashed email, hashed phoneNumber and address, example as below:

...

       <mutateMembers xmlns="https://adwords.google.com/api/adwords/rm/v201809">

           <operations>

               <ns7:operator xmlns:ns7="https://adwords.google.com/api/adwords/cm/v201809">ADD</ns7:operator>

               <operand>

                   <userListId>0000000</userListId>

                   <removeAll>false</removeAll>

                   <membersList>

                       <hashedEmail>123123123123123adfadfadfadfasdf0</hashedEmail>

                       <hashedPhoneNumber>asdfasdfasdfasdfasdfasdfsdf</hashedPhoneNumber>

                       <addressInfo>

                           <hashedFirstName>asdfasdfasdfasdfasdfasdfadsf</hashedFirstName>

                           <hashedLastName>asdfasfasdfdf12313434234</hashedLastName>

                           <countryCode>ES</countryCode>

                           <zipCode>123124</zipCode>

                       </addressInfo>

                   </membersList>
....

From the adwords api documentation, it may happen when we modify the same entity concurrently.

My question is how to identify an entity in my case? or say what's the primary field to identify an entity?

And could you give some suggestions about how to avoid this exception?

Best Regards,

Qian

[Google Ads API Forum Advisor Prod]

Hello Qian,

Thank you for reaching out. The CONCURRENT_MODIFICATION error is observed when multiple processes are trying to make changes to a single entity at once. Could you please check the Change History in the Google Ads interface to verify if there were no other changes being made to the account or no scripts were running in the background at the time of the mutate member request? Also, you can implement the error handling model in their code to handle such scenarios where partial failure should be set up and the operations that failed should be retried. Please give this a try and let us know if the issue persists.

Regards,
Nikisha Patel, Google Ads API Team

ref:_00D1U1174p._5001UV0nZJ:ref

[Qian Huang]

Hi Nikisha,

I checked change history, looks like only one script is sending request. However, the history are quite general, it only says: "Remarketing list changed", "Members are uploaded to add to this list".

Could you tell more details about that for this mutate member api, what's the defination of entity? in other words, on what field this concurrent modification happend? so we can improve our code to avoid it.

Best Regards,

Qian

[Google Ads API Forum Advisor Prod]

Hello Qian,

The entity in this scenario will refer to the user list that you are trying to update via API for mutating members of the user list. The concurrent modification error indicates that two or more processes are trying to update the same user list at same instance of time. It might also occur when a user in the AdWords web interface is updating the user list while you are updating the same entity via the API as well. Could you please confirm that even after waiting for about 30 seconds before retrying the request again, the issue is still occurring? Please refer to this guide to avoid such error in future.

Regards,
Nikisha Patel, Google Ads API Team

ref:_00D1U1174p._5001UV0nZJ:ref

[Qian Huang]

Hi Nikisha,

For one user list, we will add millions of users, so currently we chunk them into several requests, each request will contain 1000 user mutation, then we send these requests concurrently. 

If the concurrent modification error is happend on user list level, it means we can't send these request concurrently. 

But we have so many users need to add, it would be very slow if we send request in one thread, could you give some suggestion in this case?

Best Regards,

Qian

[Google Ads API Forum Advisor Prod]

Hello Qian,

As a suggestion to avoid such errors in a multi threaded environment, I would like to share some best practices from this handy guide. One recommendation in that guide includes : "When you retry requests, use an exponential backoff policy. For example, if you first pause 5 seconds before the first retry, you could pause 10 seconds after the second and 20 seconds after the third retry. Exponential back off helps ensure you are not calling the API too aggressively." As a workaround you could 1) space the requests against the same ID so that any given one finishes before the next one is sent, or 2) consolidate repeated mutates against the same entity into a single request to avoid this issue altogether. Please give this a try and let me know if you have any additional concerns.

Regards,
Nikisha Patel, Google Ads API Team

ref:_00D1U1174p._5001UV0nZJ:ref