Slow performance adding 25M+ users to adwords list

[mit...@actioniq.com]

I'm trying to add many users (25 million email hashes) to a newly created adwords list. I'm doing this via many (around 30+) threads each doing a single big MutateMembersOperand with many users. I get CONCURRENT_MODIFICATION a lot, and just retry after a random wait time, but this still causes the performance to be very bad.

I read here that some aspects of the API now allow concurrent changes. Is it true that adding users to an ad list is still something that cannot be done concurrently? Is there a plan to improve this?

Thanks!

--

Mitesh

[googleadsapi...@google.com]

Hello Mitesh,

The CONCURRENT_MODIFICATION error that you encountered could occur due to multiple processes that are trying to update the same entity at the same time. To be able to troubleshoot on our end, could you please share the SOAP logs along with your CID from one of the requests that failed with this error. You can use 'Reply Privately to author' option while responding. 

Thanks,
Milind, AdWords API Team

=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog and discussion group:
    http://googleadsdeveloper.blogspot.com/search/label/adwords_api
    https://developers.google.com/adwords/api/community/
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~

Was your question answered? Please rate your experience with us by taking a short survey.
If not -- reply to this email and tell us what else we can do to help.

Take Survey

Also find us on our blog and discussion group:
http://googleadsdeveloper.blogspot.com/search/label/adwords_api
https://developers.google.com/adwords/api/community/

[mit...@actioniq.com]

Hi Milind,

Yes I absolutely am using multiple processes. Each one calls AdwordsServices.getUserList, then calls MutateMembers with about 50000 user email hashes. They all run at the same time, because I have noticed that running one at a time is too slow (takes 1 hour). Running them all concurrently, it reduces the time to 40 mins, but its still very slow.

My real question is when will Adwords API allow concurrent adding of users to a single UserList? This seems like a very critical feature, otherwise how do other people use the API to create UserLists with 20M+ users?

[googleadsapi...@google.com]

Hi Mitesh,

Here are some workarounds to your issue:

• Instead of passing 50K emails in each thread, you can pass up to 1 million emails in a single operation. That would reduce the number of total operations required. Hence, you can run your process with less threads.
• If multi-threading is a must, when a CONCURRENT_MODIFICATION error occurs, you can use exponential backoff to spread out the threads in each retry. 
• Each thread can start with a different start time. The idea is to spread out the threads to avoid collision. 

Note that processing large data sets for a relatively longer time is normal. 

Thanks,
Poki, AdWords API Team

=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog and discussion group:
    http://googleadsdeveloper.blogspot.com/search/label/adwords_api
    https://developers.google.com/adwords/api/community/
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~

--
--
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog:
https://googleadsdeveloper.blogspot.com/
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
 
You received this message because you are subscribed to the Google
Groups "AdWords API and Google Ads API Forum" group.
To post to this group, send email to adwor...@googlegroups.com
To unsubscribe from this group, send email to
adwords-api+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
---
You received this message because you are subscribed to the Google Groups "AdWords API and Google Ads API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to adwords-api+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/adwords-api.
To view this discussion on the web visit https://groups.google.com/d/msgid/adwords-api/aecbde29-fc36-4eb5-94b2-c3f0749532bf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.