How to handle very huge amount of data upload

[Paul Joy Parayannilam]

Hello Team,

As per my requirement, I have to use the Adwords API to upload records to Adwords account against various campaigns. Please help with answers for below questions.

1. I'm planning to have about 3500 to 4000 campaigns in the Adwords side.

2. Across all campaigns together, the number of records that I have to upload will most likely run into billions.

Question:

1. What is the best approach to handle this scenario? 

2. I have read somewhere that Google suggests to keep number of records per upload to about 100k to 200k. Is this True?

3. With billions of records to be uploaded (against one client account under which i have about 4000 campaigns), what should be the strategy to be adopted. Does it support multi threading where I can upload records in parallel in an asynchronous mode?

If I perform sequentially, it might take hours or days for me to just perform the upload.

Thanks

Paul

[Vincent Racaza (AdWords API Team)]

Hi Paul,

I will just summarize my answer for your three questions.

In regards to handing large amount of records in asynchronous mode in the AdWords API, you can use the BatchJobService for this. This service allows you to perform batches of supported operations on multiple services without waiting for the operations to complete. You can refer to this guide for more information regarding batch processing.

In regards to the limitation on the number of records for batch processing, an AdWords account can have up to 1 GB of uploaded operations across all of its batch jobs that have not yet completed at any given time.

Let me know if this helps.

Thanks,

Vincent

AdWords API Team

[Paul Joy Parayannilam]

Thank You Vincent. I have below followup question.

To help me understand correctly, lets say I have about 2-3 billion records( actual could be a lot more) to be uploaded across multiple campaigns.

So, using the batchjobservice, i should be able to upload upto about 1 GB of data at any given time against CampaignCriterionService. Is that a correct understanding?

Also, how much time it will take when dealing with such huge record sets?

Thanks

Paul

[Vincent Racaza (AdWords API Team)]

Hi Paul,

Yes, your understanding is correct. The limit is 1 GB of uploaded operations across all batch jobs that have not yet completed at any given time. So if you have non-completed operations for CampaignCriterionService, then the limit is 1 GB across all operations in all batch jobs at any given time. You can also check this section for the best practices in using the BatchJobService.

In regards to how much time it will take, my apologies as we do not have any specific information on this. The overall duration would depend on the number of batch jobs and the number of operations per batch job.