API exports takes a huge amount of time due to the number of clients available

[business_intel...@vivaki.de]

Hi all,

we have an export setup which includes a total of 6 reports.

4 reports are awql and 2 are selector reports.

we have a total of 2572 clients available for download (ignoring any MCCs)

now adding 6 rports per client we have a total of 15432 reports to download.

approx. 270 client are active and deliver data in the reports the other reports are 1kb empty files.

now starting the export (last 7 days per report for conversions last 30) it makes ~8 hours to download.

checking the log we can see that there are no errors or any downtimes which cause the huge amount of time its just the tremendous amount of files. we were able to see that for a single report ~2seconds are spent. for 15k reports its around: 30864 seconds -> ~8,5hours

is there any solution to consolidate report downloads (maybe in packages)?

or is it possible to download reports on MCC level?

we use the following reports:

1. keyword_performance (preclick + postclick data in two files)
   
2. ad_performance (preclick + postclick data in two files)
   
3. campaign_performance (masterdata + sov for channel 'search')
   

restricting the accounts to exports ins't really a solution because as a service provider we do not actually no when a new client is active / added and the requiremen is to include

any new /active clients automatically.

any ideas?

[Shwetha Vastrad (AdWords API Team)]

Hi, 

A given report request can only include data from a single AdWords account. If you need to gather reporting data for multiple accounts, you need to submit a separate report request for each account by setting the clientCustomerId header. You could refer to the ParallelReportDownload Java example which shows how to download a report for all accounts directly under a manager account in multiple parallel threads. Example code in other languages is provided here. 

You could also consider using the AwReporting sample application which is an open-source Java framework for large-scale AdWords API reporting. It's based on the Java AdWords API client library, using which you can generate reports for all AdWords accounts under a specified manager account. Please note that the Account level and developer token level rate limits still apply. You may encounter RateExceededError for exceeding rate limit within the Account scope. 

Regards,

Shwetha, AdWords API Team.

[Zweitze]

I see two possibilities:

- Almost 90% of the downloads is empty, yet still you download them. Are those empty downloads needed? If not: just download the masterdata campaign report first, and only download the other reports when the first report actually has contents. This should cut back the number of reports by 75 to 80%

- Multithreading or multiple servers. Divide the workload over multiple threads, processors or servers. I don't think all these reports have to wait on each other.

Hope this helps.