Optimizing requests for big amount of data

[Boris DETRY]

I'm trying to query search terms for a period of one year (approx. 2M search terms).

Using searchstream the request is relatively fast, but the memory goes way too high. (almost 9 gigabytes)

Here is a memory snapshot: 

As you can see "Campaign " uses almost 700mb for the campaign names and resource names, here are the fields I use :

Other alternative would be using search and paginate the results (at each page I map to my custom object, so I can work with smaller objects, less memory needed). But the API response time for each page it's too slow (0.8 minutes per page if I set page_size to 10 000 it's 200 pages for 2M results so 0.8m * 200 = 2,6 Hours ).

Here is an example : 

In other words, searchstream uses too many memory and search it's too slow.

What could be a solution that don't take too long / don't explode the memory? 

I'm I missing another way to query? 

Thank you

[Google Ads API Forum Advisor]

Hi Boris,

Thank you for reaching out to us. However, I'm afraid that our team can only provide technical assistance related to Google Ads API specifically to error responses. Since your concern is more on integration, we would recommend checking our samples, then posting your concern directly to its github page. The developer specialist there should be able to provide an in-depth guidance to your concern.

Regards,

Teejay Wennie Google Ads API Team

ref:_00D1U1174p._5004Q2Wvt9W:ref

[Boris DETRY]

Thank you.

Actually the best way it's to use searchstream convert GoogleAdsRows immediately after receiving them in the buffer.  

[Google Ads API Forum Advisor]

Hi Boris,

You are correct that the searchStream would be a better option for multiple page reports, as also indicated in this guide related these methods' performance. I hope this helps.

Best regards,

Peter Laurence Google Ads API Team

ref:_00D1U1174p._5004Q2Wvt9W:ref