[Google Ads API] Page size behavior unpredictable in the new Ads API, making use of Ads API a gamble for different accounts and different times

[Robert Heise]

Hi Google Ads API team,

I stumbled into the same issue regarding a byte size page limitation in the Google Ads API as described in https://groups.google.com/d/msg/adwords-api/CXCz72ZIAbs/nLuCvDOEAQAJ. That forum entry was marked completed, though to my understanding the issue is a lot bigger bummer and has to be resolved in a solid way in the Google Ads API itself. I'd like to emphasize, that the change for the page size in the Ads API makes requests (and thus a production use of the Google Ads API) basically a gamble... Please read my considerations below why this is a major issue in the Google Ads API.

In the Adwords API the documented limit for results per page (i.e. "page size") is 10000, see https://developers.google.com/adwords/api/docs/appendix/limits#general . This behavior was predictable, same queries for different accounts and issued at different times work consistently.

In the Google Ads API Workshop in Munich in March 2019, the Google Ads team suggested to use the same limits for the Ads API as they were defined to the Adwords API. Now during the migration to the Ads API we were astonished to see that instead of a predictable row count, an actual byte size is used to limit page sizes. A byte size limitation feels awkward for the Ads API use, where the number of columns vary for GAQL queries and also the content for each requested column varies a lot for different account IDs and over time. So in the end for exactly the same GAQL query an account ID works fine, while another one fails due to hitting the 4MB page limit - although the Ads API PAGE_SIZE is still the same for both. Even worse, the fetch for the same account ID fails over time if some values (i.e. product type) become larger, thus making pages reach the 4MB limit.

This behavior in the Ads API is basically making a production use a gamble as the behavior is unpredictabl. One option is of course to change the page size in case the error occurs. Though this feels ridiculous, as the error might occur also during paging for any page inbetween - and paging is handled by the client libraries seamlessly, so actually we'd need to change the client library code... A different option is to use a very low page size which hopefully fits the same GAQL query for all accounts for all time - though this leads to even worse performance due to many additional roundtrips due to tiny pages. Obviously setting the page_size to the maximum possible value is the favored option. Though knowing the right page size in advance is impossible as it depends on the data content which can't be known in advance.

It would be great if the Ads API could apply a real page_size limitation based on number of rows, instead a byte size. Number of rows, as implemented in the Adwords API leads to a predictable behavior for same GAQL queries for all accounts for all time.

Regards,

Robert

[Google Ads API Forum Advisor Prod]

Hello Robert,

We will let our team know regarding the page size limitation concern. Someone from our team will get back soon.

Regards,
Sai Teja, Google Ads API Team

ref:_00D1U1174p._5001UAqPlE:ref

[Google Ads API Forum Advisor Prod]

Hello,

Unfortunately, the 4MB limit is based on the underlying gRPC protocol, so there's nothing we can do to increase that overall byte limit.

However, we are working on a feature that will allow downloading of search query results as a whole, either streaming or by some other mechanism, which should alleviate both the paging issue and the overhead of multiple round trips once it is ready.

I cannot speak for all client libraries, but I do have intimate knowledge of the Ruby library and I do know that paging is currently abstracted away to such a point that you actually cannot manually adjust page size between requests. We are aware of this issue as well and are planning some improvements that would allow more fine-grained control if desired. I will pass along this feedback as well to the other client library teams.

Regards,
Mike, Google Ads API Team

ref:_00D1U1174p._5001UAqPlE:ref

[Robert Heise]

Hello Mike,

thank you for the response.

At our company we're using both the ruby and the java client libraries for Google Ads API. In the Java client lib the page size can be specified. I wonder whether client libraries could handle the optimal page size themselves, since exposing a row-based page size (when the underlying gRPC uses bytes for page limits) doesn't really make sense with arbitrary sized rows depending on the account, GAQL query or date range. The streaming option sounds like a reasonable solution for this problem.

Do you have any rough date available when this improvement on the page handling will be available?

Thanks,

Robert

[Google Ads API Forum Advisor Prod]

Hello Robert,

In talking to some of my teammates, I have also found out that we are aware of this specific problem. Since "try to guess what page size will actually fall under the 4MB limit" is obviously not a reasonable guidance, we are looking into ways that we can additionally guarantee that, when using this paging mechanism, you actually get back the requested number of results regardless of size. We don't have any news to share in that effort but it is ongoing.

Unfortunately, we are not able to provide any estimates for when that or the alternative streaming mechanism might be released, but since we should hit feature parity with the AdWords API very soon, these usability concerns are likely to be the next major focus.

[Dong Chen]

Hi team,

Any update about this issue? Is there any way to exceed the 4MB limitation? I'm using Python library right now.

Thanks!

[David Wihl]

Yes, this has been addressed as of V3. You can use the GoogleAdsService:searchStream method. See the tutorial or one of several Python code examples.

-David, Google Ads API Team

[Dong Chen]

Thanks David!

However, we already tested the searchStream method and it returned "429 Resource Exhausted". I also found the doc with "By default gRPC has a maximum message size of 4 MB. " and searchStream is also built on gRPC, right?

I also tested the search method with page token and it returned "page token has expired" after running for 3 hours. The library I use is php. Since the library will handle the paging automatically, so I'm not sure if it's the library bug or the server issue. 

After all, our purpose is to query all the data completely, which seems impossible for now. Could you help give any suggestions?

--
--
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog:
https://googleadsdeveloper.blogspot.com/
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
 
You received this message because you are subscribed to the Google
Groups "AdWords API and Google Ads API Forum" group.
To post to this group, send email to adwor...@googlegroups.com
To unsubscribe from this group, send email to
adwords-api...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
---
You received this message because you are subscribed to a topic in the Google Groups "AdWords API and Google Ads API Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/adwords-api/WEOx3jFuGlI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to adwords-api...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/adwords-api/00f57b2b-f5a9-40d6-afdd-880ab5431210o%40googlegroups.com.

--

Dong(Simon) Chen |

Technical Solutions Consultant |

don...@google.com |

+1 415-736-1777

[Google Ads API Forum Advisor Prod]

Hi Dong,

Thank you for the update. Could you confirm with me if you're using the most current version of the PHP client library? And have taken a look and attempted the searchStream example shown here? SearchStream sends a single request and initiates a persistent connection with the Google Ads API regardless of the report size. Packets of data will start to download immediately while the entire remaining result is cached in a data buffer. Therefore, you should be able to load a continuous stream of results. I see that you've also provided the Request log. Could you also provide us with the response log as well?

Thank you,
Bryan, Google Ads API Team

ref:_00D1U1174p._5001UAqPlE:ref