new API Google Ads via API Adwords: too long cycle to turn raw results into a dataframe (python)

[Adwords VBI]

Hello!

I'm starting to get acquainted with the Google Ads API   -  migrating there the Adwords API scripts  and met such a huge problem for me:

a cycle that took minutes in Adwords takes HOURS  via the Ads API (( I hope I'm just doing something wrong...

The task is to get   links from all ads from all campaigns  and  I need  to get all these  parametrs: 

  account_name,

  campaign.id, 

  campaign.name, 

  campaign.status, 

  ad_group.id, 

  ad_group.name, 

  ad_group.status, 

  ad_group_ad.ad.id, 

  final_urls

In the Adwords I could get it with  the  AD_PERFORMANCE_REPORT 

The API Google Ads migration tool recommended me   ad_group_ad  report and below  is my request.

It works  well  when there are several ads on the account, but when there are hundreds of them (or thousands), the FOR ROW IN RESULTS  loop takes hours

Also tried  FOR BATCH IN RESPONSE - the same result

And it looks like the raw  unload  (with Results or Response cycle)  does not allow to rewrite the data faster. Am I wrong?  Is there any alternative?

//////////////////

ga_service = client.get_service("GoogleAdsService")

customer_service = client.get_service("CustomerService")

resource_name = customer_service.customer_path(customer_id)

customer = customer_service.get_customer(resource_name=resource_name)

ga_service = client.get_service("GoogleAdsService")

query = """

    SELECT

  customer.descriptive_name,

  campaign.id, 

  campaign.name, 

  campaign.status, 

  ad_group.id, 

  ad_group.name, 

  ad_group.status, 

  ad_group_ad.ad.id, 

  ad_group_ad.ad.final_urls

  

FROM ad_group_ad

WHERE segments.date DURING TODAY AND campaign.status = 'ENABLED' AND ad_group.status = 'ENABLED' """

search_request = client.get_type("SearchGoogleAdsRequest")

search_request.customer_id = customer_id

search_request.query = query

results = ga_service.search(request=search_request)

for row in results:

    

    campaign = row.campaign

    ad_group = row.ad_group

    ad_group_ad  = row.ad_group_ad

    df = pd.DataFrame({'account_name':[customer.descriptive_name], 'campaign.id': [campaign.id], 'campaign.name': [campaign.name],'ad_group.id':[ad_group.id], 'ad_group.name': [ad_group.name],'ad_group_ad.ad.id':[ad_group_ad.ad.id], 'ad_group_ad.ad.final_urls': [ad_group_ad.ad.final_urls]}) 

    print(              

                customer.descriptive_name,

                campaign.id,

                campaign.name,

                campaign.status,

                ad_group.id, 

                ad_group.name, 

                ad_group.status, 

                ad_group_ad.ad.id, 

                ad_group_ad.ad.final_urls

                

            )      

        

        

    

[Mat]

Hi there,

I would recommend to:

• omit the "segments.date DURING TODAY" part in the query since you don't request any metrics
• use GoogleAdsService.SearchStream
• omit the df = pd.DataFrame() part while debugging to check if that might be the bottleneck
• omit the print() part while checking how long the loop lasts, because "print" is quite slow

As a sidenote: if you need the names and not the indexes of enum values, you should eg. use "campaign.status.name" instead of "campaign.status" while iterating through the results. 

Regards

Mat

[Adwords VBI]

Hi, Mat!

Thank you  so much! It seems the main problem is the dataframe  within  the loop  -  without it  the loop runs fast.  Trying to figure out what to do with the df now))

Best,

Daria

пятница, 9 июля 2021 г. в 12:44:55 UTC+3, Mat:

[Google Ads API Forum Advisor]

Hi there,

Thank you for reaching out to Google Ads API.

If you are using the Python client library version 10.0.0 or higher this version contains some known performance bottlenecks that occur when accessing fields on a deserialized protobuf message, which can be especially problematic when processing extremely large responses.

There is a workaround. Our documentation of native versus wrapped protobuf message instances shows how the wrapped protobuf messages include a number of interface improvements, but are slower to retrieve values, whereas the native message instances don't suffer from this problem.

In your example below, if you make the below change, you should see a noticeable performance improvement:
 

for row in results:
    row = row._pb # This converts the row to its native instance.
    campaign = row.campaign
    ad_group = row.ad_group
    ad_group_ad  = row.ad_group_ad

Please let me know if that works for you, or if you have any further questions.

Regards,

Aryeh Baker Google Ads API Team

ref:_00D1U1174p._5004Q2JZchY:ref

[Adwords VBI]

Hi! 

The main problem for me  was in the dataframe. But the line row = row._pb  really helps to reduce the time  for the loop  for row in results - on large uploads. Thank you! 

понедельник, 12 июля 2021 г. в 22:49:51 UTC+3, adsapi:

[Google Ads API Forum Advisor]

Hi,

I work with Aryeh and let me continue the discussion for this concern. I am happy also to know that my colleague's suggestion helped you.

As for your issue with dataframe, it appears that this is client library specific issue. With this, I am afraid that our team would not be able to provide further comments here as it is outside of our scope already. With this, you may reach out to the author of the client library instead as they are more equipped to provide support here. You may reach out to them via this link.

Regards,

Ernie John Blanca Tacata Google Ads API Team

ref:_00D1U1174p._5004Q2JZchY:ref