Severe rate limiting when running the python example

[Cameron Warren]

Hello,

I am running the get_keyword_ideas.py example script using Python 3.6. The script runs fine except for one issue - I can't seem to get the script to complete. I'm getting the following error after running the script only a couple of times - even when running for just 1 page of results. 

googleads.errors.GoogleAdsServerFault: [RateExceededError <rateName=RATE_LIMIT, rateKey=null, rateScope=ACCOUNT, retryAfterSeconds=30>]

My understanding from the rate sheet (https://developers.google.com/adwords/api/docs/ratesheet) is that I should be able to run 10,000 operations per day. I should be running only 1 get request (for 1 page as seen in my code below). 

Any assistance as to why this error is throwing would be greatly helpful. 

Code:

from googleads import adwords

import logging

import googleads

# logging.basicConfig(level=logging.INFO, format=googleads.util.LOGGER_FORMAT)

# logging.getLogger('googleads.soap').setLevel(logging.DEBUG)

# Optional AdGroup ID used to set a SearchAdGroupIdSearchParameter.

AD_GROUP_ID = 'INSERT_AD_GROUP_ID_HERE'

PAGE_SIZE = 1

data_table = []

def main(client, ad_group_id=None):

  # Initialize appropriate service.

  targeting_idea_service = client.GetService(

      'TargetingIdeaService', version='v201806')

  # Construct selector object and retrieve related keywords.

  selector = {

      'ideaType': 'KEYWORD',

      'requestType': 'IDEAS'

  }

  selector['requestedAttributeTypes'] = [

      'KEYWORD_TEXT', 'SEARCH_VOLUME', 'CATEGORY_PRODUCTS_AND_SERVICES']

  offset = 0

  selector['paging'] = {

      'startIndex': str(offset),

      'numberResults': str(PAGE_SIZE)

  }

  selector['searchParameters'] = [{

      'xsi_type': 'RelatedToQuerySearchParameter',

      'queries': ['space cruise']

  }]

  # Language setting (optional).

  selector['searchParameters'].append({

      # The ID can be found in the documentation:

      # https://developers.google.com/adwords/api/docs/appendix/languagecodes

      'xsi_type': 'LanguageSearchParameter',

      'languages': [{'id': '1000'}]

  })

  # Network search parameter (optional)

  selector['searchParameters'].append({

      'xsi_type': 'NetworkSearchParameter',

      'networkSetting': {

          'targetGoogleSearch': True,

          'targetSearchNetwork': False,

          'targetContentNetwork': False,

          'targetPartnerSearchNetwork': False

      }

  })

  # Use an existing ad group to generate ideas (optional)

  if ad_group_id is not None:

    selector['searchParameters'].append({

        'xsi_type': 'SeedAdGroupIdSearchParameter',

        'adGroupId': ad_group_id

    })

  more_pages = True

  while more_pages:

    page = targeting_idea_service.get(selector)

    # Display results.

    if 'entries' in page:

      for result in page['entries']:

        attributes = {}

        for attribute in result['data']:

          attributes[attribute['key']] = getattr(

              attribute['value'], 'value', '0')

        print ('Keyword with "%s" text and average monthly search volume '

               '"%s" was found with Products and Services categories: %s.'

               % (attributes['KEYWORD_TEXT'],

                  attributes['SEARCH_VOLUME'],

                  attributes['CATEGORY_PRODUCTS_AND_SERVICES']))

      print

    else:

      print('No related keywords were found.')

    offset += PAGE_SIZE

    selector['paging']['startIndex'] = str(offset)

    more_pages = offset < int(page['totalNumEntries'])

if __name__ == '__main__':

  # Initialize client object.

  adwords_client = adwords.AdWordsClient.LoadFromStorage()

  main(adwords_client, int(AD_GROUP_ID) if AD_GROUP_ID.isdigit() else None)

[Teja Makani]

Hello Cameron,

The RATE_EXCEEDED error usually occurs when too many requests were made to the API in a short period of time. The rateScope parameter represents that the error is due to account rate scope. You could refer this guide to know more about account rate scope. You could try to slow down the application a bit using RateExceededError.retryAfterSeconds parameter, please refer to the "Slow down" section of this guide for more information. Let me know if you have any further questions.

Regards,
Sai Teja, AdWords API Team.

[Cameron Warren]

Hi Teja,

Thanks for your explanation. I understand why it can occur - but in my case it doesn't make sense that it's occuring. I waited a full 24 hours and re-ran my script 1 time, and the script still errored out with the same error (see below the response). You'll note that it starts to provide results, but then after about 15 or so it throws the error. As you can see from my code I have the page size set to 1 - so this shouldn't be causing the Rate Exceeded error. Am I to believe I need to use the retryafterseconds paramter after just 15 results???

Keyword with "panama canal cruise" text and average monthly search volume "18100" was found with Products and Services categories: [10153, 10017, 10802, 10150].

Keyword with "cruises from melbourne" text and average monthly search volume "8100" was found with Products and Services categories: [10153, 10017, 10802, 10150].

Keyword with "cruises from florida" text and average monthly search volume "22200" was found with Products and Services categories: [10153, 10017, 10802, 10150].

Keyword with "florida cruises" text and average monthly search volume "6600" was found with Products and Services categories: [10153, 10017, 10150].

Keyword with "cruises" text and average monthly search volume "368000" was found with Products and Services categories: [10153, 10017, 10802, 10150].

Keyword with "emerald river cruises" text and average monthly search volume "2900" was found with Products and Services categories: [12096, 10153, 10017, 10151].

Keyword with "river cruise lines" text and average monthly search volume "2400" was found with Products and Services categories: [12096, 10153, 10017, 10151].

Keyword with "world cruise" text and average monthly search volume "18100" was found with Products and Services categories: [10153, 10017, 10802, 10150].

Keyword with "caribbean cruise" text and average monthly search volume "135000" was found with Products and Services categories: [10153, 10017, 10802, 10150].

Keyword with "emerald cruises" text and average monthly search volume "2900" was found with Products and Services categories: [12096, 10153, 10017].

Keyword with "cruise" text and average monthly search volume "368000" was found with Products and Services categories: [10153, 10017, 10802, 10150].

Keyword with "river cruises" text and average monthly search volume "49500" was found with Products and Services categories: [12096, 10153, 10017, 10151].

Keyword with "boat cruise" text and average monthly search volume "22200" was found with Products and Services categories: [12096, 10153, 10017, 10802].

Keyword with "royal caribbean cruise ships" text and average monthly search volume "22200" was found with Products and Services categories: [10153, 10017, 10802, 10150].

Keyword with "royal caribbean cruises" text and average monthly search volume "246000" was found with Products and Services categories: [10153, 10017, 10802, 10150].

Keyword with "caribbean cruise ships" text and average monthly search volume "2900" was found with Products and Services categories: [10153, 10017, 10802, 10150].

Keyword with "cruise ship" text and average monthly search volume "135000" was found with Products and Services categories: [10153, 10017, 10150].

Keyword with "royal cruise" text and average monthly search volume "14800" was found with Products and Services categories: [10153, 10017, 12093, 10150].

Keyword with "royal caribbean oasis" text and average monthly search volume "5400" was found with Products and Services categories: [10153, 10017, 10150].

Keyword with "port canaveral transportation" text and average monthly search volume "880" was found with Products and Services categories: [10785, 10017, 10146, 12085, 10789].

Keyword with "scenic crystal river cruise" text and average monthly search volume "20" was found with Products and Services categories: [12096, 10153, 10017, 10151].

Keyword with "orlando to port canaveral" text and average monthly search volume "2400" was found with Products and Services categories: [10785, 10017, 10146, 13842, 12085, 13575, 10782].

Error summary: {'faultMessage': '[RateExceededError <rateName=RATE_LIMIT, rateKey=null, rateScope=ACCOUNT, retryAfterSeconds=30>]', 'requestId': '000576ef47109e300abf62118906a649', 'serviceName': 'TargetingIdeaService', 'methodName': 'get', 'operations': '1', 'responseTime': '29'}

Traceback (most recent call last):

  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/googleads/common.py", line 1377, in MakeSoapRequest

    *packed_args, _soapheaders=soap_headers)['body']['rval']

  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/zeep/proxy.py", line 42, in __call__

    self._op_name, args, kwargs)

  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/zeep/wsdl/bindings/soap.py", line 132, in send

    return self.process_reply(client, operation_obj, response)

  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/zeep/wsdl/bindings/soap.py", line 194, in process_reply

    return self.process_error(doc, operation)

  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/zeep/wsdl/bindings/soap.py", line 299, in process_error

    detail=fault_node.find('detail'))

zeep.exceptions.Fault: [RateExceededError <rateName=RATE_LIMIT, rateKey=null, rateScope=ACCOUNT, retryAfterSeconds=30>]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "keyword_planner.py", line 94, in <module>

    main(adwords_client, int(AD_GROUP_ID) if AD_GROUP_ID.isdigit() else None)

  File "keyword_planner.py", line 68, in main

    page = targeting_idea_service.get(selector)

  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/googleads/common.py", line 1389, in MakeSoapRequest

    e.detail, errors=error_list, message=e.message)

googleads.errors.GoogleAdsServerFault: [RateExceededError <rateName=RATE_LIMIT, rateKey=null, rateScope=ACCOUNT, retryAfterSeconds=30>]

[tom.wa...@raisingit.com]

Hi Cameron,

I've raised exactly this issue with the API team before - sometimes I get rate limited before I get *any* results. The only workaround I've found is, as the team have suggested here, by adding error handling logic to wait 30 seconds before retrying and just keep on hitting the API until it gives a result. Given these errors are thrown so quickly on small test runs, it doesn't inspire confidence that the API will behave robustly when used in production. I think this isn't an issue for Google's support team, it needs to be escalated to their product team, to provide an API endpoint that's fit for purpose.

Cheers,

Tom

[Cameron Warren]

Tom,

Thanks for validating my findings! We ended up moving on to a vendor solution given the those limits. Glad to hear I’m not the only one.

Get Outlook for iOS

---

From: adwor...@googlegroups.com <adwor...@googlegroups.com> on behalf of tom.wa...@raisingit.com <tom.wa...@raisingit.com>
Sent: Friday, October 12, 2018 7:19:34 AM
To: AdWords API and Google Ads API Forum
Subject: Re: Severe rate limiting when running the python example

 

--
--
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog:
https://googleadsdeveloper.blogspot.com/
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
 
You received this message because you are subscribed to the Google
Groups "AdWords API and Google Ads API Forum" group.
To post to this group, send email to adwor...@googlegroups.com
To unsubscribe from this group, send email to
adwords-api...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
---
You received this message because you are subscribed to a topic in the Google Groups "AdWords API and Google Ads API Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/adwords-api/WaJKqqvEyp4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to adwords-api...@googlegroups.com.
Visit this group at https://groups.google.com/group/adwords-api.
To view this discussion on the web visit https://groups.google.com/d/msgid/adwords-api/9d5e1f7c-4281-4a06-8d35-a9ce4a49064e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Teja Makani]

Hello, 

The rate limits can fluctuate based on different variables, including server load. That said, it is not recommend to design with a fixed Queries Per Second (QPS) limit. Our suggestion is to have the application handle such errors by following the guidelines given here. If you feel that you should still not be getting that error, please share the SOAP logs and your CID so that we can take a closer look. You can reply privately to the author while sharing the details. 

Regards,
Sai Teja, AdWords API Team.

On Thursday, September 27, 2018 at 9:19:08 AM UTC-4, Cameron Warren wrote:

[Teja Makani]

Hello,

Thanks for your patience. There is an update on this issue, you can create a new test client account under the existing test manager account and the API requests should work with unapproved developer token. Please note that the requests to the existing test accounts is not yet fixed. Please give it a try and let me know if you are facing any issues. 

Regards,
Sai Teja, AdWords API Team.

On Thursday, September 27, 2018 at 9:19:08 AM UTC-4, Cameron Warren wrote: