RetryAfterSeconds=30, Still Not Working After 30.

[Charles Michael]

Hello, I have a standard account and I'm currently running a lot of TargetingIdeaSelector lookups with v201607 (just upgraded from v201601) and I'm hitting the RATE_EXCEEDED (Account) error where the RetryAfterSeconds is 30. I've waited several minutes and relaunched my script to keep hitting the same 30 second retry error right away. I have setup my code to halt when we get a rate_exceeded error so as to prevent an endless loop of retrying, but I can't seem to figure out why its still not going through. Is there a more efficient way to space out requests so as to not hit these rate_exceeded errors (other than putting a 30 second sleep between requests)? I'd prefer to halt my scripts if we hit these errors to ensure we don't get blocked for longer, but it would be great to come up with a better way to space out requests so we don't hit these errors at all. 

Thank you. 

[Charles Michael]

As a side note, its been over 15 minutes and I'm getting the same RATE_EXCEEDED with 30 second retry notice on the first request I send. 

[Vishal Vinayak (Adwords API Team)]

Hi Charles,

When you receive the RATE_EXCEEDED  error, what do you see as the rateName in the error message? Also, since the rateScope in your error message is Account, that might mean that no matter which developer token is used to make the request, all requests to this account would be counted when calculating your limits. Anyway, I would recommend reading our guide on rate limits for more details on the error. If you are hitting the limits too often, I would recommend upgrading your account to a Standard Access Account.

Regards,

Vishal Vinayak, AdWords API Team

[Charles Michael]

[RateExceededError <rateName=RATE_LIMIT, rateKey=null, rateScope=ACCOUNT, retryAfterSeconds=30>]

I have read the rate limits guide, but the issue is that the API is not lifting the block after 30 seconds despite the API telling me to retry after 30 seconds. 

I also already have a standard account.

PS: This issue seems to be particular hitting me during peak hours, between noon - 5pm EST is when I keep hitting these blocks and I can't seem to do any requests (the 30 second retry is never lifted). Late nights though, all my requests go through without issue. No idea what would be causing this. 

Thanks

[Vishal Vinayak (Adwords API Team)]

Hi Charles,

I can recommend a couple things:

1) Since the rateScope is ACCOUNT, could you confirm if your AdWords account is being accessed via another developer token or OAuth credentials (could be through another MCC that the AdWords account is linked to)?

2) If #1 is false, i.e. you are the only entity accessing your AdWords account in your application, I would recommend slowing down the number of requests that you send per second by following some of the strategies mentioned on this page. 

Also, the number of queries that you can send to the API per second varies dynamically based on the current load on the servers. Hence you might see this error only during peak hours and the application might seem to work normally otherwise.

Regards,

Vishal, AdWords API Team

[Charles Michael]

Hi, I am the only developer with access to the API, so scenario #2 is applicable. My main issue is that the retry after 30 seconds isn't actually valid, it seems to be much longer and I can't really code a throttle if the time being reported is inaccurate. The advice from the pages linked basically says to wait 30 seconds, but when I do, I get the same error right away. I don't want to trigger a longer ban, so I've instituted a halt on my script when there's a rate exceeded error. Based on the load volume you mentioned, that would make sense why I can run my script at night without problems, but I should still be able to access it during peak hours as well. Since the weekend is coming up, I'll give it a shot with a longer sleeper in place, but it would be great to actually get an accurate figure on how long to throttle the script by as the suggestion by google's api of waiting 30 seconds isn't accurate. 

[Thanet Knack Praneenararat (AdWords API Team)]

Hello Charles,

I'm investigating this with engineering, so could you please send us your client customer ID via Reply privately to author?

And if you happen to have SOAP logs containing retryAfterSeconds, could you please send that along to me too?

Also, do you still face this situation consistently for everyday's peak time?

Thanks.

Thanet, AdWords API Team

[Charles Michael]

Hi, my scripts ran through the weekend with just a couple failures (it worked after RetryAfterSeconds=30), but since this is the first business day back, let's see if it fails during business hours. If it consistently fails after 30 seconds, I'll send you a private message with the account info. 

Thank you.

[Charles Michael]

Hi again, for the most part, it seems that its rather stable now, but between 12pm - 5pm EST on most business days (not weekends), I was hitting the retry after 30 seconds where it was taking more than 30 seconds to reset. It's a little frustrating, but for 19 hours per day it works without issue at 30 second sleepers. I was assuming that if I space my requests every 30 seconds I wouldn't hit the rateexceeded issue, but it seems apparent that its actually longer than 30 seconds during business hours which is when I assume the API is getting hit the most frequently. 

On Sunday, September 11, 2016 at 10:32:56 PM UTC-4, Thanet Knack Praneenararat (AdWords API Team) wrote:

[Thanet Knack Praneenararat (AdWords API Team)]

Hi Charles,

Could you please provide the SOAP log by Reply privately to author?

Best,

Thanet, AdWords API Team

[Charles Michael]

Unfortunately it's over 5 gigs large as I never flushed it since the beginning (over a year worth of logs). I just flushed it and will re-run. Do you want the entire log or specific entries? If the latter, what criteria would like me to include? 

[Thanet Knack Praneenararat (AdWords API Team)]

Hello Charles,

Just the latest request that you sent and got the rate exceeded error would be fine.

You don't need to send all of them.

Thanks!

Best,

Thanet, AdWords API Team

[Charles Michael]

I've sent you the logs, thanks for your help. 

[vj]

I am also getting same Error while using this service...I am getting this error even after no activity since last 24 hours on api request...

[Thanet Knack Praneenararat (AdWords API Team)]

Hello Charles, 

I've reported this to engineering.

I'll get back to you once I have more updates.

Cheers,

Thanet, AdWords API Team

[Thanet Knack Praneenararat (AdWords API Team)]

Hello,

Could you please send your SOAP logs by Reply privately to author too?
Thanks in advance.

Best,

Thanet, AdWords API Team

[Charles Michael]

Hi, was there any update to this? Since originally posting this, I've narrowed the issue to only occurring between 12pm - 5pm EST, outside of this range there are no issues with taking more than 30 seconds between requests. 

[Thanet Knack Praneenararat (AdWords API Team)]

Hello Charles,

I'm still waiting for the final answer of this.

A solution for now is to use exponential backoff policy, for example, if you first pause 5 seconds before the first retry, you could pause 10 seconds after the second and 20 seconds after the third retry, instead of following the retryAfterSeconds value.

We are discussing about the use of retryAfterSeconds but don't reach the conclusion yet.

I'll update this thread when getting more information.

Best,

Thanet, AdWords API Team

[Charles Michael]

The only issue with this strategy though is you could potentially be pausing for excessively long periods of time. During peak hours I would be pausing upwards of 60 seconds, but there isn't an efficient way for the pause time to come back down, especially when its no longer peak hours. 

If I may make a suggestion, it would be very beneficial to include a PauseTime so that developers know what the minimum pause time would be. Currently we're pretty much just guessing and if we hit the RATE_EXCEEDED response, we have to wait an unknown amount of time before making the same request over (since 30 seconds isn't guaranteed to work). If there's an accurate response number in the RATE_EXCEEDED response, that's great, but I think it would even be better to include in successful responses the minimum pause time for that particular account so that we can programmatically pause that amount of time and simply never hit the RATE_EXCEEDED page. 

For now I've implemented a flat 40 second pause time which seems to be sufficient, although to run my current keyword database, it will take about 3-4 weeks. 

[Thanet Knack Praneenararat (AdWords API Team)]

Hello Charles,

Thanks for your valuable feedback.

I'll pass this to engineering team as well.

I'll keep this post updated when I have any more news.

Cheers,

Thanet, AdWords API Team

[Charles Michael]

Sorry for bumping this again, but the issue has gotten progressively worse. Starting about an hour ago (noon EST), with wait times of 40 seconds, I'm failing with the same retry after 30 second errors. Was there any feedback from the engineering team as its very difficult to code something when the response time (30 seconds) is not valid. 

Thank you.

[Thanet Knack Praneenararat (AdWords API Team)]

Hello Charles,

Have you considered to use the exponential backoff policy as I mentioned in the previous reply?

The exponential backoff policy shouldn't be difficult to implement, as illustrated in the AddCompleteCampaignUsingBatchJob example.

I'm confirming with engineering but we haven't reached the conclusion yet.

Best,

Thanet, AdWords API Team

[Charles Michael]

Hi,

I remember when you first suggested the pseudocode for this but I never implemented it because I couldn't come up with a way to come back down in value over time. As I recall, you would double the wait time on every Retry request until the requests go through. If I ran this during peak hours, I would get a sleep time of around 100 seconds, but this is a very excessive wait time when it's not peak hours. I suppose you could cut the values in half until a retry comes back up, but this isn't really a reliable method in my opinion. It's a temporary workaround but we'd have excessive wait times in my opinion so I opted to just use a higher constant value (originally 40, I've raised it to 50 to compensate further). 

Thanks

[Thanet Knack Praneenararat (AdWords API Team)]

Hello Charles,

Could you please clarify why 100 seconds are too excessive?

I would like to use this information for discussion with engineering team as well.

Currently, the exponential backoff policy is encouraged to be used in many places of AdWords API, so 100 seconds is what we can expect in many situations.

But if you have any use cases where that waiting time is too long, then I can pass that information to the engineering team.

Thanks in advance.

Best,

Thanet, AdWords API Team

[Charles Michael]

Hello,

During peak hours, 100 seconds would be ideal. However, at 2am (EST) when the load is much less, the wait time can be as little as 10 seconds. Having to wait 90 seconds extra per request is quite an excessive amount of time to wait between requests when 10 is sufficient in offhours. I'm currently processing a batch of 40,000 requests, which at 100 seconds per request would take about 47 days. Since opening this ticket, I've gotten a little familiar with what are the peak hours to the Google API, and it seems that 20 seconds during offpeak and 40 seconds during onpeak (although this has crept up to 50 now) is sufficient to run the requests uninterrupted. 

The only issue I have with the exponential backoff policy is that it doesn't account for peak and off-peak hours. In my example, it would have increased wait times to 100 seconds during onpeak hours but would never come down during offpeak hours. While it would run, the runtime would be 47 days whereas with my guesstimated wait times, I can accomplish the same results with less than half the runtime. Ideally, getting a next wait time on each request would be the most efficient way of cutting runtime while not making too many requests. 

I'm not sure if the above is an acceptable user case, but it does seem logical to cutting the runtime of large requests. 

Thank you.

[Thanet Knack Praneenararat (AdWords API Team)]

Hi Charles,

Thanks for more insight.

I've discussed with engineering team and unfortunately, due to technical issues, we can't provide the exact waiting time via retryAfterSeconds.

So, we suggest that you treat the retryAfterSeconds as the lower bound of your waiting time,

and apply a random multiplier, which is a float number ranging between 1 and 2 to that value.

The resulting value can be used as your waiting time.

By the way, if you submit 40,000 requests, I wonder if you use BatchJobService to do that?

With BatchJobService, most of the time, you probably don't need to worry about the waiting time.

Best,

Thanet, AdWords API Team

[Charles Michael]

Thanks for the update, I'm currently waiting between 40-50 seconds which is pretty much as you suggested, a multiplier between 1-2X of 30 seconds. 

I looked into BatchJobService, and while it seems promising being able to retry requests automatically, it doesn't seem to have support for TargetingIdeaSelector. Looks like I'll need to simply wait it out and let it run the full duration.

Thanks for your help though!

[Thanet Knack Praneenararat (AdWords API Team)]

Hello Charles,

Thanks so much for your valuable feedback!

I'll update the doc so people don't get misunderstood about the retryAfterSeconds value again.

Best,

Thanet, AdWords API Team