Api quota exceeded error

[ads.rese...@gmail.com]

Hello, maybe here someone can help us out. We did a lot of refactoring regarding our sync jobs and then suddenly we received quota exceed error messages.

We followed the advice displayed here: https://developers.google.com/doubleclick-publishers/docs/troubleshooting#QuotaError.EXCEEDED_QUOTA

and limited our request to 8 per second and tried to setTimeout and sleep. Did not solve the issue. We then proceeded to wait because maybe we might just hit a daily cap. Now , same issue occurs. Any advice on how to proceed from here?

Thank you very much.

[Darijusch Behbahani]

we stopped even whole api calls since this morning and still hit the exceeded quote message

[Tesfamichael Hailegeorgis(DFP API Team)]

Hi,

To clarify, what is your network code? I am asking this as 8 queries per second is the quota limit for Premium accounts. If you have a Small Business account, the limit should be 2 queries per second. Also, please note that this quota limit applies per network. This means, if you have multiple users making a request at the same time, this can affect the quota. The same applies for the users in the UI as this quota limit is per network.

The best practice to resolve quota issues as suggested in the document is to safely wait and retry. There is a sample code here that you can use to implement a retry logic in your code.

Feel free to let us know how this goes.

Thanks,

Tesfamichael Hailegeorgis, DFP API Team

[ads.rese...@gmail.com]

Thanks for the replay.

Our Network code is: 83500759

[tesfam...@google.com]

Hi,

Thanks for providing the network code. 

Looks like you have a Premium account. As I mentioned in my previous response, the recommended approach to resolve this type of issues is to safely wait and retry:

http://googleadsdeveloper.blogspot.com/2014/04/troubleshooting-and-error-handling_11.html

Thanks,

Tesfamichael Hailegeorgis, DFP API Team

[Darijusch Behbahani]

hi,

we have such approaches in place, we disabled all scripts/calls to the api this morning and still running into this error.

[Tesfamichael Hailegeorgis(DFP API Team)]

Hi,

To clarify, do you have third parties or other applications making requests in your network? Can you double-check if you have some runaway script somewhere or some wrong logic in your code making more requests than you think?

Thanks,

Tesfamichael Hailegeorgis, DFP API Team

[Darijusch Behbahani]

Hi,

no we don't have third parties making requests to our network and as i said we disabled for debugging the whole calls through the dfp php library, when we try to do a test call, then we still get the quota exceeded, even if i come in the morning and all our sales people are not working with dfp yet.

[Darijusch Behbahani]

we just resetted our oauth secret without success, as a sidenote we get the quota exceeded on any call not on some

[Vincent Tsao]

Can you describe what your API integration is doing? What services do you use and how are you sync'ing and what entities are you sync'ing? What did you refactor?

For today, it looks like your network is making around 2 to 6 calls to CustomFieldService.getCustomFieldsByStatement() per second for a period of time (starting this morning from 8am PST for about 15 minutes). Were these calls expected? This is a very large amount of calls to getCustomFieldsByStatement() per second. Especially since it looks like you don't have that many custom targeting fields in your network.

Please double check your code for any logic errors or runaway scripts/processes.

Thanks,

Vincent Tsao, DFP API Team

[Darijusch Behbahani]

So mainly our script goes daily against different sources, so once a day we fetch lineitems, orders placement and adUnits that are modified, using a watermark to only fetch new modifications. And we fetch entities like GeoLocation (countries only), custom fields and targetingValueKeys and CustomTargetingKeys and CustomTargetingValues. But we stopped those scripts. So yesterday i just wanted so see something and run for about 15 minutes an endless loop which was doing  CustomFieldService.getCustomFieldsByStatement()  but every call returned QUOATA EXCEEDED. Normally at least 1 call should get through even if someone else would do those requests.

[Darijusch Behbahani]

So some more information on my post above, after we fetched the entities we are storing them in our database to hold the requests to dfp low. And then we do daily like 100 requests to fetch reports, but all those scripts/services have been disabled until the issue is resolved so we didn't fire anymore much stuff against the api except for my 2 tests runs yesterday and this morning against CustomFieldService.getCustomFieldsByStatement(). And i only did this to prove the theory that there must be some kind of bug with our quota, since if anyone else is using the api to exceed the quota, i should get a least one call through in the 15 minutes endless loop?

[Vincent Tsao]

Instead of an endless loop, can you try making 1 request per second - does every single request fail?

Thanks,

Vincent Tsao, DFP API Team

[Vincent Tsao]

Alright, sorry for it taking so long to get to the bottom of this. So you're correct, your API access has been blocked due to a high number of large requests to CustomTargetingService.updateCustomTargetingValues() around Jan 31, which caused issues for our servers. I believe the issue was that you may have been updating too many CT values in a single call. Can you describe the types of calls you're making to CT service so we can see if we can help optimize them and make them work within the scope of our API limits (attach your SOAP logs showing the calls)?

Also normally my team is notified when a network's API access is blocked so we are aware, but this slipped through the cracks this time.

Thanks,

Vincent Tsao, DFP API Team

[Darijusch Behbahani]

i did CustomTargetingService.updateCustomTargetingValues with 1000 entity updates maybe this was too much for your backend. i decreased it to 200, which is when i read it correctly the suggested size. What we are doing is we have entities on our side were we create a slug which is the dfp value name and then update them on change, since we changed it that every customTargetingValue now has a display name we ended up in updating all entities. so our updates are already in batches but i guess the size was just too much. Could you please reenable our network for api access?

[Vincent Tsao]

Thanks for the explanation. We'll re-enable your API access.

Thanks,

Vincent

[Darijusch Behbahani]

hi,

how long does it usually take? api is still not enabled :(

Thanks,

Dari

[Vincent Tsao]

Should be re-enabled now. Let me know if it's still not working.

Thanks,

Vincent Tsao, DFP API Team

[Darijusch Behbahani]

Hey,

just wanted to let you know that its now working :)

Can you please check, if its enough for your endwith the requests? i changed it to 200 updates and doing retries * 10 secs on quota error for 5 times and then giving up for the current run

[Darijusch Behbahani]

Hey,

so i guess our api access got again disabled, i disabled for now all our calls, can you please reenable it, since we are in europe its a bad time for us :(
should i reduce more or put some sleeps between the CustomTargetingService.updateCustomTargetingValues ? to not hammer on it, if this is the issue.

Thanks,

Dari

[Vincent Tsao]

Yeah sorry about all this. I dug into this further, for a batch size of 200 updates, you'll have to do this every 30 seconds to be safe. I.e., you should wait 30 seconds before updating another batch of 200 values. Our service right now can't support updates more frequent than that. It is something we are aware of and working on. I also understand this isn't documented, it's also something we are looking to improve.

Once you've put in this wait time, I'll have our team re-enable your API access. Thanks for your patience on this.

Vincent Tsao, DFP API Team

[Darijusch Behbahani]

Np i added a sleep of 30secs after each batch of updates/ inserts/ deactivations. Can you reenable our API once again.

Thanks,
Dari

[Vincent Tsao]

It should be re-enabled in a bit.

Thanks,

Vincent Tsao, DFP API Team