How to implement exponential backoff?

201 views
Skip to first unread message

Zoeb Bharmal

unread,
Jul 14, 2016, 6:26:33 AM7/14/16
to Google Apps Manager
Hi,

I've had an open Google case with support for an error I've been encountering which is "ERROR: 403: Not Authorized to access this resource/api - forbidden". It's been identified that GAM has been correctly setup and all relevant scopes/oauth is correct. After Google investigated, they came back with the following:

"The support engineer found the related log entries by checking the requests made at the time the screenshot was made and could confirm that the 403 error generated has the code reason "quota exceeded." This behavior is expected and explains why you see this error without a clear pattern. As documented at https://developers.google.com/admin-sdk/directory/v1/limits "quota exceeded" is caused because "the limit of concurrent requests for a certain operation has been reached."

So they recommend implementing the exponential backoff but I don't see how this can be done using GAM? I am using the latest version at the moment. Any ideas or tips would be greatly appreciated.

Thanks,
Zoeb.

Jay Lee

unread,
Jul 14, 2016, 8:59:24 AM7/14/16
to Google Apps Manager

GAM already does exponential back off on temp errors but 403 not authorized is a permanent error, not temp. If you're not authorized to do something not, why would you be authorized to do it in a few seconds time? The error you are seeing is not an API quota error so support's response doesn't really fit.

Are you sure the user you authorized gam to run as has full admin rights? What command are you running? Does it ever work?

Jay


--
You received this message because you are subscribed to the Google Groups "Google Apps Manager" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-man...@googlegroups.com.
To post to this group, send email to google-ap...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-apps-manager.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/647ebd39-7b3e-4e19-b71d-4219c80ab922%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--

Jay

Ross Scroggs

unread,
Jul 14, 2016, 9:11:34 AM7/14/16
to google-ap...@googlegroups.com
Zoeb/Jay,

I've got someone seeing this same problem; they do: gam print users allfields > users.csv
They've got 200,000+ users and the command usually works.
Got 500 users: xxx - yyy
Got 1000 users: xxx - yyy
Got 1500 users: xxx - yyy
Got 2000 users: xxx - yyy


Sometimes, they get the "ERROR: 403: Not Authorized to access this resource/api - forbidden"
after 40,000/95,000/xx,000 users.

Clearly, they are authorized so that's not the problem.

Zoeb Bharmal

unread,
Jul 14, 2016, 9:48:43 AM7/14/16
to Google Apps Manager
Hi Jay,

Yes, I am the user who authorized it and it's my account which is being used to perform the action. I am a super admin of the domain. The command am I using is:

python gam.py print users creationtime lastlogintime ou suspended groups

Now it doesn't work whenever I try the command up until two weeks ago it's been working fine with no issues.

Thanks,
Zoeb.


On Thursday, 14 July 2016 14:59:24 UTC+2, Jay Lee wrote:

GAM already does exponential back off on temp errors but 403 not authorized is a permanent error, not temp. If you're not authorized to do something not, why would you be authorized to do it in a few seconds time? The error you are seeing is not an API quota error so support's response doesn't really fit.

Are you sure the user you authorized gam to run as has full admin rights? What command are you running? Does it ever work?

Jay


On Thu, Jul 14, 2016, 6:26 AM Zoeb Bharmal <zoeb.b...@whitestratus.com> wrote:
Hi,

I've had an open Google case with support for an error I've been encountering which is "ERROR: 403: Not Authorized to access this resource/api - forbidden". It's been identified that GAM has been correctly setup and all relevant scopes/oauth is correct. After Google investigated, they came back with the following:

"The support engineer found the related log entries by checking the requests made at the time the screenshot was made and could confirm that the 403 error generated has the code reason "quota exceeded." This behavior is expected and explains why you see this error without a clear pattern. As documented at https://developers.google.com/admin-sdk/directory/v1/limits "quota exceeded" is caused because "the limit of concurrent requests for a certain operation has been reached."

So they recommend implementing the exponential backoff but I don't see how this can be done using GAM? I am using the latest version at the moment. Any ideas or tips would be greatly appreciated.

Thanks,
Zoeb.

--
You received this message because you are subscribed to the Google Groups "Google Apps Manager" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-manager+unsub...@googlegroups.com.
--

Jay

Jay Lee

unread,
Jul 14, 2016, 9:55:11 AM7/14/16
to google-ap...@googlegroups.com
With:

gam print users groups

GAM has to run 1 API call per-user to retrieve all groups the user is a member of. This is both slow and burns through a lot of quota. It *should* work but it's far from efficient. My guess is after several hundred/thousand requests Google's servers want GAM to slow down but they somehow return the wrong error (not authorized).

You can try using something a bit more efficient like:

gam print group-members

which will output the members of each of your groups, basically giving you the same information but working in the other direction.

Jay

To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/FBE1C751-A3DD-492B-9A7F-8AACF7BF24D9%40gmail.com.

For more options, visit https://groups.google.com/d/optout.
--

Jay

Zoeb Bharmal

unread,
Jul 14, 2016, 10:03:04 AM7/14/16
to Google Apps Manager
Hi Jay,

This poses a problem for me to then be able to work out from which OU the users are members of what group.. Is this issue not fixable or something because it's functionality that worked previously but now doesn't?

Thanks,
Zoeb.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-manager+unsub...@googlegroups.com.
--

Jay


--
You received this message because you are subscribed to the Google Groups "Google Apps Manager" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-manager+unsub...@googlegroups.com.

To post to this group, send email to google-ap...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-apps-manager.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Apps Manager" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-manager+unsub...@googlegroups.com.
--

Jay

Ross Scroggs

unread,
Jul 14, 2016, 10:05:25 AM7/14/16
to google-ap...@googlegroups.com
Jay,

In my user's case, they say "gam print users allfields" but do not say "groups" so there is just the "one" API call, but, of course, it has to go back to Google for each 500 users.
I'm sure you're right, the servers get overwhelmed and return the incorrect error. My user's time frame is about the same, this worked reliably up until about two weeks ago.

Zoeb Bharmal

unread,
Jul 20, 2016, 9:17:01 AM7/20/16
to Google Apps Manager
Hi Jay,

So I went ahead with your suggestion and used the following command:

python gam.py print groups-members


After a little while I got the same error as before when using a different command as previously mentioned:


ERROR: 403: Not Authorized to access this resource/api - forbidden


There is definitely something that has changed somewhere which needs looking at. I have used GAM for many commands and cannot believe two different commands are a resulting in the same error. Ross is another case where he's experiencing the same but not using the same command as myself so this would point to GAM having an inherent issue, no?


Thanks,
Zoeb

Ross Scroggs

unread,
Jul 20, 2016, 9:26:00 AM7/20/16
to google-ap...@googlegroups.com
Zoeb,

It's not GAM. Jay's response had it right, he said: My guess is after several hundred/thousand requests Google's servers want GAM to slow down but they somehow return the wrong error (not authorized).

With: python gam.py print groups-members

One API call gets the list of groups, then there are N API calls to get the members for each group.

How many groups do you have?

--
You received this message because you are subscribed to the Google Groups "Google Apps Manager" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-man...@googlegroups.com.
To post to this group, send email to google-ap...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-apps-manager.

Zoeb Bharmal

unread,
Jul 20, 2016, 9:29:33 AM7/20/16
to Google Apps Manager
Hi Ross,

There roughly 16k groups in the domain. I understand if after the x hundreds/thousands of requests but surely there has to be a point where it will allow it again? This is now the 3rd straight week the command just doesn't work anymore..

Thanks,
Zoeb.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-manager+unsub...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages