4.30 is too fast now for groups..

342 views
Skip to first unread message

Ronald Jarrell

unread,
Aug 31, 2017, 9:36:57 AM8/31/17
to GAM for G Suite
Begining of the semester, and I've got batch processes that are pushing "update member" actions that result in thousands of changes.  Blocks at a time are getting:

Add Failed: 403: quotaExceeded - Request rate higher than configured. 

I thought I was pegging the 50K/user/100sec quota, but Google says I'm only doing about 10K.

Jim Frackoviak

unread,
Aug 31, 2017, 11:17:24 AM8/31/17
to GAM for G Suite
I just upgraded to 4.30 as well and I'm receiving the same error when trying to clear all the current members out of a group.  

THE INFORMATION CONTAINED IN THIS MESSAGE (E-MAIL AND ANY ATTACHMENTS) IS INTENDED ONLY FOR THE INDIVIDUAL AND CONFIDENTIAL USE OF THE DESIGNATED RECIPIENT(S).


If any reader of this message is not an intended recipient or any agent responsible for delivering it to an intended recipient, you are hereby notified that you have received this document in error, and that any review, dissemination, distribution, copying or other use of this message is prohibited. If you have received this message in error, please notify us immediately by reply e-mail message or by telephone and delete the original message from your e-mail system and/or computer database.


Ronald Jarrell

unread,
Sep 5, 2017, 10:00:01 AM9/5/17
to GAM for G Suite
I opened a support case, 13567914 about this.  Particularly I was asking about a post on stackoverflow where they said that that error could be from a domain blacklist.  Their reply is basically that GAM needs to be changed :(.

 There's a hidden quota:

I am happy to inform that we have received a response from our product engineers regarding this matter.

It has been confirmed that you are not blacklisted. What happens is that you are hitting the Members Insert/Delete QPS quota that is enforced on top of general Admin SDK quota (this is what is seen in GCP Console).

The quotas for the requested operations are:
- Members.Insert: 15 qps (75 requests per 5 seconds). This gives maximum of 1,296,000 requests per day.
- Members.Delete: 5 qps (25 requests per 5 seconds). This gives maximum of 432,000 requests per day.

These numbers explain the behavior that is being faced. Unfortunately, we are not able to increase this particular quota. This has never been done for any of customers including customers reporting the issue in mentioned StackOverflow article. Our recommendation would be to stick to the mentioned limits and also to implement the exponential back-off mechanism to not get throttled for a long time.

As per what I understand you are using GAM and you are not using a solution you have developed. Since GAM is a third party tools, you will need to report this to the developer or always run the operation in debug mode so that the API is not slammed.


Ross, you did too damn a good job accelerating it.

Eric Dannewitz

unread,
Sep 5, 2017, 10:02:42 AM9/5/17
to google-ap...@googlegroups.com
This is what a diet of Diet Coke and chocolate give you ;-)

Sent from my iSomething
--
Eric Dannewitz 
Acalanes Union High School District
Technology Assistant
--
You received this message because you are subscribed to the Google Groups "GAM for G Suite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-man...@googlegroups.com.
To post to this group, send email to google-ap...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-apps-manager.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/d391e9b8-d65d-4006-8262-01e622a1d4f7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Ross Scroggs

unread,
Sep 5, 2017, 10:24:06 AM9/5/17
to google-ap...@googlegroups.com
Ronald,

In my advanced GAM (https://github.com/taers232c/GAMADV-X) I have limiting measures.


batch_size

        When processing items in batches, how many should be processed in each batch

        Default: 50

        Range: 1 - 1000

inter_batch_wait                                                                                                                                                                                                                                         

        When processing items in batches, how many seconds should GAM wait between batches                                                                                                                                                               

        Default: 0                                                                                                                                                                                                                                       

        Range: 0 - 60                                                                                                                                                                                                                                    


Ross

On Tue, Sep 5, 2017 at 7:02 AM, Eric Dannewitz <edann...@auhsdschools.org> wrote:
This is what a diet of Diet Coke and chocolate give you ;-)

Sent from my iSomething
--
Eric Dannewitz 
Acalanes Union High School District
Technology Assistant

On Sep 5, 2017, at 7:00 AM, Ronald Jarrell <jar...@vt.edu> wrote:

I opened a support case, 13567914 about this.  Particularly I was asking about a post on stackoverflow where they said that that error could be from a domain blacklist.  Their reply is basically that GAM needs to be changed :(.

 There's a hidden quota:

I am happy to inform that we have received a response from our product engineers regarding this matter.

It has been confirmed that you are not blacklisted. What happens is that you are hitting the Members Insert/Delete QPS quota that is enforced on top of general Admin SDK quota (this is what is seen in GCP Console).

The quotas for the requested operations are:
- Members.Insert: 15 qps (75 requests per 5 seconds). This gives maximum of 1,296,000 requests per day.
- Members.Delete: 5 qps (25 requests per 5 seconds). This gives maximum of 432,000 requests per day.

These numbers explain the behavior that is being faced. Unfortunately, we are not able to increase this particular quota. This has never been done for any of customers including customers reporting the issue in mentioned StackOverflow article. Our recommendation would be to stick to the mentioned limits and also to implement the exponential back-off mechanism to not get throttled for a long time.

As per what I understand you are using GAM and you are not using a solution you have developed. Since GAM is a third party tools, you will need to report this to the developer or always run the operation in debug mode so that the API is not slammed.


Ross, you did too damn a good job accelerating it.

--
You received this message because you are subscribed to the Google Groups "GAM for G Suite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-manager+unsub...@googlegroups.com.
To post to this group, send email to google-apps-manager@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "GAM for G Suite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-manager+unsub...@googlegroups.com.
To post to this group, send email to google-apps-manager@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/-7649919566829679438%40unknownmsgid.

For more options, visit https://groups.google.com/d/optout.



--

Ron Jarrell

unread,
Sep 7, 2017, 11:13:43 AM9/7/17
to Google Apps Manager
Well, that seems to work.  Didn't even have to change the default.  It does 25 in a lump at hyperspeed, then waits a few seconds, and after that it processes the rest of the work at 1 per second, and doesn't do any more batches.  So it's slow, which works in this case.


To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-manager+unsubscribe...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "GAM for G Suite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-manager+unsubscribe...@googlegroups.com.

To post to this group, send email to google-apps-manager@googlegroups.com.
Visit this group at https://groups.google.com/group/google-apps-manager.



--

--
You received this message because you are subscribed to a topic in the Google Groups "GAM for G Suite" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-apps-manager/ylBmGnRHi7U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-apps-manager+unsub...@googlegroups.com.

To post to this group, send email to google-apps-manager@googlegroups.com.
Visit this group at https://groups.google.com/group/google-apps-manager.

Ross Scroggs

unread,
Sep 7, 2017, 11:23:58 AM9/7/17
to google-ap...@googlegroups.com
Ron,

I believe that you can speed it up by not using the defaults. Gam handles the Google errors saying to slow down by doing what Google calls exponential backoff. The trick is to avoid getting those errors in the first place.
Do the following: gam config batch_size 25 inter_batch_wait 2 save
What you should see is 25 actions, 2 second pause, 25 actions, 2 second pause, ...

Test and let me know the results.

Ross

To unsubscribe from this group and all its topics, send an email to google-apps-manager+unsubscribe...@googlegroups.com.

To post to this group, send email to google-apps-manager@googlegroups.com.
Visit this group at https://groups.google.com/group/google-apps-manager.

--
You received this message because you are subscribed to the Google Groups "GAM for G Suite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-manager+unsub...@googlegroups.com.

To post to this group, send email to google-apps-manager@googlegroups.com.
Visit this group at https://groups.google.com/group/google-apps-manager.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/CAGWftA-prF_phUGZ5Yp56RZGXFzMG91OtUFjgGFr%2B6nSbpvE7Q%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--

Ron Jarrell

unread,
Sep 7, 2017, 11:36:19 AM9/7/17
to Google Apps Manager
Yea, that seems to be what it's doing for the adds, but when we got into the removes it started backing off again.  I think that's because the remove qps is much smaller than the add.  I'll have to play with the batch size I guess to handle both.







--

--
You received this message because you are subscribed to a topic in the Google Groups "GAM for G Suite" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-apps-manager/ylBmGnRHi7U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-apps-manager+unsub...@googlegroups.com.

To post to this group, send email to google-apps-manager@googlegroups.com.
Visit this group at https://groups.google.com/group/google-apps-manager.

Ron Jarrell

unread,
Sep 7, 2017, 11:46:47 AM9/7/17
to Google Apps Manager
20 seems to be the magic number.


To unsubscribe from this group and all its topics, send an email to google-apps-manager+unsubscribe...@googlegroups.com.

To post to this group, send email to google-apps-manager@googlegroups.com.
Visit this group at https://groups.google.com/group/google-apps-manager.
Reply all
Reply to author
Forward
0 new messages