Groups Provisioning throttling/limits?

116 views
Skip to first unread message

David Spangler

unread,
May 4, 2012, 11:30:14 AM5/4/12
to google-app...@googlegroups.com
Hello,

I have a situation where we are attempting to populate around 90,000 groups using 5 different threads against the Provisioning API.  When we started at 5pm last night, we had "expected" throughput of around 200-300 users per minute processed.  After 4 hours, the rates dropped more than in half.  Did we reach an API limit?  If so, is it possible to request a lift of this limit temporarily until all the Groups are provisioned?

Attached is a graph of what last night looked like.

Thanks for looking at this.
Users per 5 min 2012-05-04 09-15.jpg

David Spangler

unread,
May 4, 2012, 2:02:33 PM5/4/12
to google-app...@googlegroups.com
In addition, we are interested in whether or not adding another 5 threads will increase the overall rates, i.e. is the process for provisioning Groups using the same API actions as "regular" user provisioning, hitting the same services that might be affecting the limits?

In addition to the existing 5 threads, there are 5 threads that are dedicated to user provisioning and 1 for password sync.  Are these additive?  In other words, we would be going from 11 to 16 threads overall, would this help?

Claudio Cherubino

unread,
May 4, 2012, 2:06:43 PM5/4/12
to google-app...@googlegroups.com
Hi David,

All limits are per thread so adding more threads should definitely increase the throughput.
Just be prepared to slow down using exponential backoff in case you get a 503 response code.
For more details, check our Usage Limits page in the docs:


Claudio

--
You received this message because you are subscribed to the Google Groups "Google Apps Domain Information and Management APIs" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-apps-mgmt-apis/-/MSD9rGLQ6AcJ.

To post to this group, send email to google-app...@googlegroups.com.
To unsubscribe from this group, send email to google-apps-mgmt...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-apps-mgmt-apis?hl=en.

David Spangler

unread,
May 4, 2012, 2:19:06 PM5/4/12
to google-app...@googlegroups.com
There was a period where the 5 drivers were working at the 1 transaction per second (for 4 hours), then there was a drop-off in speed (by half).  Does this indicate that we reached some limit?  If we add more threads will it also work properly for some time until another limit is reached?


On Friday, May 4, 2012 1:06:43 PM UTC-5, Claudio Cherubino wrote:
Hi David,

All limits are per thread so adding more threads should definitely increase the throughput.
Just be prepared to slow down using exponential backoff in case you get a 503 response code.
For more details, check our Usage Limits page in the docs:


Claudio
In addition, we are interested in whether or not adding another 5 threads will increase the overall rates, i.e. is the process for provisioning Groups using the same API actions as "regular" user provisioning, hitting the same services that might be affecting the limits?

In addition to the existing 5 threads, there are 5 threads that are dedicated to user provisioning and 1 for password sync.  Are these additive?  In other words, we would be going from 11 to 16 threads overall, would this help?



On Friday, May 4, 2012 10:30:14 AM UTC-5, David Spangler wrote:
Hello,

I have a situation where we are attempting to populate around 90,000 groups using 5 different threads against the Provisioning API.  When we started at 5pm last night, we had "expected" throughput of around 200-300 users per minute processed.  After 4 hours, the rates dropped more than in half.  Did we reach an API limit?  If so, is it possible to request a lift of this limit temporarily until all the Groups are provisioned?

Attached is a graph of what last night looked like.

Thanks for looking at this.

--
You received this message because you are subscribed to the Google Groups "Google Apps Domain Information and Management APIs" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-apps-mgmt-apis/-/MSD9rGLQ6AcJ.

To post to this group, send email to google-apps-mgmt-apis@googlegroups.com.
To unsubscribe from this group, send email to google-apps-mgmt-apis+unsub...@googlegroups.com.

Claudio Cherubino

unread,
May 4, 2012, 2:21:31 PM5/4/12
to google-app...@googlegroups.com
I don't think we were specifically throttling your app, but instead it might have been that our servers were overloaded.
Did you get any specific error message? We'll send you 503s if it is your app that needs to slow down the requests.

Claudio

To view this discussion on the web visit https://groups.google.com/d/msg/google-apps-mgmt-apis/-/lYs7PleFp4EJ.

To post to this group, send email to google-app...@googlegroups.com.
To unsubscribe from this group, send email to google-apps-mgmt...@googlegroups.com.

David Spangler

unread,
May 4, 2012, 2:24:49 PM5/4/12
to google-app...@googlegroups.com
There were no errors reported in the driver.


On Friday, May 4, 2012 1:21:31 PM UTC-5, Claudio Cherubino wrote:
I don't think we were specifically throttling your app, but instead it might have been that our servers were overloaded.
Did you get any specific error message? We'll send you 503s if it is your app that needs to slow down the requests.

Claudio

On Fri, May 4, 2012 at 11:19 AM,  wrote:
There was a period where the 5 drivers were working at the 1 transaction per second (for 4 hours), then there was a drop-off in speed (by half).  Does this indicate that we reached some limit?  If we add more threads will it also work properly for some time until another limit is reached?


On Friday, May 4, 2012 1:06:43 PM UTC-5, Claudio Cherubino wrote:
Hi David,

All limits are per thread so adding more threads should definitely increase the throughput.
Just be prepared to slow down using exponential backoff in case you get a 503 response code.
For more details, check our Usage Limits page in the docs:


Claudio
In addition, we are interested in whether or not adding another 5 threads will increase the overall rates, i.e. is the process for provisioning Groups using the same API actions as "regular" user provisioning, hitting the same services that might be affecting the limits?

In addition to the existing 5 threads, there are 5 threads that are dedicated to user provisioning and 1 for password sync.  Are these additive?  In other words, we would be going from 11 to 16 threads overall, would this help?



On Friday, May 4, 2012 10:30:14 AM UTC-5, David Spangler wrote:
Hello,

I have a situation where we are attempting to populate around 90,000 groups using 5 different threads against the Provisioning API.  When we started at 5pm last night, we had "expected" throughput of around 200-300 users per minute processed.  After 4 hours, the rates dropped more than in half.  Did we reach an API limit?  If so, is it possible to request a lift of this limit temporarily until all the Groups are provisioned?

Attached is a graph of what last night looked like.

Thanks for looking at this.

--
You received this message because you are subscribed to the Google Groups "Google Apps Domain Information and Management APIs" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-apps-mgmt-apis/-/MSD9rGLQ6AcJ.

To post to this group, send email to google-apps-mgmt-apis@googlegroups.com.
To unsubscribe from this group, send email to google-apps-mgmt-apis+unsubscri...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-apps-mgmt-apis?hl=en.

David Spangler

unread,
May 4, 2012, 2:31:32 PM5/4/12
to google-app...@googlegroups.com
Also, does it matter if the threads are using different admin IDs? Should we be using different admin IDs for each thread?

Claudio Cherubino

unread,
May 4, 2012, 2:36:43 PM5/4/12
to google-app...@googlegroups.com
I'll need to double check, but in general Google Data API requests' limits are per user, so using different admins should allow to work around them.

Claudio

To view this discussion on the web visit https://groups.google.com/d/msg/google-apps-mgmt-apis/-/IU4gwh0GW_4J.

To post to this group, send email to google-app...@googlegroups.com.
To unsubscribe from this group, send email to google-apps-mgmt...@googlegroups.com.

Luke Tracy

unread,
May 17, 2012, 3:15:25 PM5/17/12
to google-app...@googlegroups.com
Hi Claudio, I'm working with David and lead up our provisioning team.  We did eventually finish our migration of the 84,000 groups, but the rate dropped off again on day 4  to 1 transaction per 20 seconds per thread.  We had to increase from 5 drivers/threads to 10 in order to finish in the nick of time.  A week later we attempted to use the provisioning API's to move 4500 users from one sub-org to another to support their move to Google mail and calendar and migration of their existing data.  The response times from the Google API servers were once again abissmal.  We were processing 1 transaction per 5 seconds per thread.  We see delays of 1.5-2 seconds in response times to simple API calls like retrieveCustomerId, and 4 or more seconds to updateOrganizationUser.  Some sanitized log file examples below to show the gaps in our timestamps as our process sits idle waiting for a response from the Google API servers:

[05/10/12 04:47:58.294]:GA1 ST:appsForYourDomain.retrieveCustomerId: sDomain = umich.edu
[05/10/12 04:48:00.452]:GA1 ST:appsForYourDomain.updateOrganizationUser: CustomerId = *********************


[05/10/12 04:48:00.453]:GA1 ST:appsForYourDomain.updateOrganizationUser: NewOrgUnitPath = All Services
[05/10/12 04:48:04.551]:GA1 ST:addHandler: attr-name  == 'GivenName'


How can we verify that this was due to overloaded servers or being caused by something else, perhaps something we can correct or compensate for locally?  If we do verify it was overloaded servers, for 5 days, what can be done to shore up Google's provisioning API infrastructure and is that being done?

We have other large migrations to complete before we are fully deployed and we need to find a way of predicting and protecting the throughput of our provisioning processes or our deployments will continue to be plagued with these issues and Google's image will suffer here at the University of Michigan.

Thank you for any help you can provide us.

Luke Tracy
Technical Manager
Identity and Access Management
Universtiy of Michigan
Claudio

Stafford Marquardt

unread,
May 17, 2012, 3:26:33 PM5/17/12
to google-app...@googlegroups.com
Hi Luke and David,

We'll be happy to take a deeper look at this for you - can you please file a support ticket with Enterprise Support so that we can discuss and investigate in more detail?

Thanks!
Stafford
Reply all
Reply to author
Forward
0 new messages