Getting a Error 403: Serving Limit Exceeded - servingLimitExceeded

[Josh]

Jay,

      Today we ran a command using GAM 3.03  using  "gam print users allfields > users.csv"  at the halfway point we had an error "Error 403: Serving Limit Exceeded - servingLimitExceeded"

any ideas?

[Scott Latimore]

Same here - trying to print all groups to find out which ones are admincreated in preparation for a move to GADS. We have 1,459 groups on our domain. The report stopped at 1291. 

All mail correspondence to and from the Chapel Hill- Carrboro City Schools is subject to the North Carolina Public Records Law, which may result in monitoring and disclosure to third parties, including law enforcement.

[Kirke Johnson]

We hit the same error this morning after adding more groups to a membership synchronization process, running GAM 3.0. We are dealing with groups containing a few hundred members up to 5400 and over 10,000 in the largest ones. This is happening at points where we are simply downloading the lists of member addresses in each group. This was working fine with the largest groups until yesterday's addition of a few smaller groups.

A search of Google documentation did not return anything on this variety of Error 403. Anyone figured out this one yet?

--
You received this message because you are subscribed to the Google Groups "Google Apps Manager" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-man...@googlegroups.com.
To post to this group, send email to google-ap...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-apps-manager.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/5180f5fe-c4a0-49f5-ab39-dc1670bc170e%40googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--

Kirke Johnson                               kjoh...@pcc.edu
Email Administrator, TSS , Sylvania Campus   
Portland Community College, Portland, OR, USA     (971) 722-4368

[Ken Cheney]

There is an API call limit that Google has.. that could be responsible for this message.  I don't know for sure though.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/CACO3tANtA2v%2BESg3aNhf6Ggu9jwX-EDRLMj%3DcpjabM8tmokf-g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Gam User Man]

I was seeing this error also, on GAM 3.0.2.  I updated to 3.0.6 and the problem seems to have been addressed, so try updating GAM if you're having this issue.

[Graham Peden]

I am getting this too - It driving me up the wall!

Some facts..

1. getting this on multiple commands but in particular gam print groups settings todrive
   
2. This has been going on for about 1 week
3. typically runs of about 100 requests before failing (one request per group out of about 1600 groups)
4. using GAM 3.06 
   
5. have re-created new client id's multiple times
   
6. checking the api logs shows max requests per second of around 0.65
   
7. I have another app, written by myself, under a different project name but running against the same domain that uses the Admin sdk and makes many hundereds of api calls and that is working correctly
8. used 506 of 100,000 request from my quota today
   

---

This email and any attachment to it are confidential. Unless you are the intended recipient, you may not use, copy or disclose either the message or any information contained in the message. If you are not the intended recipient, you should delete this email and notify the sender immediately. Clifton College & Clifton College Services Ltd, 32 College Road, Clifton, Bristol. BS8 3JH

---

[Matthew Raven]

Just started getting this yesterday. I have a few large jobs (gam called through Powershell) that cycle through all ~15000 users on our domain that work without issue, but most commands that pull info from a group give the Error 403.

I've tried using different keys from different projects, and it doesn't seem to make a difference.

[Alberto Martínez]

Hi.

We don't use GAM, but still I have seen the same error when retrieving group members from several (1000) groups. I don't think that this issue is related to GAM. It seems to me that Google has set up some congestion barrier or that barrier is being reached globally for the first time.

Sensible backoff times like the ones we or GAM use don't avoid this so there is little to do, other than accepting it.

[John Williams]

I started getting this error today as well    I've been able to update a few random groups individually but when doing them in bulk I get that 403 error   Serving limit exceeded   

Once I get that Even the individual updates via GAM throw the error

[GK Admin]

Getting an error:

"Error 403: Serving Limit Exceeded - serviceLimit"
When trying to run any group settings (who can post, who can view members, etc).

I've tried:

Redownloading GAM (latest)

Recreating appengine JSON credentials

Recreating appengine project all together

Reverting to older version of GAM

All paths still result in the aforementioned error.

-Thomas

www.gkservices.com

We’re social. Follow us. LinkedIn / Twitter / YouTube /Facebook

This e-mail and any attachments may contain confidential information. If you are not the intended recipient, please notify the sender immediately by return e-mail and destroy any copies. Any dissemination or use of this information by a person other than the intended recipient is unauthorized and may be illegal. 

[Ben P]

I'm getting the exact same thing without fail. I'm approx 100 groups via the Python script and shell script that ingests the names of the groups via a CSV file.

Sometimes I get a 403 for all 100+ groups, sometimes it's half of them, sometimes it's only a couple — while the GAM is tool is useful, it's creating more problems by being inconsistent in it's application of permissions.

Need help!

Thanks

[Andrew Moore]

i dont know but its driving me nuts. I was on 3.04 and started to always get it in the last couple of days, tried updating it...same issue.

i cant even run: python gam.py info group collegeannouncement > test.txt for a group with only 1800 member without getting the error....pulling my hair out

[Ed Hillis]

I began seeing this yesterday. Today, thinking I needed to repeat some part of the setup, I followed the instructions at https://code.google.com/p/google-apps-manager/wiki/CreatingClientSecretsFile, going as far as creating a new project. At the point of creating the project, I found out the Google App Engine Admin Console was not enabled for my account, which surprised me. However, after re-enabling it and completing the instructions, GAM is working for me again.

Ed

[Ed Hillis]

Never mind...it only worked for a few requests. If I go to the Developers Console, though, I see in Settings the option to "Enable billing to access the full set of services and increased usage limits." Hope these errors are not due to a change in the business model.

[Ed Hillis]

It is on again off again for me. If I run a command five times it works on two of the tries.

[Andrew Moore]

mine appears to be only when i export to a file.

if i run this: python gam.py info group collegeannouncement 

it works and displays no problems

but if I run this: python gam.py info group collegeannouncement > test.txt 

i get the error every time

[Andrew Moore]

I am now getting the error all the time, from looking through my api usage history i am no where near my daily quota. So i am thinking this might be a GAM issue. Can someone from GAM Please respond with an update or anything?

[Ben P]

Ditto to Andrew's comment. It's become pretty much unusable for even basic purposes with the constant 403 errors. I have Billing enabled in our Developers Console and I'm at less than 1% of the 100,000 daily requests for Groups Settings API and well below any per/second quota that is in place.

We need an update please :)

[Joshua K]

At the same time this started with GAM, I also started seeing it in my own .NET based API project when accessing the GroupsSettings API only. No issues so long as I avoid that API. I can pull up a list of groups using the Groups API, but cannot see the settings on any group using the GroupsSettings API. So I'm not convinced its a GAM issues, might be a problem with the API? I never ever get a successful request for group settings.

At first I suspected this was related to GADS because we just started syncing groups on the day this 403 issues started. I thought maybe GADS was using up my quota but I can't see any quota that's used more than 0% on any API projects or anywhere else!

[Andrew Moore]

i think you are onto something about the groupsettings api. I only get the error when invoking: gam.py info group [groupname] If I run any number of other gam queries i never get an error, just when i try to pull down the group info

[Joshua K]

Its suddenly working again! I've been updating groups for the past 30 minutes, no more 403 errors.

**I misspoke earlier, what I meant was the Groups scope of the Directory API works fine (there is no Groups API), and just the GroupsSettings API was having fits.

[Ben P]

Based on what I'm reading here https://developers.google.com/admin-sdk/groups-settings/limits

I think there are potentially a couple issues.

1) The requests may be just coming in too quickly and are resulting in the 403

2) The HTTP response of 403 contains information on why the request failed. What GAM should ultimately do is catch the exception and retry that request with the exponential backoff algorithm, which uses a small delay before retrying the failed call. GAM must have this!!

Data and API call integrity is a must!

On Thursday, December 12, 2013 8:22:26 AM UTC-8, Josh wrote:

[Joshua K]

I spoke too soon, getting 403 errors again.

I don't know if its really a GAM issue because I get the error with a single request, I get it with my own API project and I also get it using their online API "try it now" tool: https://developers.google.com/admin-sdk/groups-settings/v1/reference/groups/get#try-it

[Alberto Martínez]

I already said that this is not a GAM failure. Seems to me that it's entirely related with the groupssetings API endpoint.

I don't know why people here keeps posting about specific GAM commands and retries. What we (I include myself despite not using GAM) is some info about this issue from a source close to Google. Jay Lee, are you there?

[Jay Lee]

Good morning all (or whatever time it is on your piece of earth),

It seems the Groups Settings API is returning a new error I've not seen before. Because I was not aware of this error, GAM was quitting after seeing it instead of retrying the operation. It does seem though that after backing off and retrying (sometimes needing to retry many times), the API call will eventually succeed.

I've thus uploaded GAM 3.07 whose only change is to perform a retry when it gets the serviceLimit error on Group Settings API calls.

https://googledrive.com/host/0B0YvUuHHn3MnbFl6N0k1UXcwdVk/

please give it a shot and let me know if it works (by retrying the operation until it works). It won't be as fast since it can take a number of minutes until the error stops being returned but it should allow operations to complete.

Jay Lee

On Mon, May 5, 2014 at 5:07 AM, Alberto Martínez <alberto_...@deusto.es> wrote:

I already said that this is not a GAM failure. Seems to me that it's entirely related with the groupssetings API endpoint.

I don't know why people here keeps posting about specific GAM commands and retries. What we (I include myself despite not using GAM) is some info about this issue from a source close to Google. Jay Lee, are you there?

--

You received this message because you are subscribed to the Google Groups "Google Apps Manager" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-man...@googlegroups.com.
To post to this group, send email to google-ap...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-apps-manager.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/CAFoHHqMkAr2uARR0-sTF9-z2M7PC-3NOm8GniWhnzL_KjVLjJg%40mail.gmail.com.

[Olaf Gretenkort]

Hi

On my site it works now

thx Olaf
 

[Joshua K]

Thanks for the update. Unfortunately no luck here. The first group update I tried worked but every time after that I get the serving limit error:

Temp error serviceLimit. Backing off 16 seconds...attempt 5/10
Temp error serviceLimit. Backing off 32 seconds...attempt 6/10
Temp error serviceLimit. Backing off 60 seconds...attempt 7/10
Temp error serviceLimit. Backing off 60 seconds...attempt 8/10
Temp error serviceLimit. Backing off 60 seconds...attempt 9/10
Temp error serviceLimit. Backing off 60 seconds...attempt 10/10
Error 403: Serving Limit Exceeded - serviceLimit

[Jay Lee]

Joshua, did you create your own client_secrets.json file? What does:

gam oauth info

Show?

--
Jay Lee
Sent from my Moto X

--
You received this message because you are subscribed to the Google Groups "Google Apps Manager" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-man...@googlegroups.com.
To post to this group, send email to google-ap...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-apps-manager.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/25f134d0-51dc-40b1-9926-485cb462d1fa%40googlegroups.com.

[Olaf Gretenkort]

PS: I get now the same Problem that it works in a scrip not fully.
I get the same sa Joshua.

Temp error serviceLimit. Backing off 16 seconds...attempt 5/10
Temp error serviceLimit. Backing off 32 seconds...attempt 6/10
Temp error serviceLimit. Backing off 60 seconds...attempt 7/10
Temp error serviceLimit. Backing off 60 seconds...attempt 8/10
Temp error serviceLimit. Backing off 60 seconds...attempt 9/10

But at the Moment is has not stoped because i dont get the ..attempt 10/10.

[Joshua K]

I do have my own client id and secret. I wouldn't expect it to work. Like I said, I get the same thing running a groups settings get or update using the .NET API library or from the try-it groups settings page. Either I have a problem with my domain or its a problem with the API.

OAuth File: C:\Users\example\Desktop\gam\oauth2.txt
Client ID: 555555555555.apps.googleusercontent.com
Secret: bdexample4iDbQnIq08FDCb7
Scopes:
  https://www.googleapis.com/auth/userinfo.email
  https://www.googleapis.com/auth/admin.directory.group
  https://www.googleapis.com/auth/admin.directory.orgunit
  https://www.googleapis.com/auth/admin.directory.user
  https://www.googleapis.com/auth/admin.directory.device.chromeos
  https://www.googleapis.com/auth/admin.directory.device.mobile
  https://apps-apis.google.com/a/feeds/emailsettings/2.0/
  https://apps-apis.google.com/a/feeds/calendar/resource/
  https://apps-apis.google.com/a/feeds/compliance/audit/
  https://apps-apis.google.com/a/feeds/domain/
  https://www.googleapis.com/auth/apps.groups.settings
  https://www.googleapis.com/auth/calendar
  https://www.googleapis.com/auth/admin.reports.audit.readonly
  https://www.googleapis.com/auth/admin.reports.usage.readonly
  https://www.googleapis.com/auth/drive.file
  https://www.googleapis.com/auth/apps.licensing
  https://www.googleapis.com/auth/admin.directory.user.security
  https://www.googleapis.com/auth/admin.directory.notifications
  https://www.googleapis.com/auth/siteverification
  https://mail.google.com/
Google Apps Admin: abc...@example.com

On Monday, May 5, 2014 8:15:09 AM UTC-7, Jay Lee wrote:

Joshua, did you create your own client_secrets.json file? What does:

gam oauth info

Show?

--
Jay Lee
Sent from my Moto X

On May 5, 2014 11:13 AM, "Joshua K" <jkil...@hemetusd.k12.ca.us> wrote:

Thanks for the update. Unfortunately no luck here. The first group update I tried worked but every time after that I get the serving limit error:

Temp error serviceLimit. Backing off 16 seconds...attempt 5/10
Temp error serviceLimit. Backing off 32 seconds...attempt 6/10
Temp error serviceLimit. Backing off 60 seconds...attempt 7/10
Temp error serviceLimit. Backing off 60 seconds...attempt 8/10
Temp error serviceLimit. Backing off 60 seconds...attempt 9/10
Temp error serviceLimit. Backing off 60 seconds...attempt 10/10
Error 403: Serving Limit Exceeded - serviceLimit

--
You received this message because you are subscribed to the Google Groups "Google Apps Manager" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-manager+unsub...@googlegroups.com.

[Ullfig, Roberto]

I’m seeing this problem now as well:

 

# gam update group cclab settings allow_external_members true

Error 403: Serving Limit Exceeded – serviceLimit

 

Any ideas? Seems to only affect group settings as we are provisioning new accounts just fine.

 

---

Roberto Ullfig – rul...@uic.edu

ACCC Research Programmer

To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-man...@googlegroups.com.

To post to this group, send email to google-ap...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-apps-manager.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/25f134d0-51dc-40b1-9926-485cb462d1fa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Google Apps Manager" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-man...@googlegroups.com.

To post to this group, send email to google-ap...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-apps-manager.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/aa5cd003-2780-43af-bcf5-f4972a7c77d8%40googlegroups.com.

[Cram]

Hi

today gam 3.0.7 is out.

it looks like the issue is fixed with this release:

Temp error serviceLimit. Backing off 16 seconds...attempt 5/10

Temp error serviceLimit. Backing off 32 seconds...attempt 6/10

Temp error serviceLimit. Backing off 60 seconds...

To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-manager+unsub...@googlegroups.com.

To post to this group, send email to google-ap...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-apps-manager.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/25f134d0-51dc-40b1-9926-485cb462d1fa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Apps Manager" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-manager+unsub...@googlegroups.com.

[Graham Peden]

3.07 fixes this problem for me and allows my scripts to eventually complete - Many thanks

It still leaves the question as to why this is happening. It is as though Google is throttling traffic to this api across all users.

I was going to suggest it might be specifically throttling the GAM application but I guess it probably doesn't know the exact nature of each client application. 

Perhaps this is turning out to be a particularly used api and google is struggling to meet demand.

--
You received this message because you are subscribed to a topic in the Google Groups "Google Apps Manager" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-apps-manager/l8uCGCXSdDA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-apps-man...@googlegroups.com.

To post to this group, send email to google-ap...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-apps-manager.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/f08c0f49-22ac-4981-8598-6804c4abf38e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

---

This email and any attachment to it are confidential. Unless you are the intended recipient, you may not use, copy or disclose either the message or any information contained in the message. If you are not the intended recipient, you should delete this email and notify the sender immediately. Clifton College & Clifton College Services Ltd, 32 College Road, Clifton, Bristol. BS8 3JH

---

[Greg Fellner]

It would appear that the problem still exists, the only difference is now it takes longer before it fails.

I'm getting the following almost constantly with any type of "info group" command.

Temp error serviceLimit. Backing off 16 seconds...attempt 5/10

Temp error serviceLimit. Backing off 32 seconds...attempt 6/10

Temp error serviceLimit. Backing off 60 seconds...attempt 7/10

Temp error serviceLimit. Backing off 60 seconds...attempt 8/10

Temp error serviceLimit. Backing off 60 seconds...attempt 9/10

Temp error serviceLimit. Backing off 60 seconds...attempt 10/10

Error 403: Serving Limit Exceeded - serviceLimit

On Monday, May 5, 2014 7:51:35 AM UTC-6, Jay Lee wrote:

Good morning all (or whatever time it is on your piece of earth),

It seems the Groups Settings API is returning a new error I've not seen before. Because I was not aware of this error, GAM was quitting after seeing it instead of retrying the operation. It does seem though that after backing off and retrying (sometimes needing to retry many times), the API call will eventually succeed.

I've thus uploaded GAM 3.07 whose only change is to perform a retry when it gets the serviceLimit error on Group Settings API calls.

https://googledrive.com/host/0B0YvUuHHn3MnbFl6N0k1UXcwdVk/

please give it a shot and let me know if it works (by retrying the operation until it works). It won't be as fast since it can take a number of minutes until the error stops being returned but it should allow operations to complete.

Jay Lee

On Mon, May 5, 2014 at 5:07 AM, Alberto Martínez <alberto_...@deusto.es> wrote:

I already said that this is not a GAM failure. Seems to me that it's entirely related with the groupssetings API endpoint.

I don't know why people here keeps posting about specific GAM commands and retries. What we (I include myself despite not using GAM) is some info about this issue from a source close to Google. Jay Lee, are you there?

--
You received this message because you are subscribed to the Google Groups "Google Apps Manager" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-manager+unsub...@googlegroups.com.

To post to this group, send email to google-ap...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-apps-manager.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/CAFoHHqMkAr2uARR0-sTF9-z2M7PC-3NOm8GniWhnzL_KjVLjJg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

NOTICE:  This confidential e-mail message is only for the intended recipients.  If you are not the intended recipient, be advised that disclosing, copying, distributing, or any other use of this message, is strictly prohibited.  In such case, please destroy this message and notify the sender.

[Andrew Moore]

Google is aware of the issue and is looking into it:

Hello Andrew,

 

Thank you for your message. I understand that you receive an error message Error 403: Serving Limit Exceeded.

 

So far we have a few more customers that are experiencing this issue, and we are working to provide a solution, in order for us to help you better we will need a full screenshot of the error message and also we will need the HTTP headers, you can follow these steps to collect the HTTP headers:

 

1. Click the menu icon in the top-right. This is the icon with three horizontal lines, colloquially known as the 'hot dogs'.

2. Choose 'Tools'->'Developer tools' to bring up the console.

3. Select 'Network'.

5. Click the small black circle in the top-right to begin recording. It will turn red when active.

6. Reproduce the issue. You'll begin to see data collecting in the console window.

7. Click the red circle to stop recording.

8. Right-click in the console window and select 'Save as HAR with Content' to download the .har output file.

9. Send us the HAR file.

 

I will leave this case open for now, please feel free to reply to this message if you have any questions or if you need further assistance.

 

Sincerely,

 

Winter

Enterprise Support

[Kirke Johnson]

We are noting a new problem, which started for us yesterday afternoon.
All attempts to list group membership with GAM, including the new
3.07, are failing with the following:

"Error 403: Access Not Configured. Please use Google Developers
Console to activate the API for your project. - accessNotConfigured"

>>> email to google-apps-man...@googlegroups.com.
>>> To post to this group, send email to google-ap...@googlegroups.com.
>>> Visit this group at http://groups.google.com/group/google-apps-manager.
>>> To view this discussion on the web visit

>>> https://groups.google.com/d/msgid/google-apps-manager/25f134d0-51dc-40b1-9926-485cb462d1fa%40googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Google Apps Manager" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an

> --
> You received this message because you are subscribed to the Google Groups
> "Google Apps Manager" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to google-apps-man...@googlegroups.com.
> To post to this group, send email to google-ap...@googlegroups.com.
> Visit this group at http://groups.google.com/group/google-apps-manager.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/google-apps-manager/CAAibfrmjaJcMq52Lox-UrAxT9tyX0PkM2obP5vrmqJZc7s46SA%40mail.gmail.com.

>
> For more options, visit https://groups.google.com/d/optout.



--

Kirke Johnson kjoh...@pcc.edu
Email Administrator, TSS , Sylvania Campus
Portland Community College, Portland, OR, USA (971) 722-4368