GAM timing out on large task it used to have no issue with

[Seth Dimbert (Hillel Yeshiva)]

I run GAMADV locally on my M1 MacBook Air. I run the following command regularly in order to make sure all of my faculty receive group emails:

gam update group All_Facult...@domain.org add member ou_and_children /Faculty

It takes some time to cycle through the ~400 accounts, but it always works. Starting yesterday, the command has been dying on me; it simply stops somewhere in the middle and, even if I let it sit for an hour, things just... stop. This last time, I let it run overnight but it stopped at 385/404.

GAM is up to date. Any ideas?

-SD

[Danny Dillon - NOAA Affiliate]

Same here, started yesterday. Numerous routine reports...

The normal runtime for my daily token report is ~40m.

It was already double that this morning, and still only 5% through my list of accounts, so I killed it.

Danny

--
You received this message because you are subscribed to the Google Groups "GAM for Google Workspace" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-man...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/c6097b9f-80de-4298-8909-b6793168a462n%40googlegroups.com.

[Joe Barham]

A colleague and I are both experiencing the same issue here. We are using the GAM tool to download device information of about 170k ChromeBooks. Up until yesterday we were able to download the information in about 20 minutes but currently it is failing after about an hour. 

Does anyone know where the cached information is stored before it is pushed to the CSV? Also, does anyone know if there are any log files?

Thanks,

Joe

[Jay Lee]

Can someone try running with debug enabled? Are API responses just taking a long time?

Jay

To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/53ecd06a-4c5b-4f92-b4d5-25cf152f64b9n%40googlegroups.com.

[Justin H]

Started seeing it yesterday afternoon with Groups batch runs. This call seems particularly problematic, ccc.hosted.frontend.directory.v1.DirectoryMembers.Delete, but it could be more. Group syncs will process batches of say 15 or maybe 50 actions, then do nothing for multiple minutes before finally resuming. I think one large batch (about 2000 group changes) eventually timed out altogether.

[Joe Barham]

Ran it again, here is the error message I got: Anyone know what this means?

C:\Data>gam print cros timerangeorder descending fields serialnumber,asset,model,status,ou,osversion,recentusers,activetimeranges,ethernetMacAddress listlimit 1 > gsuite.csv
Getting all CrOS Devices, may take some time on a large Google Workspace Account...
Got 98000 CrOS Devices...WARNING: List ChromeOSdevices Invalid Input: pageToken retryWARNING: List ChromeOSdevices Invalid Input: pageToken retryCrOS Device Print Failed: Invalid Input: CrEBCIivhaLxmvUCEqUBCm73kCYcdP_____3_ef83duZy8iezMmdndLKzZ6b0svLz5vSnsnImtLInsiay5rGzcfOmZ3PTS8M8v8A_v_-9_3n_N3bmcvInszJnZ3Sys2em9LLy8-b0p7JyJrSyJ7Imsuaxs3Hzpmdz00vDPL__hDQ_QUhrf4jyMSZKyM5AAAAAIvj2W9IAVAAWgsJKtm323v_CW4QA2CFvZLEBHIGCNTv1o4GGgISAA

C:\Data>

[Seth Dimbert (Hillel Yeshiva)]

@jay,

I added config debug_level 1 to my command and it flew through the process, producing hundreds of screens of output. The actual process completed in normal time (so that's a win) but I don't think I have anything useful to share here.

Here's what I entered: gam config debug_level 1 update group All_Facult...@domain.org add member ou_and_children /Faculty

On Wednesday, January 5, 2022 at 11:05:37 AM UTC-5 Justin H wrote:

[Ross Scroggs]

Seth,

A side note, changing add to sync should speed up the process considerably and eliminate all of the duplicate member messages.

gam update group All_Facult...@domain.org sync member ou_and_children /Faculty

Ross

--

ross.s...@gmail.com

To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/9783b3f8-fd58-41bc-8b90-9a0a6de3cc49n%40googlegroups.com.

[Mitchell Hoffman]

Hello All,

Sounds like similar issue I'm seeing.

Command was:
gam config debug_level 1 print groups email members

Error message:

reply: 'HTTP/1.1 502 Bad Gateway\r\n'
header: Content-Type: text/html; charset=UTF-8
header: Referrer-Policy: no-referrer
header: Content-Length: 1613
header: Date: Wed, 05 Jan 2022 15:05:42 GMT
header: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

[Danny Dillon - NOAA Affiliate]

Intermittent and different runs hang on different accounts.

# /gamx/gam config debug_level 1 ou '/UMS' print tokens | tee tokens.out

all good and fast until hang after account 48/77
Took 5m29s for 49/77 to show...
then fast to the end

Getting Access Tokens for ums.n...@noaa.gov (48/77)
send: b'GET /admin/directory/v1/users/ums.nws.cr%40noaa.gov/tokens?fields=items%28clientId%2CdisplayText%2Canonymous%2CnativeApp%2CuserKey%2Cscopes%29&prettyPrint=true&alt=json HTTP/1.1\r\nHost: admin.googleapis.com\r\naccept: application/json\r\naccept-encoding: gzip, deflate\r\nuser-agent: GAMADV-XTD3 6.10.00 - https://github.com/taers232c/GAMADV-XTD3 / Ross Scroggs <ross.s...@gmail.com> / Python 3.9.7 final / Linux-3.10.0-1160.49.1.el7.x86_64-x86_64-with-glibc2.17 x86_64 / (gzip)\r\nx-goog-api-client: gdcl/2.30.0 gl-python/3.9.7\r\ncontent-length: 0\r\nauthorization: Bearer ya29.a0ARrdaM_BKWHP4nVok9mEajE6zSZD4GC4llL5mTYLgb8seOJ7vyf8qDdmXB7pG6gMaTRsUDorpRGgt-ybEKq27sg0Leuw6J0AofUWFZ2-9SISAfBuCjLnnjJ4uZQW9-es5krvrQUOLDgrzATOAYNU4NrkZOFikfo\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: ETag: "9zPsLeP9ycRovNIzMe3UcREkFqt8n_JNHD4izBpxyt4/vyGp6PvFo4RvsFtPoIWeCReyIC8"
header: Content-Type: application/json; charset=UTF-8
header: Vary: Origin
header: Vary: X-Origin
header: Vary: Referer
header: Content-Encoding: gzip
header: Date: Wed, 05 Jan 2022 17:30:32 GMT
header: Server: ESF
header: Cache-Control: private
header: X-XSS-Protection: 0
header: X-Frame-Options: SAMEORIGIN
header: X-Content-Type-Options: nosniff

header: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

header: Transfer-Encoding: chunked   ---- was stuck here for minutes
Getting Access Tokens for ums.n...@noaa.gov (49/77)
send: b'GET /admin/directory/v1/users/ums.nws.er%40noaa.gov/tokens?fields=items%28clientId%2CdisplayText%2Canonymous%2CnativeApp%2CuserKey%2Cscopes%29&prettyPrint=true&alt=json HTTP/1.1\r\nHost: admin.googleapis.com\r\naccept: application/json\r\naccept-encoding: gzip, deflate\r\nuser-agent: GAMADV-XTD3 6.10.00 - https://github.com/taers232c/GAMADV-XTD3 / Ross Scroggs <ross.s...@gmail.com> / Python 3.9.7 final / Linux-3.10.0-1160.49.1.el7.x86_64-x86_64-with-glibc2.17 x86_64 / (gzip)\r\nx-goog-api-client: gdcl/2.30.0 gl-python/3.9.7\r\ncontent-length: 0\r\nauthorization: Bearer ya29.a0ARrdaM_BKWHP4nVok9mEajE6zSZD4GC4llL5mTYLgb8seOJ7vyf8qDdmXB7pG6gMaTRsUDorpRGgt-ybEKq27sg0Leuw6J0AofUWFZ2-9SISAfBuCjLnnjJ4uZQW9-es5krvrQUOLDgrzATOAYNU4NrkZOFikfo\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: ETag: "9zPsLeP9ycRovNIzMe3UcREkFqt8n_JNHD4izBpxyt4/vyGp6PvFo4RvsFtPoIWeCReyIC8"
header: Content-Type: application/json; charset=UTF-8
header: Vary: Origin
header: Vary: X-Origin
header: Vary: Referer
header: Content-Encoding: gzip
header: Date: Wed, 05 Jan 2022 17:36:03 GMT
header: Server: ESF
header: Cache-Control: private
header: X-XSS-Protection: 0
header: X-Frame-Options: SAMEORIGIN
header: X-Content-Type-Options: nosniff

header: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

header: Transfer-Encoding: chunked
Getting Access Tokens for ums.nw...@noaa.gov (50/77)
send: b'GET /admin/directory/v1/users/ums.nws.hq%40noaa.gov/tokens?fields=items%28clientId%2CdisplayText%2Canonymous%2CnativeApp%2CuserKey%2Cscopes%29&prettyPrint=true&alt=json HTTP/1.1\r\nHost: admin.googleapis.com\r\naccept: application/json\r\naccept-encoding: gzip, deflate\r\nuser-agent: GAMADV-XTD3 6.10.00 - https://github.com/taers232c/GAMADV-XTD3 / Ross Scroggs <ross.s...@gmail.com> / Python 3.9.7 final / Linux-3.10.0-1160.49.1.el7.x86_64-x86_64-with-glibc2.17 x86_64 / (gzip)\r\nx-goog-api-client: gdcl/2.30.0 gl-python/3.9.7\r\ncontent-length: 0\r\nauthorization: Bearer ya29.a0ARrdaM_BKWHP4nVok9mEajE6zSZD4GC4llL5mTYLgb8seOJ7vyf8qDdmXB7pG6gMaTRsUDorpRGgt-ybEKq27sg0Leuw6J0AofUWFZ2-9SISAfBuCjLnnjJ4uZQW9-es5krvrQUOLDgrzATOAYNU4NrkZOFikfo\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: ETag: "9zPsLeP9ycRovNIzMe3UcREkFqt8n_JNHD4izBpxyt4/vyGp6PvFo4RvsFtPoIWeCReyIC8"
header: Content-Type: application/json; charset=UTF-8
header: Vary: Origin
header: Vary: X-Origin
header: Vary: Referer
header: Content-Encoding: gzip
header: Date: Wed, 05 Jan 2022 17:36:03 GMT
header: Server: ESF
header: Cache-Control: private
header: X-XSS-Protection: 0
header: X-Frame-Options: SAMEORIGIN
header: X-Content-Type-Options: nosniff

header: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

header: Transfer-Encoding: chunked

...

Thanks,

Danny

To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/b4ac230f-1585-4d63-9e73-49002c913b65n%40googlegroups.com.

[Joe Barham]

I'm getting something different:

send: b'GET /admin/directory/v1/customer/my_customer/devices/chromeos?pageToken=CrABCJj3u7GCm_UCEqQBCm73cXpzKP_____3_ef83dvNzZzHys7JytKZnMub0svLmc_Sx8iey9KdmcnOys2ZmZ7OmZ7PTS8M8v8A_v_-9_3n_N3bzc2cx8rOycrSmZzLm9LLy5nP0sfInsvSnZnJzsrNmZmezpmez00vDPL__hC4zQchrf4jyMSZKyM5AAAAANeMhY5IAVAAWgsJKtm323v_CW4QA2DP56w-cgYI5IvXjgYaAhIA&projection=FULL&fields=nextPageToken%2Cchromeosdevices%28recentUsers%2Ftype%2CannotatedAssetId%2CserialNumber%2CorgUnitPath%2CdeviceId%2CrecentUsers%2Femail%2CethernetMacAddress%2CosVersion%2CactiveTimeRanges%2FactiveTime%2Cmodel%2Cstatus%2CactiveTimeRanges%2Fdate%29&maxResults=200&prettyPrint=true&alt=json HTTP/1.1\r\nHost: admin.googleapis.com\r\naccept: application/json\r\naccept-encoding: gzip, deflate\r\nuser-agent: GAMADV-XTD3 6.04.21 - https://github.com/taers232c/GAMADV-XTD3 / Ross Scroggs <ross.s...@gmail.com> / Python 3.9.6 final / Windows-10-10.0.14393-SP0 AMD64 / (gzip)\r\nx-goog-api-client: gdcl/2.11.0 gl-python/3.9.6\r\ncontent-length: 0\r\nauthorization: Bearer ya29.A0ARrdaM9drHkq_zEozt2TjrkmU-1Kpm6wtgZxWpzq1crZPNtCaTw6R-ngU7WBZ_gZkPHFQdt0NyJbDClPbkCKpbYRepPv1n5dCB5YTO0TCAAP31Qb9PK7duoWE5BczIRUQhGy1qOzgI9tFuQYsC0khWEJ2rmmzd0\r\n\r\n'
reply: 'HTTP/1.1 400 Bad Request\r\n'

header: Vary: Origin
header: Vary: X-Origin
header: Vary: Referer

header: Content-Type: application/json; charset=UTF-8
header: Content-Encoding: gzip
header: Date: Wed, 05 Jan 2022 17:21:43 GMT

header: Server: ESF
header: Cache-Control: private
header: X-XSS-Protection: 0
header: X-Frame-Options: SAMEORIGIN
header: X-Content-Type-Options: nosniff
header: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
header: Transfer-Encoding: chunked

ERROR:  JSON: {'error': {'code': 400, 'message': 'Invalid Input: CrABCJj3u7GCm_UCEqQBCm73cXpzKP_____3_ef83dvNzZzHys7JytKZnMub0svLmc_Sx8iey9KdmcnOys2ZmZ7OmZ7PTS8M8v8A_v_-9_3n_N3bzc2cx8rOycrSmZzLm9LLy5nP0sfInsvSnZnJzsrNmZmezpmez00vDPL__hC4zQchrf4jyMSZKyM5AAAAANeMhY5IAVAAWgsJKtm323v_CW4QA2DP56w-cgYI5IvXjgYaAhIA', 'errors': [{'message': 'Invalid Input: CrABCJj3u7GCm_UCEqQBCm73cXpzKP_____3_ef83dvNzZzHys7JytKZnMub0svLmc_Sx8iey9KdmcnOys2ZmZ7OmZ7PTS8M8v8A_v_-9_3n_N3bzc2cx8rOycrSmZzLm9LLy5nP0sfInsvSnZnJzsrNmZmezpmez00vDPL__hC4zQchrf4jyMSZKyM5AAAAANeMhY5IAVAAWgsJKtm323v_CW4QA2DP56w-cgYI5IvXjgYaAhIA', 'domain': 'global', 'reason': 'invalid'}]}}

[Ross Scroggs]

Mitchell,

Was everything OK up until this point?

Did GAM terminate on th Bad Gateway?

Ross

--

ross.s...@gmail.com

To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/b4ac230f-1585-4d63-9e73-49002c913b65n%40googlegroups.com.

[Mitchell Hoffman]

Hi Ross,

Yes I believe so. Before adding debug, this was the output error:

Traceback (most recent call last):
  File "__init__.py", line 58568, in ProcessGAMCommand
  File "__init__.py", line 26131, in doPrintGroups
  File "__init__.py", line 389, in executeBatch
  File "googleapiclient/_helpers.py", line 131, in positional_wrapper
  File "googleapiclient/http.py", line 1565, in execute
  File "googleapiclient/http.py", line 1500, in _execute
googleapiclient.errors.HttpError: <HttpError 502 when requesting https://admin.googleapis.com/batch returned "Bad Gateway". Details: "<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 502 (Server Error)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/> id=logo aria-label=Google></span></a>
  <p><b>502.</b> <ins>That’s an error.</ins>
  <p>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.  <ins>That’s all we know.</ins>
">

Thank you,

Mitch

To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/F29FDF6F-B1D2-44E5-BD12-6311F2FA65A2%40gmail.com.

[Ross Scroggs]

Joe,

How long had this been running? The API returns data in chunks, each chunk has a nextPageToken field that GAM passes back to Google to get the next chunk.

Here Google is complaining that the nextPageToken they just gave us is invalid; so maybe don't give us an invalid nextPageToken Google.

The current version 6.13.05 has some retry logic fir this case.

Since you are on an older version do:

Install latest version.

gam update project

gam oauth create

gam user us...@domain.com check serviceaccount

Ross

--

ross.s...@gmail.com

To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/e670ba93-4d59-4472-af51-231d49d048fbn%40googlegroups.com.

[Joe Barham]

What was taking about 16 to 20 minutes to pull down 170k device records is now failing after an hour. 

As a test my colleague did a fresh install of the latest version of GAM on a laptop and ran GAM off site (to make sure nothing in our network is causing the issue).

He got the same errored results as I did. 

[Jay Lee]

PLEASE DON'T POST DEBUG OUTPUT WITHOUT REMOVING THE AUTHORIZATION BEARER HEADER

If you've done so, run:

gam oauth revoke

to revoke the access token and prevent others from accessing your domain.

Jay

Jay Lee

To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/8450421f-4b1c-4704-8151-b5fd250e97e6n%40googlegroups.com.

[Ross Scroggs]

Danny,

Your stall was 5 minutes, 31 seconds. I was listing group members and had a 5 minute. 30 seond stall.

Very interesting,

Ross

--

ross.s...@gmail.com

To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/CAPt271YJ2WP-JU8_EKukZRNXGgpXfwjZ256vPTQ%3Dg4EQ_8OX_A%40mail.gmail.com.

[Joe Barham]

I'm running GAM on a SQL Server VM, in order to utilize it in a SSIS Script. 

The VM's specs:

  Windows Server 2016

  6 procs - xeon  2.1ghz 

  24 gb RAM

Is this good enough? 

After watching Task Manager for some time, the average utilization is no more than 4% to 6% CPU & about 275 to 300 mb of ram

[Danny Dillon - NOAA Affiliate]

Yes, my observed stalls were all ~5.5m.  

Things ran pretty smoothly last night, looking at runtimes it appears that there were few if any occurrences.

Thanks,

Danny Dillon

IBSS Corporation

NOAA OCIO

SO/CIO/SDD

202-525-8008 ( voice mail )

To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/FABA4FF6-9FEC-45AF-A8A8-E9E29861279B%40gmail.com.