connection reset by peer

1,958 views
Skip to first unread message

Dan Schwartz

unread,
Jan 30, 2017, 3:04:24 PM1/30/17
to Google Apps Manager

I have a script which goes through all our users e-mail settings and reports on setting changes. Usually it runs fine, but occasionally I get an error message from gam that the connection was reset.

Got 13500 users...
Got 14000 users...
Got 14500 users...
Got 15000 users...
Got 15288 users...
done getting 12020 users.

ERROR: [Errno 104] Connection reset by peer

I think there is something timing out or the oauth token is expiring.  Is there a way to recover nicely from this error and continue, rather than starting all over.  Since the job takes hours to run, it would be nice to have it recover and continue rather than aborting.

I've upgraded to GAM 4.11, but still occasionally see the error.

Dan 




Ross Scroggs

unread,
Jan 30, 2017, 3:30:06 PM1/30/17
to google-ap...@googlegroups.com
Dan,

What is the command you're executing?

Ross

--
You received this message because you are subscribed to the Google Groups "GAM for G Suite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-manager+unsub...@googlegroups.com.
To post to this group, send email to google-apps-manager@googlegroups.com.
Visit this group at https://groups.google.com/group/google-apps-manager.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/CAOXHi9Oa%2B5ogAhwKn3HJg1rMSvfWdk957%3DqWJfVERjjtpR3BVA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.



--

Dan Schwartz

unread,
Jan 31, 2017, 7:14:20 AM1/31/17
to Google Apps Manager

​     ​
python gam.py all users print forward
​​ | sort > forwarding.txt



Jay Lee

unread,
Jan 31, 2017, 7:40:22 AM1/31/17
to Google Apps Manager

Connection reset by peer means the other side hung up without even saying goodbye. Often this is flakey Wi-Fi, lack of bandwidth or a firewall/proxy appliance that's misbehaving.

Try the command while tethered to your phone or from your home internet and see if you get better results.

Jay


On Tue, Jan 31, 2017, 7:14 AM Dan Schwartz <da...@lehigh.edu> wrote:

​     ​
python gam.py all users print forward
​​ | sort > forwarding.txt

On Mon, Jan 30, 2017 at 3:30 PM Ross Scroggs <ross.s...@gmail.com> wrote:
Dan,

What is the command you're executing?

Ross

On Mon, Jan 30, 2017 at 12:03 PM, Dan Schwartz <da...@lehigh.edu> wrote:

I have a script which goes through all our users e-mail settings and reports on setting changes. Usually it runs fine, but occasionally I get an error message from gam that the connection was reset.

Got 13500 users...
Got 14000 users...
Got 14500 users...
Got 15000 users...
Got 15288 users...
done getting 12020 users.

ERROR: [Errno 104] Connection reset by peer

I think there is something timing out or the oauth token is expiring.  Is there a way to recover nicely from this error and continue, rather than starting all over.  Since the job takes hours to run, it would be nice to have it recover and continue rather than aborting.

I've upgraded to GAM 4.11, but still occasionally see the error.

Dan 




--
You received this message because you are subscribed to the Google Groups "GAM for G Suite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-man...@googlegroups.com.
To post to this group, send email to google-ap...@googlegroups.com.



--

--
You received this message because you are subscribed to the Google Groups "GAM for G Suite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-man...@googlegroups.com.
To post to this group, send email to google-ap...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "GAM for G Suite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-man...@googlegroups.com.
To post to this group, send email to google-ap...@googlegroups.com.

Visit this group at https://groups.google.com/group/google-apps-manager.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/CAOXHi9NeZsrDXLf3OR%2BjXo1k0G29qJFZFp1Mziez0hNd6UMoMQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.
--

Jay

Dan Schwartz

unread,
Jan 31, 2017, 8:27:59 AM1/31/17
to Google Apps Manager
The server running the script is on our universities HA vmware cluster and it has lots of bandwidth and great connectivity.  Last time it fails was on 1/26 between 3 and 5am.  I'll try and watch to see if it fails again if there are any networking issues at the time.  

Thanks - Dan


--
Dan Schwartz | LTS - Systems Engineering  | Lehigh University | da...@lehigh.edu | (610) 758-5061

To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-manager+unsub...@googlegroups.com.
To post to this group, send email to google-apps-manager@googlegroups.com.



--

--
You received this message because you are subscribed to the Google Groups "GAM for G Suite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-manager+unsub...@googlegroups.com.
To post to this group, send email to google-apps-manager@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "GAM for G Suite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-manager+unsub...@googlegroups.com.
To post to this group, send email to google-apps-manager@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/CAOXHi9NeZsrDXLf3OR%2BjXo1k0G29qJFZFp1Mziez0hNd6UMoMQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
--

Jay

--
You received this message because you are subscribed to the Google Groups "GAM for G Suite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-manager+unsub...@googlegroups.com.
To post to this group, send email to google-apps-manager@googlegroups.com.

Visit this group at https://groups.google.com/group/google-apps-manager.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/CA%2BVVBp_esW7NfsqNPehvoesw6NQSHd0_BoHeh5YZEAasD9CZ1A%40mail.gmail.com.

Dan Schwartz

unread,
Jan 31, 2017, 9:36:57 AM1/31/17
to Google Apps Manager
Just in doing some testing, it failed again (and my 10-threads.txt file was empty) -

export GAM_THREADS=10; time python gam.py all users print forward |sort  > 10-threads.txt
Getting all users in G Suite account (may take some time on a large account)...
Got 15286 users...
done getting 12025 users.

ERROR: [Errno 104] Connection reset by peer

real 36m49.875s
user 2m2.200s
sys 0m7.100s

Looking at the console I see some errors (but not a lot) -


Inline image 1

--
Dan Schwartz | LTS - Systems Engineering  | Lehigh University | da...@lehigh.edu | (610) 758-5061

To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-manager+unsubscribe...@googlegroups.com.



--

--
You received this message because you are subscribed to the Google Groups "GAM for G Suite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-manager+unsubscribe...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "GAM for G Suite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-manager+unsubscribe...@googlegroups.com.

To post to this group, send email to google-apps-manager@googlegroups.com.
Visit this group at https://groups.google.com/group/google-apps-manager.
--

Jay

--
You received this message because you are subscribed to the Google Groups "GAM for G Suite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-manager+unsubscribe...@googlegroups.com.

To post to this group, send email to google-apps-manager@googlegroups.com.
Visit this group at https://groups.google.com/group/google-apps-manager.

+KimNilsson

unread,
Feb 6, 2017, 8:52:18 AM2/6/17
to GAM for G Suite
Just tried that... damn, that took a while! :-)

$ export GAM_THREADS=10; time gam all users print forward |sort  > 10-threads.txt
Getting all users in G Suite account (may take some time on a large account)...
Got 5208 users...
done getting 4738 users.

real 43m42.398s
user 5m30.280s
sys 0m10.035s


Jay Lee

unread,
Feb 6, 2017, 9:03:30 AM2/6/17
to GAM for G Suite
You can try bumping your threads higher assuming you have the RAM for it (most newer systems should). Also make sure you're up on GAM 4.11, there were some recent improvements to bulk commands that should produce faster results.

But yeah, GAM has to make 4,738 API calls for this command, one per user. That's never going to be terribly fast.

Jay

--
You received this message because you are subscribed to the Google Groups "GAM for G Suite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-man...@googlegroups.com.
To post to this group, send email to google-ap...@googlegroups.com.

Visit this group at https://groups.google.com/group/google-apps-manager.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/e74035ff-1af1-4b90-be99-c010b1b03cbb%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
--

Jay

Dan Schwartz

unread,
Feb 8, 2017, 7:32:21 AM2/8/17
to Google Apps Manager


My job ran fine for the past 8 days and then died again this morning.  This time with a traceback "dump".  Is there a way to make the code recover nicely on these longer jobs?  Perhaps with a renewal of the authentication token or something that backs off for a short time and then retries?  When it takes over an hour to run the command, it would be nice to recover and continue.

Got 14000 users...
Got 14500 users...
Got 15000 users...
Got 15293 users...
done getting 12031 users.
Traceback (most recent call last):
  File "/home/ludas/gam/gam.py", line 10313, in <module>
    sys.exit(ProcessGAMCommand(sys.argv))
  File "/home/ludas/gam/gam.py", line 10050, in ProcessGAMCommand
    printShowForward(users, True)
  File "/home/ludas/gam/gam.py", line 5356, in printShowForward
    userId=u'me')
  File "/home/ludas/gam/gam.py", line 586, in callGAPI
    return method(**parameters).execute()
  File "/home/ludas/GAM-4.11/oauth2client/util.py", line 137, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/home/ludas/GAM-4.11/googleapiclient/http.py", line 833, in execute
    method=str(self.method), body=self.body, headers=self.headers)
  File "/home/ludas/GAM-4.11/googleapiclient/http.py", line 160, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "/home/ludas/GAM-4.11/oauth2client/transport.py", line 159, in new_request
    credentials._refresh(orig_request_method)
  File "/home/ludas/GAM-4.11/oauth2client/client.py", line 744, in _refresh
    self._do_refresh_request(http)
  File "/home/ludas/GAM-4.11/oauth2client/client.py", line 775, in _do_refresh_request
    body=body, headers=headers)
  File "/home/ludas/GAM-4.11/oauth2client/transport.py", line 282, in request
    connection_type=connection_type)
  File "/home/ludas/GAM-4.11/httplib2/__init__.py", line 1654, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/home/ludas/GAM-4.11/httplib2/__init__.py", line 1394, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/home/ludas/GAM-4.11/httplib2/__init__.py", line 1350, in _conn_request
    response = conn.getresponse()
  File "/usr/lib/python2.7/httplib.py", line 1099, in getresponse
    raise ResponseNotReady()
httplib.ResponseNotReady

--
Dan Schwartz | LTS - Systems Engineering  | Lehigh University | da...@lehigh.edu | (610) 758-5061

On Mon, Feb 6, 2017 at 9:03 AM, Jay Lee <jay...@gmail.com> wrote:
You can try bumping your threads higher assuming you have the RAM for it (most newer systems should). Also make sure you're up on GAM 4.11, there were some recent improvements to bulk commands that should produce faster results.

But yeah, GAM has to make 4,738 API calls for this command, one per user. That's never going to be terribly fast.

Jay
On Mon, Feb 6, 2017 at 8:52 AM +KimNilsson <there.is.no.substitute@gmail.com> wrote:
Just tried that... damn, that took a while! :-)

$ export GAM_THREADS=10; time gam all users print forward |sort  > 10-threads.txt
Getting all users in G Suite account (may take some time on a large account)...
Got 5208 users...
done getting 4738 users.

real 43m42.398s
user 5m30.280s
sys 0m10.035s


--
You received this message because you are subscribed to the Google Groups "GAM for G Suite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-manager+unsub...@googlegroups.com.
To post to this group, send email to google-apps-manager@googlegroups.com.
--

Jay

--
You received this message because you are subscribed to the Google Groups "GAM for G Suite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-manager+unsub...@googlegroups.com.
To post to this group, send email to google-apps-manager@googlegroups.com.

Visit this group at https://groups.google.com/group/google-apps-manager.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/CA%2BVVBp8WRVfkpZ6Kd2D7%3Dbot3VO7fS3SUeMudygSTd3Dd5wV8Q%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages