google-clock-sync-manager.service failure on 1068.8.0 and GCE

[Charlie Moad]

Starting with 1068.8.0 we are seeing the google-clock-sync-manager.service fail about half of the time. This is causing a handful of issues with our software where time synchronization is expected. I opened an issue here (https://github.com/GoogleCloudPlatform/compute-image-packages/issues/304) but I'm not positive that is the right place.

-- Logs begin at Tue 2016-07-19 14:39:17 UTC, end at Wed 2016-07-20 16:00:45 UTC. --

Jul 19 14:39:27 localhost systemd[1]: Started Google Compute Engine Clock Sync Daemon.

Jul 19 14:39:27 k8s-node-backend01.c.geofeedia-qa1.internal google-clock-sync[1102]: INFO Starting GCE clock sync

Jul 19 15:39:27 k8s-node-backend01.c.geofeedia-qa1.internal python2.7[1102]: Traceback (most recent call last):

Jul 19 15:39:27 k8s-node-backend01.c.geofeedia-qa1.internal python2.7[1102]:   File "/usr/share/oem/google-compute-daemon/manage_clock_sync.py", line 85, in <module>

Jul 19 15:39:27 k8s-node-backend01.c.geofeedia-qa1.internal python2.7[1102]:     Main()

Jul 19 15:39:27 k8s-node-backend01.c.geofeedia-qa1.internal python2.7[1102]:   File "/usr/share/oem/google-compute-daemon/manage_clock_sync.py", line 81, in Main

Jul 19 15:39:27 k8s-node-backend01.c.geofeedia-qa1.internal python2.7[1102]:     lock_file.RunExclusively(lock_fname, HandleClockDriftToken(watcher, OnChange))

Jul 19 15:39:27 k8s-node-backend01.c.geofeedia-qa1.internal python2.7[1102]:   File "/usr/share/oem/google-compute-daemon/manage_clock_sync.py", line 51, in HandleClockDriftToken

Jul 19 15:39:27 k8s-node-backend01.c.geofeedia-qa1.internal python2.7[1102]:     Handler, initial_value='')

Jul 19 15:39:27 k8s-node-backend01.c.geofeedia-qa1.internal python2.7[1102]:   File "/usr/share/oem/google-compute-daemon/metadata_watcher.py", line 74, in WatchMetadataForever

Jul 19 15:39:27 k8s-node-backend01.c.geofeedia-qa1.internal python2.7[1102]:     response = self.urllib2.urlopen(req)

Jul 19 15:39:27 k8s-node-backend01.c.geofeedia-qa1.internal python2.7[1102]:   File "/usr/share/oem/python/lib64/python2.7/urllib2.py", line 127, in urlopen

Jul 19 15:39:27 k8s-node-backend01.c.geofeedia-qa1.internal python2.7[1102]:     return _opener.open(url, data, timeout)

Jul 19 15:39:27 k8s-node-backend01.c.geofeedia-qa1.internal python2.7[1102]:   File "/usr/share/oem/python/lib64/python2.7/urllib2.py", line 404, in open

Jul 19 15:39:27 k8s-node-backend01.c.geofeedia-qa1.internal python2.7[1102]:     response = self._open(req, data)

Jul 19 15:39:27 k8s-node-backend01.c.geofeedia-qa1.internal python2.7[1102]:   File "/usr/share/oem/python/lib64/python2.7/urllib2.py", line 422, in _open

Jul 19 15:39:27 k8s-node-backend01.c.geofeedia-qa1.internal python2.7[1102]:     '_open', req)

Jul 19 15:39:27 k8s-node-backend01.c.geofeedia-qa1.internal python2.7[1102]:   File "/usr/share/oem/python/lib64/python2.7/urllib2.py", line 382, in _call_chain

Jul 19 15:39:27 k8s-node-backend01.c.geofeedia-qa1.internal python2.7[1102]:     result = func(*args)

Jul 19 15:39:27 k8s-node-backend01.c.geofeedia-qa1.internal python2.7[1102]:   File "/usr/share/oem/python/lib64/python2.7/urllib2.py", line 1214, in http_open

Jul 19 15:39:27 k8s-node-backend01.c.geofeedia-qa1.internal python2.7[1102]:     return self.do_open(httplib.HTTPConnection, req)

Jul 19 15:39:27 k8s-node-backend01.c.geofeedia-qa1.internal python2.7[1102]:   File "/usr/share/oem/python/lib64/python2.7/urllib2.py", line 1187, in do_open

Jul 19 15:39:27 k8s-node-backend01.c.geofeedia-qa1.internal python2.7[1102]:     r = h.getresponse(buffering=True)

Jul 19 15:39:27 k8s-node-backend01.c.geofeedia-qa1.internal python2.7[1102]:   File "/usr/share/oem/python/lib64/python2.7/httplib.py", line 1045, in getresponse

Jul 19 15:39:27 k8s-node-backend01.c.geofeedia-qa1.internal python2.7[1102]:     response.begin()

Jul 19 15:39:27 k8s-node-backend01.c.geofeedia-qa1.internal python2.7[1102]:   File "/usr/share/oem/python/lib64/python2.7/httplib.py", line 409, in begin

Jul 19 15:39:27 k8s-node-backend01.c.geofeedia-qa1.internal python2.7[1102]:     version, status, reason = self._read_status()

Jul 19 15:39:27 k8s-node-backend01.c.geofeedia-qa1.internal python2.7[1102]:   File "/usr/share/oem/python/lib64/python2.7/httplib.py", line 365, in _read_status

Jul 19 15:39:27 k8s-node-backend01.c.geofeedia-qa1.internal python2.7[1102]:     line = self.fp.readline(_MAXLINE + 1)

Jul 19 15:39:27 k8s-node-backend01.c.geofeedia-qa1.internal python2.7[1102]:   File "/usr/share/oem/python/lib64/python2.7/socket.py", line 476, in readline

Jul 19 15:39:27 k8s-node-backend01.c.geofeedia-qa1.internal python2.7[1102]:     data = self._sock.recv(self._rbufsize)

Jul 19 15:39:27 k8s-node-backend01.c.geofeedia-qa1.internal python2.7[1102]: socket.error: [Errno 104] Connection reset by peer

Jul 19 15:39:27 k8s-node-backend01.c.geofeedia-qa1.internal systemd[1]: google-clock-sync-manager.service: Main process exited, code=exited, status=1/FAILURE

Jul 19 15:39:27 k8s-node-backend01.c.geofeedia-qa1.internal systemd[1]: google-clock-sync-manager.service: Unit entered failed state.

Jul 19 15:39:27 k8s-node-backend01.c.geofeedia-qa1.internal systemd[1]: google-clock-sync-manager.service: Failed with result 'exit-code'.

[Charlie Moad]

Created https://github.com/coreos/bugs/issues/1468 as recommended.