docker: Error response from daemon: failed to initialize logging driver

[matthieu...@signality.com]

We followed the guidelines in https://cloud.google.com/community/tutorials/docker-gcplogs-driver to upload docker logs into StackDriver and it had worked like a charm for the last 3 months. Since recently though, the following error randomly pops up while trying to start a bunch of docker containers :

docker: Error response from daemon: failed to initialize logging driver: unable to connect or authenticate with Google Cloud Logging: rpc error: code = InvalidArgument desc = transport: Get http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token: net/http: timeout awaiting response headers.

The reproducibility is unpredictable, it happens for one or more docker containers simultaneously, and it impacts different docker containers for different executions. No service outage has been reported in https://status.cloud.google.com/, but it seems related to the Google Cloud Logging endpoint access.

How should we properly handle this exception ? Has anyone had similar experience using gcplogs ?

[Igor Peshansky]

We've had a recent report of a similar issue. Note that gcplogs is a tool provided by a third party (Docker), and, as such, not officially supported by Google. You would have to raise the issue with the Docker community, especially since it's intermittent. It's very possible that the driver is simply not retrying properly, but it's up to Docker folks to determine. If they track it down to an issue with the underlying client library, we'd take over.

The supported solution is to use the Stackdriver logging agent. An alternative that worked in the other reported case was to install the agent (https://cloud.google.com/logging/docs/agent/installation) and use the "fluentd" Docker log driver to report the logs (the default configuration should be sufficient). If you choose to go that route, please let us know if it works for you.

        Igor
-- sent from a mobile device, please excuse tyops and omissns

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To post to this group, send email to google-stackdr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/92552618-3db5-4db9-b2ff-63cd2bfdd543%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.