can't get docker to log to stackdriver.

[tim....@csst.co.nz]

I am trying to get my docker images to log to stackdriver but nothing I do seems to work.

First I tried the obvious

GOOGLE_APPLICATION_CREDENTIALS=$HOME/secrets/development-area-baae5fc698ab.json \

docker \

--debug \

--config . \

--log-level debug \

run \

--name keycloak_server \

--env-file=.env_development \

--log-driver=gcplogs \

--log-opt gcp-project=development-area-218700 \

-p 8080:8080 \

jboss/keycloak

This gives the error message

keycloak_server

DEBU[0000] [hijack] End of stdout                       

docker: Error response from daemon: failed to initialize logging driver: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.

ERRO[0000] error waiting for container: context canceled 

I tried exporting that into my environment. I tried using the full path of the full path of the key file. Nothing I did worked.

Then I tried setting up another container with fluentd google cloud plugin.  That didn't work either giving the error message

2018-11-08 00:45:59 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"

keycloak_fluentd | /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- /var/lib/gems/2.3.0/gems/fluent-plugin-google-cloud-0.7.0/lib/fluent/plugin/out_google_cloud (LoadError)

then I installed fluentd into my own laptop and tried to use it without docker.

I set up a config file like this

<source>

@type tail

path /....../test.log

pos_file //...../test.log.pos

tag foobar.tsv

format syslog

</source>

<match **>

@type s3

aws_key_id XXXXXXX

aws_sec_key XXXXXXXXX

s3_bucket logs

s3_region ap-southeast-2

check_apikey_on_start true

</match>

<match **>

@type google_cloud

</match>

This ships the logs to amazon S3 but doesn't ship to stackdriver. There are no errors, it just doesn't do anything.  Note the credentials are exported into the environment.

How can I get my docker containers to log to stackdriver? 

Thanks.

[Igor Peshansky]

The gcplogs Docker driver is not maintained by the Stackdriver team and may be broken. You'd have to contact the maintainers if it doesn't work for you.

The recommended approach to ingest Docker logs into Stackdriver from a GCE or EC2 VM instance is to install the Stackdriver logging agent (https://cloud.google.com/logging/docs/agent/), which is based on fluentd, and set up the fluentd Docker log driver. The documentation pages I've referenced also contain the instructions for authorizing the agent property.

There's currently no supported way to ingest Docker logs via an agent running in a container, or from a non-cloud machine.

        Igor
-- sent from a mobile device, please excuse tyops and omissns

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To post to this group, send email to google-stackdr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/53b831df-7ca0-4843-813f-77ec5a8bfddc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Igor Peshansky]

To clarify, you should be just fine running your own service in a container on a Cloud VM. The agent, however, has to run on the VM itself, not in a container. You can ingest logs from your container via the agent.

        Igor

On Thu, Nov 8, 2018 at 3:30 PM Tim Uckun <tim....@csst.co.nz> wrote:

Thanks for the explanation. That's very disappointing though because I was intending to run my container our own servers. I guess I'll look into other options.

Cheers.

--

Tim Uckun

Director of data and analytics

+64275524911