Stackdriver monitoring of services contained in GKE

[Tim Child]

In the Stackdriver documentation, it refers to the Monitoring Agent for monitoring third-party applications such as Cassandra and Elastic Search.  But the monitoring agent doesn't seem to be supported on anything but a few VM types.

We are running things such as Cassandra and Elastic on GKE in containers and I would like to get better monitoring enabled. Should I try and run monitoring agent in the container?

[nau...@google.com]

Hello Tim,

As per documentation, "Stackdriver Monitoring is enabled by default when you create a new cluster using the gcloud command-line tool or the Google Cloud Platform Console."[1] Can you provide more detail on the non-supporting VM types as described in your message? 

In addition, it seems you are looking to get the logging within the containers. You can check out fluentd [2] to collect log files from within the containers. For information on collecting logs files using fluentd-sidecar-gcp, refer to this github link [3]. In brief, you can try using sidecar container to gather the required files within containers with Fluentd and sending them to the Google Cloud Logging service. You can refer to this documentation [4] for more information on side-car container. Here is another reference link which may be helpful. 

I would also suggest you to post questions related to this on google public kubernetes-users group [5] where experts on this can answer your inquiries. 

[1] https://cloud.google.com/container-engine/docs/monitoring#creating_a_cluster_with_monitoring
[2] https://cloud.google.com/solutions/customizing-stackdriver-logs-fluentd
[3] https://github.com/kubernetes/contrib/blob/b70447aa59ea14468f4cd349760e45b6a0a9b15d/logging/fluentd-sidecar-gcp/README.md
[4] https://kubernetes.io/docs/concepts/cluster-administration/logging/#basic-logging-in-kubernetes
[5] https://groups.google.com/forum/#!forum/kubernetes-users

[Tim Child]

Hi,

Thanks for your response, I am getting the logs correctly, and the GKE cluster is being monitored (for example I get the CPU usage by node) - that just worked. It's more the monitoring of the services in pods themselves.  For instance, for Cassandra I would like JVM metrics, and tasks as detailed here: https://cloud.google.com/monitoring/agent/plugins/cassandra

Fluentd is more for log shipping, I think what I would like is a collectd configuration that works with Stackdriver that I could use. I believe the monitoring agent is based on collectd.

Thanks again,

Tim.

[nau...@google.com]

Yes, collectd running as sidecar container should be able to get the metrics you need from the services in pods themselves. 

[Adam Sandor]

As I described on this thread using Stackdriver Agent to monitor containerised applications is not possible because the write_gcm plugin used by collectd to push metrics to SD is not able to label the metrics properly. All metrics pushed by SD Agent will show up in SD as GCE Instance metrics only labeled with the ID of the GCE Instance.

[Paolo Mainardi]

I am in the same situation, trying to understand how to use SD to monitor POD's services, like Mysql or Redis.

It there an official way to do that ?

[Tim Child]

Paolo, Did you get anywhere with this? 

[Olivier Cervello]

Same here, trying to set up JVM logs to flow correctly to Stackdriver.

In the docs we have a plugin for it: https://cloud.google.com/monitoring/agent/plugins/jvm so the customer thought this was possible for his use case.

Unfortunately, the Stackdriver monitoring / logging agents do not work on GKE with COS ... You can still use Ubuntu but that would mean that we recommend changing the operating system just to have it working.

I think atm the only solution is to deploy fluentd ourselves and change the configmap for it, like explained here: https://cloud.google.com/solutions/customizing-stackdriver-logs-fluentd.

Anyone to confirm this ?