Fluentd-elasticsearch-kibana logging in GKE

[Michael Ukpong]

So guys.

I simply just want to do this simple thing: log all logs from pods within a GKE cluster into elastic search, so that i can analyse it in Kibana. we intend to build very advanced and specific, context based monitoring and management for our product, but first things first: all logs must be consolidated into some elasticsearch repo.  

This is very similar to what happens by default when you enable logging when creating a cluster from google developer console, the only difference is that the logs are consolidated in Google Cloud Logging, which is NOT what we want.

A lot of documentation every where within Kubernetes.io and GitHub, but honestly i've become more confused that clear.

• https://github.com/kubernetes/kubernetes/tree/master/examples/elasticsearch
• There above is documentation about setting up elastic search clusters, but it looks to me like its relevant if i wanted to setup elasticsearch as a service, not necessarily to collect log information from the underlying kubernetes system.
  
• https://github.com/kubernetes/kubernetes/blob/release-1.1/docs/getting-started-guides/logging-elasticsearch.md
  
• This above is closer to what i want... it says that enable elastic-search logging by setting the environment variable: KUBE_LOGGING_DESTINATION=elasticsearch But this assumes I downloaded Kubernetes and I am deploying it myself on GCE.. But thats not my use case, my use case is that I am using GKE.
  
• https://github.com/kubernetes/kubernetes/tree/master/cluster/addons  and   https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch
  
• This above is even closer to what i want, because it says that all i need is shipped natively into kubernetes as an add on... but how to enable it for GKE?

I'm tempted to just run `kubectl create -f ` on the 4 files i can see on https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch  But it looks like a hack to me... how would i be sure the fluent-d logging agents would be automatically deployed on each container/pod with that approach? how would i be sure things will work just as fine as it is with the Google Cloud Logging setup that comes out of the box? what of when my cluster scales to more that one node? would things still work fine?

Thanks, any insight would be greatly appreciated!

[Robert Bailey]

Enabling logging to elastic search on Google Container Engine is pretty simple:

1. Disable the GKE logging to Google Cloud Logging (so that you aren't running two fluentd pods per machine).

2. Run the fluentd-es pod on all of your nodes. Since DaemonSets don't yet exist in GKE (they are still an alpha feature), you can use host ports plus a high replica count to ensure that exactly one pod runs on each of your nodes.

3. Kubectl create -f the 4 files you found.

This is effectively what is done by the shell scripts when you set KUBE_LOGGING_DESTINATION to elasticsearch on GCE. 

--
You received this message because you are subscribed to the Google Groups "Containers at Google" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-contain...@googlegroups.com.
To post to this group, send email to google-c...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-containers.
For more options, visit https://groups.google.com/d/optout.

[Andreas Heim]

Dumb question: How do you disable Google Cloud Logging? Is it possible without creating a new cluster?

[Robert Bailey]

In the cloud console, when you are creating a new cluster uncheck the box "Turn on Cloud Logging". If you are using gcloud, pass `--no-enable-cloud-logging` when creating the cluster using `gcloud container clusters create`. 

Right now there isn't a way to change your logging preference after you have created a cluster. 

[Andreas Heim]

That's what i thought. Thanks for answering!

[Brett Coffin]

An other noobe question.
How do you set KUBE_LOGGING_DESTINATION to elasticsearch on GCE ?

Do you ssh into the running Instance group ? Where is located the kubernetes folder on the Google Container Engine intance? I can't even find the default-config file.

[Michael Ukpong]

On GCE it happens just before you run the script kube-up.sh to setup Kubernetes in GCE

See 

https://github.com/kubernetes/kubernetes/blob/release-1.1/docs/getting-started-guides/logging-elasticsearch.md

--
You received this message because you are subscribed to a topic in the Google Groups "Containers at Google" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-containers/Q1nvl8IAbqc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-contain...@googlegroups.com.

To post to this group, send email to google-c...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-containers.
For more options, visit https://groups.google.com/d/optout.

--

Michael Ukpong

C2G Consulting

259 Etim Inyang Cresent | Victoria Island | Lagos 

Phone: +234-1-4610792 | Cell: +234-803-3473455

Email: muk...@c2gconsulting.com | URL: http://www.c2gconsulting.com