FluentD as a kubernetes DaemonSet and Deployment

68 views

Skip to first unread message

Hemant Yadav

unread,

Nov 19, 2018, 3:42:47 AM11/19/18

to Fluentd Google Group

Problem

I have a kubernetes deployement for logging infrastructure. Where FluentD daemonset collects logs from the various node, forwards it to fluentD aggregator, which later forwards it to elastic search.

I ran into a bigger problem. In my logger infrastructure fluentD is running in 2 modes.

As a kubernetes DaemonSet 2) As a normal kubernetes Deployment

Now the problem is "path /var/log/fluentd-buffers/*" in DaemonSet is getting created inside the kubernetes node so even if the POD goes down the file bufffer path will be intact. However in case of Deployment type the file path is inside the container, so if pod goes down/restart the file buffer will be removed.

My queries are:

Is their any way we can mount the kubernetes VM node path and specify that in file buffer. In this case it will be like my case 1 (DaemonSet FluentD) and I am OK with it.
[BEST] Or Any way we can mount network storage like I have done for elasticsearch storage in yaml file.
volumeMounts:
- name: storage
mountPath: "/data"

Below is configuration, where the plugin is @forward

    <match kubernetes.**>
      @type forward
      send_timeout 60s
      recover_wait 10s
      hard_timeout 60s
      <server>
        name worker
        host {{.Values.fluentd.host}}.{{.Release.Namespace}}
        port {{.Values.fluentd.port}}
        weight 60
      </server>
      <buffer>
        @type file
        path /var/log/fluentd-buffers/fluentd.node.buffer // THIS FILE IS GETTING CREATED
      </buffer>
    </match>

FluentD aggregator config, where I am forwarding logs to elastic search. Here the plugin is elasticsearch

<match kubernetes.var.log.containers.es-master-*.*>
  @type elasticsearch
  logstash_format false
  index_name esmaster
  type_name logs
  host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
  port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
  <buffer>
    @type file
    path /var/log/fluentd-buffers/fluentd.aggregator.esmaster.buffer // THIS FILE IS NOT GETTING CREATED
  </buffer>
</match>

/fluentd/plugins # fluent-gem list

*** LOCAL GEMS ***

cool.io (1.5.3)
dig_rb (1.0.1)
elasticsearch (6.1.0)
elasticsearch-api (6.1.0)
elasticsearch-transport (6.1.0)
excon (0.62.0)
faraday (0.15.3)
fluent-plugin-elasticsearch (2.12.0)
fluentd (1.3.0, 0.12.43)
http_parser.rb (0.6.0)
json (2.1.0)
msgpack (1.2.4)
multi_json (1.13.1)
multipart-post (2.0.0)
oj (2.18.3)
openssl (default: 2.0.9)
psych (default: 2.2.2)
serverengine (2.1.0)
sigdump (0.2.4)
string-scrub (0.0.5)
strptime (0.2.3)
thread_safe (0.3.6)
tzinfo (1.2.5)
tzinfo-data (1.2018.5)
yajl-ruby (1.4.1)
/fluentd/plugins #
/fluentd/plugins #