Tag processing

[Satnam Singh]

Here is a snippet of a config file that I use to collect Docker log files:

<source>

  type tail

  format json

  time_key time

  path /varlog/containers/*.log

  pos_file /varlog/es-containers.log.pos

  time_format %Y-%m-%dT%H:%M:%S

  tag kubernetes.*

  read_from_head true

</source>

<match kubernetes.**>

   type elasticsearch

   log_level info

   include_tag_key true

   host elasticsearch-logging.default

   port 9200

   logstash_format true

   flush_interval 5s

   # Never wait longer than 5 minutes between retries.

   max_retry_wait 300

   # Disable the limit on the number of retries (retry forever).

   disable_retry_limit

</match>

The log files get ingest with tag values like kubernetes.varlog.containers.monitoring-heapster-controller-x0qin_default-POD-fc67d68abe886a666043f325cac6c4dfd54bb450186c6e4172e14869dbda038c.log

This tag contains the name of the Kubernetes pod: monitoring-heapster-controller-x0qin_default, the name of the Kubernetes container: POD, and the corresponding Docker container: fc67d68abe886a666043f325cac6c4dfd54bb450186c6e4172e14869dbda038c.

I would like modify the tag so it is just the pod name and the container name e.g. monitoring-heapster-controller-x0qin_default-POD

How can I do this? Should I use something like this? https://github.com/fluent/fluent-plugin-rewrite-tag-filter

Or is there I way I can do it with some kind of regex direclty in the Fluentd config?

Thank you kindly.

Cheers,

Satnam

[Naotoshi Seo]

If I were you, I may use fluent-pugin-record-reformer like:

<source>

  type dummy

  tag kubernetes.varlog.containers.monitoring-heapster-controller-x0qin_default-POD-fc67d68abe886a666043f325cac6c4dfd54bb450186c6e4172e14869dbda038c.log

</source>

<match kubernetes.**>

  type record_reformer

  enable_ruby true

  tag ${tag_suffix[3].split('POD').first}POD

</match>

<match monitoring-heapster-controller-x0qin_default-POD>

  type stdout

</match>

Best,

Naotoshi a.k.a. sonots

[Satnam Singh]

Thank you kindly for your help which I have incorporated into https://github.com/GoogleCloudPlatform/kubernetes/pull/7539

Please let me know if you have any suggestions for improvement.

Thank you kindly.

Cheers,

Satnam