Fluentd to both Splunk and ElasticSearch

[anandar...@gmail.com]

Our Ops team is at a point where they do not want to give up Splunk until ELK stack is proven and stabilized. So we would like to collect logs to both Splunk and ElasticSearch.

Thoughts and feedback on how to do this in Fluentd

[Yasin Amadmia]

Hi Aanandar,

We have exactly same use-case. 

We do it by first tagging the events that needs to go to both ElasticSearch and Splunk as `elasticsearch,splunk` then set a custom filed `output` whose value is equal to tag. On the fluentd server we have match statements as below. We use `rewrite_tag_filter` plugin https://github.com/fluent/fluent-plugin-rewrite-tag-filter. The plugin acts upon attributes and NOT on Tags hence we have to store tags in some attribute like `output` (via record_transformer)

<match **.elasticsearch>

  to elasticsearch

</match>

<match **.splunk>

  to splunk

</match>

<match **.elasticsearch,splunk>

  type copy

  <store>

    @type rewrite_tag_filter

    rewriterule1 output elasticsearch elasticsearch

  </store>

  <store>

    @type rewrite_tag_filter

    rewriterule1 output splunk splunk

  </store>

</match>