I'm not sure about how to add a tag/field before being forwarded off

[Christian Hedegaard]

In my environment each host runs fluentd with a very minimal config. It uses the syslog plugin to listen on 5140, rsyslog forwards all syslog to localhost:5140, and then fluentd forwards “**” to our central log server.

 

What I would like to do is add a field to each syslog message BEFORE it gets forwarded to the central log server.

 

This plugin seems to do what I need it to do: https://github.com/sonots/fluent-plugin-record-reformer

 

However, I’m not sure what the correct way to get it involved is, before the message leaves the host for the central log server.

 

Here’s how I would normally use the reformer plugin:

<match syslog.**>

  type record_reformer

  renew_record false

  enable_ruby false

  output_tag es.${tag_suffix[1]}

  <record>

    fqdn host.fqdn.com

  </record>

</match>

 

So assuming there is an input tag of syslog.kern.crit, reformer would add  { “fqdn” : “host.fqdn.com” } to the message, and output the tag as es.kern.crit.

 

But here’s the rest of the hosts config:

<ROOT>

  <match **>

    type forward

    flush_interval 5s

    buffer_chunk_limit 5M

    heartbeat_type tcp

    <server>

      host log_server

      port 24224

    </server>

  </match>

  <source>

    type syslog

    port 5140

    bind 127.0.0.1

    tag syslog

  </source>

  <match debug.**>

   type stdout

  </match>

  <source>

    type forward

    port 24224

  </source>

</ROOT>

 

I’m worried that the initial <match **> would forward the syslog message BEFORE it gets a chance to run through the reformer plugin. I need each host to add its own fqdn record to its syslog messages BEFORE it forwards them off to the central log server.

 

Any ideas?

[Masahiro Nakagawa]

Hi,

Fluentd tries to match the tag in order from the top to bottom.

So if your "<match syslog.**>" is placed after "<match **>",

then all input records are emitted to "<match **>".

"<match **>" should be the end of configuration I think.

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Christian Hedegaard]

That’s PERFECT! Thanks!

 

One more question.

 

I’m taking advantage of individual config files in a conf.d directory and loading them with “include conf.d/*.conf”. Is there any way to be sure that one config loads before another? Can I number them like 00-something and 01-something and know for sure that 01 would get loaded after 00? Or can I alphabetize them or something? I think I remember testing this a couple months ago and it not working that way.

 

Having the ability to maintain final config order like that would be a HUGE boon to using the conf.d capabilities.

 

I’m using the official td-agent debian apt-repo and packages.

 

Thanks again!

[Kiyoto Tamura]

They are loaded in alphabetical order for now: https://github.com/fluent/fluentd/blob/master/lib/fluent/config.rb#L213

>Or can I alphabetize them or something? I think I remember testing this a couple months ago and it not working that way.

Do you have an example? If so, I will try to reproduce it.

Kiyoto

Check out Fluentd, the open source data collector for high-volume data streams

[Christian Hedegaard]

Thanks Kiyoto.

 

It’s been a while since I tested it, and it was a very brief test. I’ll keep in mind the alphabetical order though. Instead of 00, 01, etc I can just use AA, BB, J

 

I don’t have a need for this right now, but I might need it soon’ish as I’m adding more and more configs to our central log server.

[Kiyoto Tamura]

Hey Christian,

It's using Ruby's sort method against a list, so 00,01 should work equally as well as AA, BB.

If that doesn't seem to be the case, let us know =)

Kiyoto

[Masahiro Nakagawa]

I added match order section to config-file article.

http://docs.fluentd.org/articles/config-file#match-order