Changing 'source' field

[Gregory Taylor]

I'm not sure if this is a graylog2 question or a fluentd question, so I'm going to assume the latter for the sake of getting this started.

I've got td-agent paired with graylog 1.1, and everything is working great. The thing I'm having trouble figuring out is how to change the 'source' of these messages. Here's what I've got now:

http://i.imgur.com/HKeqP2X.png

I'd like to stick a prefix on the 'source' value that is currently in there denoting which Google Compute project the host is in. We have overlapping hostnames between projects, so this is a super important distinction to make. It looks like 'source' defaults to the machine's hostname.

My config currently looks like this:

https://gist.github.com/gtaylor/a14c03c696b40c1799fe

Notice my current (failed) attempt at the top using record_transformer. My first attempt was to change 'source' directly there. This looks to have added a 'source' field to the payload, but graylog is still seeing the old source value (sans prefix):

http://i.imgur.com/byTCCKd.png

I then tried 'hostname', thinking it was a reasonable guess. I also saw mention of a host_param value, so that was attempted with no success as well. Right now I'm leaving the 'hostname' field in, but it's not optimal since graylog is grouping some of the volume metrics by the 'source' field.

Any ideas?

[Mr. Fiber]

From gelf plugin implementation, setting 'host' and '_source' to records is needed?

https://github.com/emsearcy/fluent-plugin-gelf/blob/51306d6d1c9dacd75f1c18c80a7e8a3d0834e0f1/lib/fluent/plugin/out_gelf.rb#L36

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Gregory Taylor]

I had tried with just 'host' over-written, and ended up with this: http://i.imgur.com/n2TXwhf.png. 

Here is what the config for this instance looked like at the time: https://gist.github.com/gtaylor/0fcede2079d869d53c2d

As you can see, I specifically set 'host' at the top <match> tag level. I had 'source' in there as well in the previous example because I was grasping at straws at that point. It seems like td-agent is over-writing what I provide.

--
You received this message because you are subscribed to a topic in the Google Groups "Fluentd Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fluentd/KkbgnXHoo38/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fluentd+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Greg Taylor

(864) 888-7964

http://gc-taylor.com

[Gregory Taylor]

Sorry, I had the wrong screenshot on my reply. Here is what I was seeing: http://i.imgur.com/jlSuOzT.png

Here is the config that got me to that point: https://gist.github.com/gtaylor/0fcede2079d869d53c2d

Notice how the message has the correct "host" value, but the 'source' field is still the old value. I am assuming that this host value is being sent in some other way that I have yet to discover how to override.

[Mr. Fiber]

How about using 'use_record_host true' paramer?

https://github.com/emsearcy/fluent-plugin-gelf/blob/51306d6d1c9dacd75f1c18c80a7e8a3d0834e0f1/lib/fluent/plugin/out_gelf.rb#L46

It seems overwriting default 'host' value.

https://github.com/Graylog2/gelf-rb/blob/61906a7c52d9e30c5dca3606da3daa5d46220a9b/lib/gelf/notifier.rb#L25

https://github.com/Graylog2/gelf-rb/blob/61906a7c52d9e30c5dca3606da3daa5d46220a9b/lib/gelf/notifier.rb#L163

[Gregory Taylor]

Excellent, that did it!

Thanks for the help. I now understand how easy this would have been to figure out myself, if only I had bothered to look at out_gelf.rb from the beginning!

[Jason Fowler]

I know this is an old thread but it appears the most relevant. 

I am testing a Fluentd to Gray log configuration. 

Most information is working however in Graylog I am not getting the originating source address.

Log Source -> Fluentd  ->  Graylog 

1.1.1.1            2.2.2.2        3.3.3.3

  

<store>

    type gelf

    host 3.3.3.3

    port 12201

    use_record_host true

    flush_interval 5s

</store>

My goal here is that Graylog would have the IP 1.1.1.1 as the source.

Fluentd is td-agent following this: http://docs.fluentd.org/v0.12/articles/install-by-deb

GELF configured following this: http://www.fluentd.org/guides/recipes/graylog2#sts=Graylog2 GELF input