Include Source Address Tag

[Kyle Kniffin]

Is there anyway to include the IP address of the host that is sending Syslog messages to the server in the fields?

I have checked out the various ways to modify a record and include ${Hostname} but that simply includes the hostname of the machine that Fluentd is running on.

The hostname parameter is not included in the syslog itself either to parse it out.

Appreciate the help.

[Kyle Kniffin]

As a follow-up I tried having the syslog message forwarded first to rsyslog and then having it forward to fluentd. It then includes the address of the sending device's IP in the syslog message.

However, it would be nice if fluentd already supported this and did not require a separate piece of software to be used.

Appreciate the help.

[Christian Hedegaard]

I use the syslog input plugin and then the record reformer plugin to do basically this.

 

We use chef to deploy everything so I do something like:

 

<source>

  type syslog

  port 5140

  bind 127.0.0.1

  tag syslog_original

</source>

 

<match syslog_original.**>

   type record_reformer

   renew_record false

   enable_ruby false

   tag syslog.${tag_suffix[1]}

   <record>

     fqdn <%= node[:internal_fqdn] %>

     vpc <%= node['dns']['vpc'] %>

     site <%= node['dns']['site'] %>

   </record>

</match>

 

So in our case, chef drops the config in place and then fills out the fqdn, vpc, site, etc.. It could also add the ipaddress and a number of other things.

 

If your case you could just do “ip www.xxx.yyy.zzz” and add it manually under the “record” part.

 

But record_reformer should do it for you!

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kyle Kniffin]

Thanks for the assistance. I think however that may not be what I am looking for.

I have a large number of Cisco devices that I would like to log directly to fluent.

When the messages come in, they do not have the IP address of the device that is sending the logs.

With other syslog systems, they basically take the IP address of the device that is sending the logs and add it to the syslog record.

I can accomplish this through syslog-ng and having it forward it to fluent or via rsyslog. But I would like to not have to use those extra pieces of software if possible.

Is there anyway that fluent can take the IP address of the device that is sending the syslog messages to it and append it to the record?

As i said I tried hostname, that does not work and simply adds the hostname of the server that fluent is running on.

The address also is not included in the syslog message itself.

Thanks again for the assistance.

[Masahiro Nakagawa]

Do you use in_syslog plugin?

It is now impossible because there is no requirement for now.

I'm not sure this feature is useful for Fluentd users

so I can't judge in_syslog plugin should support or not.

For now, extending the plugin seems easy way to resolve the issue.

https://github.com/fluent/fluentd/blob/ac389d56bcac3d48ec5a1e9f26e20bf8ab19c560/lib/fluent/plugin/in_syslog.rb#L154

module Fluent
  class SyslogWithSourceInput < Input
    Plugin.register_input('syslog_with_source', self)

  ...

    def receive_data(data, addr)
      @parser.call(data) { |time, record|
        unless time && record
          log.warn "invalid syslog message", :data => data
          return
        end

        pri = record.delete('pri')

        record['source'] = addr[3] # addr[3] returns hostname of sending device
        emit(pri, time, record)
      }

   rescue => e

      ...

   end

[Kyle Kniffin]

Yes I do use the syslog plugin.

I would think this feature would be standard as it is included in other paid enterprise solutions.

I am using this for Cisco devices which are a high pulsation of composites that use this.

Appreciate the help.

You received this message because you are subscribed to a topic in the Google Groups "Fluentd Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fluentd/A2awnRFQsRw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fluentd+u...@googlegroups.com.

[Kyle Kniffin]

Sorry autocorrect got me. I meant a large number of companies use Cisco equipment.

Thanks

[Masahiro Nakagawa]

Open an issue: https://github.com/fluent/fluentd/issues/390

[Kyle Kniffin]

Appreciate it.

Thanks,

Kyle Kniffin
ky...@kniffins.com

[Masahiro Nakagawa]

Could you check this PR?

https://github.com/fluent/fluentd/pull/391

[Kyle Kniffin]

Seems to be working good. Much appreciated.

What is the recommended way to handle syslog messages with non-standard timestamps? Is it best to just put format to none on the syslog input and then parse it out later with the parser plugin?

Examples:

20246: 031858: Jul 29 22:15:39.657 UTC","msg_type":"%LINK-5-CHANGED
Jul 29 18:21:38 server1 TAC_PLUS_ACCT: Tue Jul 29 18:21:32 2014\t1.1.1.1\ttest\ttty1\t2.2.2.2\tstart\ttask_id=1131\ttimezone=UTC\tservice=shell\tstart_time=1416652491"

[Kyle Kniffin]

The syslog software I currently use has the ability to timestamp incoming syslog messages with the receiving Syslog server time so that in case the sending device's time is inaccurate for some reason. Is there an option or easy way to accomplish this similar option as well with fluent?

Thanks again for the assistance and sorry for all the questions.

[Kyle Kniffin]

I see that fluent-plugin-record-reformer is a possible option for the datetime.

[Kiyoto Tamura]

Hi Kyle,

It totally depends on the variety of your syslog formats as well as how much filtering/processing you need to do. I will lay out some options below

1. @repeatedly recently wrote a parser plugin that accepts multiple formats: https://github.com/repeatedly/fluent-plugin-multi-format-parser

2. @sonots' record-reformer is pretty versatile

3. if you need to route messages differently based on the values of different fields, @y-ken's fluent-plugin-rewrite-tag-filter is a popular choice.

4. As you said, format none + fluent-plugin-parser is also an option (There is a tutorial about this here: http://www.fluentd.org/guides/recipes/parse-syslog)

Kiyoto

Check out Fluentd, the open source data collector for high-volume data streams