Newbie question

[Tim Washburn]

Hello All,

I'm trying to setup the following for sending logs from my security event manger to my Hadoop cluster using Fluentd:

Q-Radar Seim ---> FluentD -----> Webhdfs on my hadoop cluster

Q-Radar is configured to send syslog events to Fluentd on port 5000, tcp.  Using tcpdump I can see Q-Radar making the connection on the correct port to my datanode where Fluentd is running. 

The connection from Fluentd to my namenode is created as logged in the td-agent-log. However no events from Q-Radar are processed.

+++++++++++++++++++++++++++++++

tail -50 /var/log/td-agent/td-agent.log

+++++++++++++++++++++++++++++++

2015-10-20 09:03:39 -0700 [info]: starting fluentd-0.12.12

2015-10-20 09:03:39 -0700 [info]: gem 'fluent-mixin-config-placeholders' version '0.3.0'

2015-10-20 09:03:39 -0700 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'

2015-10-20 09:03:39 -0700 [info]: gem 'fluent-plugin-mongo' version '0.7.10'

2015-10-20 09:03:39 -0700 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.4.1'

2015-10-20 09:03:39 -0700 [info]: gem 'fluent-plugin-s3' version '0.5.9'

2015-10-20 09:03:39 -0700 [info]: gem 'fluent-plugin-scribe' version '0.10.14'

2015-10-20 09:03:39 -0700 [info]: gem 'fluent-plugin-td' version '0.10.27'

2015-10-20 09:03:39 -0700 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.1'

2015-10-20 09:03:39 -0700 [info]: gem 'fluent-plugin-webhdfs' version '0.4.1'

2015-10-20 09:03:39 -0700 [info]: gem 'fluentd' version '0.12.12'

2015-10-20 09:03:39 -0700 [info]: adding match pattern="hdfs.*.*" type="webhdfs"

2015-10-20 09:03:39 -0700 [info]: adding source type="syslog"

2015-10-20 09:03:39 -0700 [info]: using configuration file: <ROOT>

  <source>

    type syslog

    port 5000

    bind 0.0.0.0

    tag system

    with_priority true

  </source>

  <match hdfs.*.*>

    type webhdfs

    host usherlnn01.global.bio-rad.com

    port 50070

    path /user/log/%Y%m%d_%H/access.log.usherldn02.global.bio-rad.com

    username hdfs

    flush_interval 10s

    time_slice_format %Y%m%d%H

  </match>

</ROOT>

2015-10-20 09:03:39 -0700 [info]: webhdfs connection confirmed: usherlnn01.global.bio-rad.com:50070

++++++++++++++++++++++

td-agent.conf 

++++++++++++++++++++++

# setup for accepting evnets and sending to hdfs

# Receive events from 5000/tcp

# This is used by log forwarding and the fluent-cat command

<source>

  type syslog 

  port 5000 

  bind 0.0.0.0

  tag system

</source>

<match hdfs.*.*>

  type webhdfs

  host usherlnn01.global.bio-rad.com

  port 50070

  path /user/log/%Y%m%d_%H/access.log.${hostname}

  username hdfs

  flush_interval 10s

</match>

Given the above what have I missed?

Regards

TimW

[Mr. Fiber]

There is no error?

How about inserting following section to check events are coming or not.

<filter hdfs.**>

  type stdout

</filter>

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tim Washburn]

No net change with the filter parameters.

Is the syslog source looking to the local system syslog daemon for events or acting as a syslog like listener for events from any source that can send to it?

Thanks

TimW

[Mr. Fiber]

Ah, wait.

You set `tag system` in syslog input but you use `<match hdfs.*.*>` for webhdfs output.

This is why webhdfs plugin doesn't receive logs.

Check this document: http://docs.fluentd.org/articles/life-of-a-fluentd-event

[Tim Washburn]

Hello,

Based on your comments and a bit of reading I've altered the source to the following. I know see events in the td-agent log. This is good. However I know have the "incoming chunk is broken" message with the payload. The payload from Q-Radar is formatted in a json format.

+++++++++++++++++++

<source>

  type forward

  port 5000

  bind 0.0.0.0

</source>

<match msg>  # look for "msg" if a match send the event to hdfs?

  type webhdfs

  host usherlnn01.global.bio-rad.com

  port 50070

  path /user/log/%Y%m%d_%H/access.log.${hostname}

  username hdfs

  flush_interval 10s

</match>

log event snippet from td-agent.log:

++++++++++++++++++++++

2015-10-21 09:48:50 -0700 [warn]: incoming chunk is broken: source="host: 10.1.249.38, addr: 10.1.249.38, port: 49097" msg={"name"=>"TrendMicro", "version"=>"1.0", "isoTimeFormat"=>"yyyy-MM-dd'T'HH:mm:ss.SSSZ", "type"=>"Event", "src"=>"192.101.136.117", "dst"=>"208.78.71.34", "srcPort"=>"54788", "dstPort"=>"53", "payload"=>"<190>May 15 2015 17:11:27: %FWSM-6-302016: Teardown UDP connection 144582141468486621 for Internet_DNS:192.101.136.117/54788 to outside:208.78.71.34/53 duration 0:00:00 bytes 258\n"}

+++++++++++++++++++++++

On Tuesday, October 20, 2015 at 9:13:36 AM UTC-7, Tim Washburn wrote:

[Mr. Fiber]

You said you use syslog protocol. Why change syslog to forward?

[Tim Washburn]

Yes it was configured for syslog but being that syslog stream coming from Q-Radar was not being processed I changed to a json output from Q-Radar and reconfigured fluentd for the forward plugin since it accepted json as an input. 

[Mr. Fiber]

Fluentd's forward protocol is not simple JSON.

Your Q-Radar generates this protocol format?

See protocol section in this article > http://docs.fluentd.org/articles/in_forward

in_tcp or in_udp with format json may meet your requirement.

Masahiro

[Tim Washburn]

Hello,

Thank you for the advice. The in_tcp along with fiddling with the <match ...> parameters in the conf file made everything work.

Regards

TimW