Fluentd filter

[Parima Soni]

Hi All,

Rightnow, I am working on fluentd.config file for centralizing the logs.previously I was working with logstash, I wrote grok filter for logstash config file, Now I need to write the same concept in fluentd config file with fluentd standards.

Can anyone help me to write fluentd filter for RFC5425 syslog.

" 
      grok {
      break_on_match => true
      match => [ 
        "message", "%{SYSLOG5424LINE}",
        "message", "%{SYSLOGLINE}"
      ]
    }
 
    if [syslog5424_ts] {
      # Handle RFC5424 formatted Syslog messages
      
      mutate {
        remove_field => [ "message", "host" ]
    add_tag => [ "syslog5424" ]
      }
    mutate {
        rename => {
            "syslog5424_app" => "services"
            "syslog5424_msg" => "message"
            "syslog5424_host" => "host"
        }
        remove_field => ["syslog5424_ver", "syslog5424_proc"]
    }
      if [syslog5424_pri] {
        # Calculate facility and severity from the syslog PRI value
        ruby {
          code => "event.set('severity', (event.get('syslog5424_pri').to_i).modulo(8))"
        }
        ruby {
          code => "event.set('facility', ((event.get('syslog5424_pri').to_i) / 8))"
        }
        mutate {
          remove_field => [ "syslog5424_pri" ]
        }
      }
 
if [syslog5424_msgid] {
# Extract the log zone level, zone name and workflow
ruby {
    code => "event.set('Level',((event.get('syslog5424_msgid').split(/\-+/))).at(0))"
}
ruby {
    code => "event.set('Zone',((event.get('syslog5424_msgid').split(/\-+/))).at(1))"
}
ruby {
    code => "event.set('Workflow',((event.get('syslog5424_msgid').split(/\-+/))).at(2))"
    }
    mutate {
        remove_field => [ "syslog5424_msgid" ]
    }
   }   
      date {
        match => [ "syslog5424_ts", "ISO8601" ]
        remove_field => [ "syslog5424_ts", "timestamp" ]
      }
    }
    else {
      # Handle RFC3164 formatted Syslog messages
      mutate {
        add_tag => [ "syslog3164" ]
      }
    }
}
"

Thanks for your help.

Regards:
Parima

[Eduardo Silva]

Hi Parima,

Fluentd already comes with pre-configured parsers for Syslog, please review the following doc:

http://docs.fluentd.org/v0.12/articles/in_syslog

Regards

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Parima Soni]

Hi Eduardo,

Thanks for your reply.

I am using @type=syslog in Source, and I am getting logs in syslogrfc5424, but I want to mutate some fields and remove them and also some fields need to be renamed.like I was using in grok filter in logstash config file.

Please assist me on same .

Regards:
Parima

Regards

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.

[Eduardo Silva]

My suggestion is:

- Use syslog input plugin, make sure to set message_format to rfc5424

- Once the data goes through the input and parsed, then you need to filter it (filter allows to modify a record outside of the input plugin context)

- To exclude or match specific fields you can use grep filter, for more advanced fields handling check record_transformer filter. You can use both filters each one after the other in your configuration.

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Eduardo Silva

Open Source, Treasure Data

http://www.treasuredata.com/opensource

 

[Parima Soni]

Hi Eduardo, 

Please see the below where I am wrinting the filter grep and record_transformation in my td-agent config file, But I am not getting mutate fields and rename fields in kibana.

<filter syslog>

@type grep

regexp1 message SYSLOG5424LINE

regexp2 SYSLOG5424 SYSLOGLINE

regexp3 prival prival

regexp4 timestamp timpstamp

regexp5 host hostname

</filter>

<filter syslog>

@type record_transformer

  enable_ruby

mutate

  remove_message,remove_host #remove fields.

  <record>

    add_tag ${syslog5424}

  </record>

</filter>

<filter syslog>

@type record_transformer

  enable_ruby

mutate

  remove_syslog5424_ver,remove_syslog5424_proc #remove fields.

renew_record true

  <record>

syslog5424_app ${syslog5424_app.to_s}

 add_tag ${services}

syslog5424_msg ${syslog5424_msg.to_s}

 add_tag ${message}

syslog5424_host ${syslog5424_host.to_s}

 add_tag ${host}

  </record>

</filter>

Please assist me to write custome filter in correct way for syslog5424.

Regards:

Parima Soni

[Eduardo Silva]

Please put your full config file in a public address to see it (gist, pastebin, etc)

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

[Parima Soni]

Hi Eduardo, 

Please find the full config file here in below:

# Listen to incoming data over SSL

 <source>

    @type syslog

    path /var/log/syslog

    port 42185

    bind 0.0.0.0

    tag syslog_rfc5424

    </source>

 # Store Data in Elasticsearch --> Here Fluentd and ElasticSearch are deployed on the same system.

<match *.**>

    @type copy

    <store>

      @type stdout

    </store>

    <store>

      @type elasticsearch

      host 172.17.0.2

      port 9200

    index_name fluentd

    type_name syslog

    logstash_format true

    flush_interval 10s

</store>

</match>

Please check and let me know the corrections in my config file,

Thanks & Regards:

Parima

[Eduardo Silva]

Likely your filters are not being applied because of the filter match rule:

- Your input are tagged with syslog_rfc5424

- Your filter expect "syslog"

Instead add a wildcard to your filter rules, e.g: <filter syslog**>

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Parima Soni]

Hi Eduardo,

Thanks for your suggestions, but now I am not able to integrate fluentd with elasticsearch.

Can you please help me to write custom filter in fluentd as I wrote grok filter in logstash. I want the exact same in fluentd standards.

for Example: 

Dummy :

I want like this :

Prival=15,

hostname=abc.com

Services=XXXX

zone=ZL-xxx-ow

message=abs is in under process

type=syslog

timesatamp=current date

message_ID: 123ghg6

Thanks for your help.

Regards:

Parima

[Eduardo Silva]

The record_modifier plugins allows you to ingest/modify records, some of these things are already handled by syslog plugin by Fluentd. Your previous configuration looks OK to ingest data into Elasticsearch.

I suggest you try that plugin and make sure the output looks OK for stdout, once is ready add the elasticsearch setup, then you should be able to query the indices from elasticsearch (curl http://host:ip/_cat/indices) 

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Parima Soni]

HI Eduardo, 

that is not working for me as per my requirements:

this is my hard_coded message to elasticsearch through fluentd 

                "47:53,553 INFO main log4jExample main - <15>1 DockerMc001.hp.com OrderSrv - ZL-Desma-OW - BOM OrderID state updated to Production"

I want to segregate this message to syslog rfc5424 fields.

can you please help me to set according to the same.

its urgent.

Thanks

Parima

[Eduardo Silva]

Please upload you most recent configuration to a public place for review like pastebin 

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.