Fuentd syslogs are not filtering in logstash using grok filter

[Parima Soni]

Hi All,

I need help to filter the  logs in logstash using grok filter.

Logs are coming from fluentd:
Fluentd syslog:-

Example: 2017-06-30 22:23:23 +0000 syslog.daemon.info: {"host":"70518cc9d3bf[1056]:","ident":"","message":"53:23Z DockerMc001.hp.com PaymentSrv - ZL-DESMA-OW - BOM PaymentID 3 state updated to Production"}

Logstash grok filter:
filter {
    # Manually parse the log, as we want to support both RCF3164 and RFC5424
    grok {
      break_on_match => true
      match => [
        "message", "%{SYSLOG5424LINE}",
        "message", "%{SYSLOGLINE}"
      ]
    }

    if [syslog5424_ts] {
      # Handle RFC5424 formatted Syslog messages
     
      mutate {
        remove_field => [ "message", "host" ]
         add_tag => [ "syslog5424" ]
      }
     mutate {
           rename => {
                "syslog5424_app" => "services"
                "syslog5424_msg" => "message"
                "syslog5424_host" => "host"
           }
           remove_field => ["syslog5424_ver", "syslog5424_proc"]
     }
      if [syslog5424_pri] {
        # Calculate facility and severity from the syslog PRI value
        ruby {
          code => "event.set('severity', (event.get('syslog5424_pri').to_i).modulo(8))"
        }
        ruby {
          code => "event.set('facility', ((event.get('syslog5424_pri').to_i) / 8))"
        }
        mutate {
           remove_field => [ "syslog5424_pri" ]
        }
      }
     
      date {
        match => [ "syslog5424_ts", "ISO8601" ]
        remove_field => [ "syslog5424_ts", "timestamp" ]
      }
    }
    else {
      # Handle RFC3164 formatted Syslog messages
      mutate {
        add_tag => [ "syslog3164" ]
      }
    }
}

I am getting logs in logstash:

"@timestamp" => 2017-06-30T19:40:14.132Z,
      "@version" => "1",
          "host" => "172.17.0.4",
          "type" => "http",
       "message" => "shutting down input type=\"http\" plugin_id=\"object:3ffb1328078c\"",
          "tags" => [
        [0] "_grokparsefailure",
        [1] "syslog3164"
    ]
}
 
anyone has any idea to fix this issue, please let me know.

It will be appreciatable

Thanks & Regards:
Parima

[Mr. Fiber]

Is this logstash question?

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Parima Soni]

Yes, it's related to Logstash,

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.

[Raimonds Liepiņš]

I am experiencing the same issue, the script provided is not quite working. My format is [ value = field ] [ value2 = field2 ] [ value3 = field3 ].