Fluentd parser plugin pattern not matched with data

[Saurabh]

Hi I'm writing a custom plugin for fluentD, I need to use this plugin to parse one field from syslog5424 format, the ruby code works just fine when you run it without fluentd but not when it runs as a plugin  

error:

 #<Fluent::Plugin::Parser::ParserError: pattern not matched with data '[kubernetes@47450 app="api-mma" pod-template-hash="9849d88d6" namespace_name="qa" object_name="api-mma-9849d88d6-59mzd" container_name="api-mma" vm_id="91cdf0e0-b1fb-45bc-8a3f-c85b027af505"]'>

The custom plugin:

require 'fluent/plugin/parser'

module Fluent::Plugin

  class TKGIMetadataParser < Parser

    # Register this parser as 'tkgi_metadata'

    Fluent::Plugin.register_parser('tkgi_metadata', self)

    # `delimiter` is configurable with ' ' as default

    config_param :delimiter, :string, default: ' '

    # `time_format` is configurable

    config_param :time_format, :string, default: nil

    def configure(conf)

      super

      if @delimiter.length != 1

        raise ConfigError, "delimiter must be a single character. #{@delimiter} is not."

      end

    end

    def parse(text)

      source, key_values = text.split(' ', 2)

      source.gsub!(/[\[!@%&"0-9]/,'')

      record = {}

      key_values.gsub!(/[\]]/,'')

      key_values.split(@delimiter).each do |kv|

        k, v = kv.split('=', 2)

        record[k] = v

      end

      record.merge!(source: source)

      yield source, record

    end

  end

end

fluentd.conf

<filter k8s> 

 @type parser 

 key_name syslog5424_sd 

 reserve_data true 

 reserve_time true 

 <parse> 

  @type tkgi_metadata 

 </parse> 

</filter>  

sample log:

[kubernetes@47450 app="api-mma" pod-template-hash="9849d88d6" namespace_name="qa" object_name="api-mma-9849d88d6-59mzd" container_name="api-mma" vm_id="91cdf0e0-b1fb-45bc-8a3f-c85b027af505"]

[Kentaro Hayashi]

Hi,

It seems that there is a tkgi metadata parser [1], but does not fit for you?

[1] https://github.com/srbhklkrn/fluent-plugin-tkgi-metadata-parser [1]

Any way, It seems that time is missing.

Typically, parser yields time and record in pair.

def parse(text)

  ...

  yield time, record

end

ref. https://docs.fluentd.org/plugin-development/api-plugin-parser

Regards,

2020年12月30日水曜日 5:40:10 UTC+9 saurabhku...@gmail.com: