Need some help to parse log on different filter and send to different output

[Vinoth Narasimhan]

I  am new to fluentd. I  am trying to send the different format message into a different output using tags and filter.

The event are comes in different format 

one with the name matches the regexp resellerID.I need to parse this log as special and apply some filter and send to different output

other event as default logs which send to different output. 

I am not able achieve this with the following rules

<source>

  @type tail

  path /root/fluentd/1.log

  pos_file /var/log/td-agent/1.log.pos

  read_from_head

  tag tail

  tag event

  format json

</source>

<filter tail.**>

  @type parser

  key_name message

  format json

  reserve_data true

</filter>

<filter event.**>

  @type parser

  key_name log

  format /(?<bms>{.*resellerId.*})/

  reserve_data true

  suppress_parse_error_log true

  time_format %Y-%m-%dT%H:%M:%S.%NZ

</filter>

<filter event.**>

  @type parser

  key_name bms

  format json_in_json

  reserve_data true

  suppress_parse_error_log true

  time_format %Y-%m-%dT%H:%M:%S.%NZ

</filter>

## File output

## match tag=local.** and write to file

<match tail.**>

  @type file

  path /var/log/td-agent/access

</match>

<match event.**>

  @type file

  path /var/log/td-agent/event

</match>

Please shed some light on how to parse the event with different filters and send them to different output

[Mr. Fiber]

I'm not sure your actual log format so I don't know why such parser chain is needed.

Generally, if you need to chain parser plugin, writing own parser is better.

BTW, there are several approches to apply different filters and outputs

1. copy + label

https://docs.fluentd.org/v0.12/articles/routing-examples#re-route-event-to-other-label

Copying all events to different pipelines. After copied, you can apply different filters / outputs.

2. rewrite tag by record content

https://docs.fluentd.org/v0.12/articles/routing-examples#re-route-event-by-record-content

If you can route events to different pipeline by record content, rewrite-tag-filter is helpful.

Of course, avoiding mixed logs is best :)

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Vinoth Narasimhan]

Thanks Masahiro for ur reply.

I am trying here to separate the event into a different match output in a different index name in ES. To test this I play with the output as file.

The event comes with 3 different format.

All the event data should go to some default index in ES

If the event has the format apache2 access  logs then it goes to the index "access" in ES

If the event has some inner json. I need to apply the parser json_in_json and send them to other index.

Are you suggesting this:

<source>

type tail

..

@label @access

@label @json

</source>

<label @access>

..

access filter and match stuff

</label>

<label @json>

..

parse json_in_json

</label>

Please suggest where the copy comes.

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.

[Vinoth Narasimhan]

For the format on detail.

Format 1: 

{"log":"[2017-11-13T02:02:00.000Z] [main] [157] [INFO ] Hello World \n","stream":"stdout","time":"2017-11-10T12:15:04.49863245Z"}

Format 2 :

{"log":"127.0.0.1 - frank [14/Nov/2017:02:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326 1258 \"http://www.example.com/start.html\" \"Mozilla/4.08 [en] (Win98; I ;Nav)\"\n","stream":"stdout","time":"2017-11-11T07:01:00.092878761Z"}

Format 3:

{"log":"[2017-11-13T02:02:00.000Z] [n.access.LogbackAccessContext] [main] [157] [INFO ] Configured the Logback-access: context=[default], config=[classpath:logback-access.xml] {\"timestamp\":\"1510217470\",\"event\":\"participantStatus\",\"rese\":\"1114468272\",\"participant\":\"d26322ce5\",\"services\":[\"share\",\"audioVideo\"],\"profile\":{\"name\":\xyz\",\"productUserId\":\"example.com\",\"isOwner\":true,\"locale\":\"en_US\",\"type\":\"participant\"}\n","stream":"stdout","time":"2017-11-10T12:15:04.49863245Z"}

All the format logs will parsed as below and send to default index in ES

{

log:".....",

stream: "...",

+time

}

The format with apache2 access will go the index "access"

The format with inner josn will go to "json"

I used the way that suggested using route by tag.

I can achive the last 2 . But i cannot parse  all the logs into the default index.

Below is the sample conf i tried:

<source>

  @type tail

  path /root/fluentd/1.log

  pos_file /var/log/td-agent/1.log.pos

  read_from_head

  tag worker.*

  format none

</source>

<match worker.**>

  remove_tag_prefix worker

  add_tag_prefix metrics.event

  @type route

  <route **>

    copy

    @label @access

  </route>

  <route **>

    copy

    @label @json

  </route>

  <route **>

    copy

    @label @default

  </route>

</match>

<label @default>

  <filter metrics.event.**>

    @type parser

    format json 

    key_name log

    reserve_data true

  </filter>

  <match metrics.event.**>

    @type file

    path /var/log/td-agent/default

  </match>

</label>

<label @access>

  <filter metrics.event.**>

    @type parser

    format apache2

    key_name log

    suppress_parse_error_log true

    ignore_key_not_exist true

    time_format %d/%b/%Y:%H:%M:%S %z

  </filter>

  <match metrics.event.**>

    @type file

    path /var/log/td-agent/apacheaccess

  </match>

</label>

<label @json>

  <filter metrics.event.**>

     @type parser

     key_name log

     format /(?<ms>{.*"participantStatus.*})/

     suppress_parse_error_log true

     ignore_key_not_exist true

  </filter>

  <filter metrics.event.**>

     @type parser

     key_name ms

     format json_in_json

     suppress_parse_error_log true

     ignore_key_not_exist true

     time_key timestamp

  </filter>

  <match metrics.event.**>

    @type file

    path /var/log/td-agent/json

  </match>

</label>

I am stuck with the above conf

On Friday, December 1, 2017 at 8:16:45 PM UTC+5:30, repeatedly wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.

[Mr. Fiber]

For such cases, rewrite-tag-filter seems better.

Here is an example conf and test log:

- fluent.conf

<source>

  @type tail

  @id tail_input

  path /path/to/test.log

  <parse>

    @type json

    time_format %iso8601

  </parse>

  read_from_head true

  tag log

</source>

<match log>

  @type rewrite_tag_filter

  @id rewrite_tag_filter

  <rule>

    key     log

    pattern \}$

    tag     json

  </rule>

  <rule>

    key     log

    pattern ^\[

    tag     access

    invert  true

  </rule>

  <rule>

    key     log

    pattern .+

    tag     default

  </rule>

</match>

<filter json>

  @type record_transformer

  <record>

    type json

  </record>

</filter>

<filter access>

  @type record_transformer

  <record>

    type access

  </record>

</filter>

<filter default>

  @type record_transformer

  <record>

    type default

  </record>

</filter>

<match {json,access,default}>

  @type stdout

</match>

- test.log

{"log":"[2017-11-13T02:02:00.000Z] [main] [157] [INFO ] Hello World \n","stream":"stdout","time":"2017-11-10T12:15:04.49863245Z"}

{"log":"127.0.0.1 - frank [14/Nov/2017:02:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326 1258 \"http://www.example.com/start.html\" \"Mozilla/4.08 [en] (Win98; I ;Nav)\"\n","stream":"stdout","time":"2017-11-11T07:01:00.092878761Z"}

{"log":"[2017-11-13T02:02:00.000Z] [n.access.LogbackAccessContext] [main] [157] [INFO] Configured the Logback-access: context=[default], config=[classpath:logback-access.xml] {\"timestamp\":\"1510217470\",\"event\":\"participantStatus\",\"rese\":\"1114468272\",\"participant\":\"d26322ce5\",\"services\":[\"share\",\"audioVideo\"],\"profile\":{\"name\":\"xyz\",\"productUserId\":\"example.com\",\"isOwner\":true,\"locale\":\"en_US\",\"type\":\"participant\"}\n","stream":"stdout","time":"2017-11-10T12:15:04.49863245Z"}

- result

2017-11-10 21:15:04.498632450 +0900 default: {"log":"[2017-11-13T02:02:00.000Z] [main] [157] [INFO ] Hello World \n","stream":"stdout","type":"default"}
2017-11-11 16:01:00.092878761 +0900 access: {"log":"127.0.0.1 - frank [14/Nov/2017:02:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326 1258 \"http://www.example.com/start.html\" \"Mozilla/4.08 [en] (Win98; I ;Nav)\"\n","stream":"stdout","type":"access"}
2017-11-10 21:15:04.498632450 +0900 json: {"log":"[2017-11-13T02:02:00.000Z] [n.access.LogbackAccessContext] [main] [157] [INFO] Configured the Logback-access: context=[default], config=[classpath:logback-access.xml] {\"timestamp\":\"1510217470\",\"event\":\"participantStatus\",\"rese\":\"1114468272\",\"participant\":\"d26322ce5\",\"services\":[\"share\",\"audioVideo\"],\"profile\":{\"name\":\"xyz\",\"productUserId\":\"example.com\",\"isOwner\":true,\"locale\":\"en_US\",\"type\":\"participant\"}\n","stream":"stdout","type":"json"}

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.