concatenate logs and parse them after

1,558 views
Skip to first unread message

thomas

unread,
Jul 28, 2016, 10:31:35 AM7/28/16
to Fluentd Google Group
I'm trying to do the following life event :

1- logs comes from docker in JSON

2- Concatenate logs with fluent-plugin-concat

3 - Parse logs int multiple fields (with regex : (?:(?<logtime>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2},\d{1,3}) (?<app_loglevel>[A-Z]+) +\[(?<app>[a-z]+):(?<process>[a-z]+)\] (?<log>(?:.|\s)*)

4- Output to ES / stdout

I am able to do 1,2 and 4 points, and it works well, but I don't know how to do #3
Regex is for example usable in the in_tail plugin ; but ' can't figure how to do this in filters.

Does anyone have an idea ?


Below my current conf
<source>
  @type tail
  path /var/log/containers/*.log
  pos_file /tmp/es-containers.log.pos
  time_format %Y-%m-%dT%H:%M:%S.%N
  tag kubernetes.*
  format json
  read_from_head true
  keep_time_key true
</source>

<filter kubernetes.*>
  @type concat
  key log
  multiline_start_regexp /^\d{4}-\d{1,2}-\d{1,2}/
</filter>

<match **>
  @type stdout
</match>


thanks,
thomas

Mr. Fiber

unread,
Jul 28, 2016, 10:56:13 PM7/28/16
to Fluentd Google Group
Does anyone have an idea ?

fluent-plugin-parser filter help you.



Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

thomas

unread,
Aug 31, 2016, 4:22:35 AM8/31/16
to Fluentd Google Group
@masahiro , thanks ; combined with fluent-plugin-concat it is what i was looking for
Reply all
Reply to author
Forward
0 new messages