I'm trying to do the following life event :
1- logs comes from docker in JSON
2- Concatenate logs with fluent-plugin-concat
3 - Parse logs int multiple fields (with regex : (?:(?<logtime>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2},\d{1,3}) (?<app_loglevel>[A-Z]+) +\[(?<app>[a-z]+):(?<process>[a-z]+)\] (?<log>(?:.|\s)*)
4- Output to ES / stdout
I am able to do 1,2 and 4 points, and it works well, but I don't know how to do #3
Regex is for example usable in the in_tail plugin ; but ' can't figure how to do this in filters.
Does anyone have an idea ?
Below my current conf
<source>
@type tail
path /var/log/containers/*.log
pos_file /tmp/es-containers.log.pos
time_format %Y-%m-%dT%H:%M:%S.%N
tag kubernetes.*
format json
read_from_head true
keep_time_key true
</source>
<filter kubernetes.*>
@type concat
key log
multiline_start_regexp /^\d{4}-\d{1,2}-\d{1,2}/
</filter>
<match **>
@type stdout
</match>
thanks,
thomas