Using multiple parsers on a single Syslog Input

[Erwin]

Hello all,

I'm tasked to collect, parse and forward logs from various equipments to a loki instance.
I'm not used to fluentbit so I'm trying to understand how everything works, and I feel a bit lost...

In my environnement, fluentbit will receive all logs trough syslog, preferably on port 514.
However, since we're collecting logs from different equipements, the format is not always the same, and does not conform to the syslog's RFCs, so I need to write custom parsers.

However, I don't understand how i can use different parsers on a single INPUT, since I have to write the parser I want to use directly in the [INPUT] section for the syslog plugin...

Also, I can't say that I want to use this parser only for specific source IP adresses, and another parser for another range of IP adresses?
I'm feeling like I'm missing something, can someone correct me if I'm wrong?

The way I see it, I'm forced to open multiple ports and have different parsers associated to them on my fluentbit instance.


Regards.

[Simon Geard]

I'm very new to Fluent Bit myself (about 2 weeks experience with it), so I'm uncertain if this is recommended practice - but I'm finding that you can do a lot of useful post-processing using filters... e.g. I'm using the tail input to read a directory full of logs, then using parser filters to extract information from the log filenames and contents. If you're trying to separate several log sources from a single input, maybe a combination of parser and rewrite-tag filters could be what you're looking for?

Simon.

[Erwin]

Hello Simon,

that's exactly what I decided to do, and so far so good!
I think I'm forced to use a parser on a syslog input, so I juste use a simple regex that captures everything in a group named "log".
Then a rewrite_tag filter (there's always a hostname that I can use to easily differentiate logs) and then another parser log.
End finally the output to Loki.

One weird thing tough, is that if I use a single Loki output for all my logs, I get no error, but not all my messages are delivered, only messages from one of my parsers.
And if I use 2 outputs with the same sets of labels, I get errors because my messages are "out of order".
Using 2 outputs with different labels for each of my parsers seems to work.

Thanks for your answer Simon, I was not sure it was the correct way to do it, but I guess filters are the way to go :)

--
Erwin