Multi process on file_output and priority in in_tail

Skip to first unread message

Anirudh Venkatesh

Sep 24, 2021, 4:33:02 PM9/24/21
to Fluentd Google Group

While use multiprocess worker threads, are there examples available to configure out_file plugin to be able to write to the same destination?

Also, is there a way to make certain logs have more priority over others when using in_tail plugin?

The scenario is as follows -
We are trying to stream 4 kinds of logs to a central host that collects these from 30 different forwarder hosts.

One of these logs needs to be prioritized so that they are shipped faster than the others. Is there a way to do this?

Also on the central host where we collect all the logs, we want to use multi process workers so that in_forward can make use of multiple cpu cores to speed the process of ingestion but with out_file we want to be able to collect these logs to the same file.

For eg.,
Forwarder_host1 - file1.txt
Forwarder_host2 - file2.txt
Forwarder_host3 - file3.txt

Is collected on the aggregator (centralized host) as -

This works as expected with single process but when using multiprocess (2 processes), we see that files are collected as follows -

Where 0 and 1 indicate the worker threads, is there a way with multi-process where out file can still write it to /path/Forwarder_host1_file1.txt and so on instead of splitting by worker thread?

Any examples on either will be very useful. Thanks for your help!


Reply all
Reply to author
0 new messages