Multi process on file_output and priority in in_tail

[Anirudh Venkatesh]

Hello,

While use multiprocess worker threads, are there examples available to configure out_file plugin to be able to write to the same destination?

Also, is there a way to make certain logs have more priority over others when using in_tail plugin?

The scenario is as follows -

We are trying to stream 4 kinds of logs to a central host that collects these from 30 different forwarder hosts.

One of these logs needs to be prioritized so that they are shipped faster than the others. Is there a way to do this?

Also on the central host where we collect all the logs, we want to use multi process workers so that in_forward can make use of multiple cpu cores to speed the process of ingestion but with out_file we want to be able to collect these logs to the same file.

For eg.,
Forwarder_host1 - file1.txt

Forwarder_host2 - file2.txt

Forwarder_host3 - file3.txt

Is collected on the aggregator (centralized host) as -

/path/Forwarder_host1_file1.txt

/path/Forwarder_host2_file2.txt

/path/Forwarder_host3_file3.txt

This works as expected with single process but when using multiprocess (2 processes), we see that files are collected as follows -

/path/0/Forwarder_host1_file1.txt
/path/1/Forwarder_host1_file1.txt

/path/0/Forwarder_host2_file2.txt
/path/1/Forwarder_host2_file2.txt

/path/0/Forwarder_host3_file3.txt
/path/1/Forwarder_host3_file3.txt

Where 0 and 1 indicate the worker threads, is there a way with multi-process where out file can still write it to /path/Forwarder_host1_file1.txt and so on instead of splitting by worker thread?

Any examples on either will be very useful. Thanks for your help!

Regards,

Anirudh