segregation of specific container logs using fluentd

[Paras Joshi]

Hello ,

I am new to fluentd . i need some help on segregation of specific container logs to cloud-watch logs groups .

Currently , i have the setup the fluentd setup on my EKS cluster , from where i am pushing all the cluster logs to cloudwatch . Thats seems to be Ok .

Now i need to check if the we could send specific logs which are created inside the container var/audit/ like below to specific log groups . 

access.audit.json  

activity.audit.json

authentication.audit.json  

config.audit.json

currently every log files are unified and aggregated .

I need to segregate few specifc files in fluentD and sent it to cloudwatch .

Please let me know if someone has any details or good blog to implement it .

Thanks in advance.

Br,

Paras

[Paras Joshi]

hello ,

Can anyone guide here . I am bit stuck here .

Br,

Paras

[cosmo09...@gmail.com]

Hi,

When using in_tail to tailing the following files:

> Now i need to check if the we could send specific logs which are created inside the container var/audit/ like below to specific log groups . 

> access.audit.json  

> activity.audit.json

> authentication.audit.json  

> config.audit.json

in_tail would be attached the different tags which is based on filenames when you use asterisk on tag parameter:
* https://docs.fluentd.org/input/tail#tag

Then, you can use built-in placeholder feature on out_cloudwatch_logs' log_stream_name parameter:
https://github.com/fluent-plugins-nursery/fluent-plugin-cloudwatch-logs#out_cloudwatch_logs

You can use built-in placeholders for cloudwatch_logs configuration like this:

log_stream_name var-audit-${tag} # Or, var-audit-${tag[0]}

auto_create_stream true

Built-in placeholders explanations is here:

https://docs.fluentd.org/configuration/buffer-section#placeholders

Hope this helps.

Best,

Hiroshi

2021年4月7日水曜日 3:03:08 UTC+9 zoom2...@gmail.com: