Changing logstash by fluentd

[C. L. Martinez]

Hi all,

 How complicated can it be to migrate the logstash configuration to fluentd? We are experiencing serious performance problems on our Elastic server due to the high volume of logs we receive. We have performed log load tests using fluentd and the performance is far superior.

 Any sample?

[Pedro Sánchez]

Hi Martinez,

I am not an expert in Fluentd but as much as it is able to ingest JSON data and output events to Elasticsearch the integration won't be hard, feel free to share your entire use case here so we can help.

Are you using Fluentd instead of Logstash but at the end sending events to Elasticsearch? I think it is always good to use software from the same vendor to increase compatibility (Logstash->Elasticsearch).

Maybe you can elaborate the infrastructure you have in mind.

Regards,

Pedro 'snaow' Sanchez.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/CAEjQA5KRVuPY934m52ySmny%3DkB38trv300p7%2BzQ-mzVd0-tg%3Dw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[C. L. Martinez]

Hi Pedro,

 

 We are managing between 6K-6.5k EPS in our ELK/Wazuh infrastructure ... Our ELK infrastructure consists in one master node (with elastic+kibana+wazuh app) and two data nodes (elastic+logstash). Our EL data nodes comes with 16 GB RAM and 4 vCPU. When we need to process IDS logs with Wazuh and ELK, logstash tries to consume all free memory in both data nodes and they start to consume all swap. Same deployment, but using fluentd and redirecting all IDS logs to fluentd, works without problems and we don't sufering perfomance problems... Maybe/probably we need to balance our work load using more EL data nodes, but it is not possible at this moment ...

Thanks.