Nginx logs to s3 using syslog+fluentd

[Sanket Gupta]

Hi
Recently started looking at fluentd and it seems to be a great piece of software
We run a cluster of Nginx servers which currently upload the hourly logs to S3 using cronjobs.We would now like to use Fluentd to push all logs to a central server which will then process and upload the logs to s3.Have a few questions on this

1. We plan to send data to fluentd using syslog protocol.Should we run the syslog enabled fluentd instance locally  and then configure forwarding to central server or should we write logs directly to the central server? Does the local+forwarding approach offer any network\compression\efficiency benefits
2. We generally do not lose data in file based logging. How sure can we be about data not being lost if we use fluentd with syslog protocol?
3. We would like the central fluentd instance to upload the logs to s3 and have separate files for each server with the IP of the server present  in the name of each file along with timestamp! How do we go about implementing this?

Thanks for all your support!

[Masahiro Nakagawa]

Hi Sanket,

> Recently started looking at fluentd and it seems to be a great piece of software

Thank you for trying Fluentd ;)

1. We recommend to setup Fluentd locally because Fluentd has reliable buffering and load balancing with out_forward plugin.

    It is standard way of Fluentd centralized logging.

2. You can use TCP syslog protocol and file buffer with out_forward. In this case, your received log is stored in the file first.

    After that, out_forward flushes file content to another server.

3. Hmm. If you use unique tag instead of IP, you can simply use out_s3 plugin with out_forest plugin.

https://github.com/tagomoris/fluent-plugin-forest#forestoutput

out_forest creates an output plugin with actual tag dynamically.

Server1: in_syslog -> syslog.server1_tag -> out_forward -> in_forward -> out_forest(create out_s3 with server1_tag) -> out_s3

Server2: in_syslog -> syslog.server2_tag -> ...

If you want to use IP(or hostname) in the S3 path, maybe you need rewrite-tag-filter plugin.

https://github.com/fluent/fluent-plugin-rewrite-tag-filter

rewrite-tag-filter routes the event using actual record value and can reference it for tag routing.

For example:

    rewriterule1 domain ^(mail)\.(example)\.com$ site.$2$1  # $1 refs (mail), $2 refs (example)

Flow example is below:

Server1: in_syslog -> syslog.IP1(or hostname1) -> (out and in)forward -> out_rewrite_tag_filter -> out_forest -> out_s3.

Server2: in_syslog -> syslog.IP2(or hostname2) -> ...

tagomori-san or yoshi-ken san:

Do you have any idea for 3 case?

Thanks,

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Sanket Gupta]

Masahiro
Thanks a lot for all your insights.

Let me try implementing your suggestions for 3 and will get back with more questionsvery soon.

Keep up the great work