Fluentd in_tail - Does it support log rotation of the source file which is getting tailed

[hyginous neto]

Fluentd in_tail - Does it support log rotation of the source file which is getting tailed?

Is there a inbuilt feature in fluentd ?

[Eduardo Silva]

Yes, it does: 

http://docs.fluentd.org/v0.12/articles/in_tail

On Fri, Jun 30, 2017 at 5:53 PM, hyginous neto <hygi...@gmail.com> wrote:

Fluentd in_tail - Does it support log rotation of the source file which is getting tailed?

Is there a inbuilt feature in fluentd ?

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Eduardo Silva

Open Source, Treasure Data

http://www.treasuredata.com/opensource

 

[hyginous neto]

Hi Eduardo,

Thanks for your answer.

Can you provide an example on how fluentD handles log file rotation itself?

This would help us a lot.

On Friday, 30 June 2017 20:28:22 UTC-7, Eduardo Silva wrote:

Yes, it does: 

http://docs.fluentd.org/v0.12/articles/in_tail

On Fri, Jun 30, 2017 at 5:53 PM, hyginous neto <hygi...@gmail.com> wrote:

Fluentd in_tail - Does it support log rotation of the source file which is getting tailed?

Is there a inbuilt feature in fluentd ?

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Eduardo Silva]

It basically works like this (@repeatedly please correct me if I am wrong)

- Files are monitored over every change (data modification, renamed, deleted)

- When a monitored file is renamed, it's considered a "rotation" if the inode number is always the same.

- File rotated keeps being monitored until "rotate_wait" expires (every 5 seconds by default)

- Fluentd in the meanwhile is scanning the monitored "path" for new file additions every "refresh_interval" expiration.

- If a new file with the same name of the original rotated file appears (and have a different inode number), is tailed from the beginning.

So from a configuration perspective rotate_wait and refresh_interval values are the key to manage rotated files properly, if you have a high frequency of rotated files, make sure to have a low refresh_interval value so Fluent can trap these changes.

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[hyginous neto]

Thanks Eduardo, but still my question is not answered. 

But does fluentd trigger file rotation?

[Eduardo Silva]

Fluentd don't do file rotation, this is mostly done by logrotate or Docker log handler. What Fluentd does is deal with files being rotated

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[hyginous neto]

does fluentd trigger file rotation? 

i could see parameters like  `--log-rotate-age` and `--log-rotate-size`, hence want to check if fluentd itself can trigger log rotation

OR we need logrotate kind of process to do it.

[Eduardo Silva]

that new option available since 0.14.7 is only to auto rotate Fluentd logs (no third party logs or monitored files)

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[hyginous neto]

got it, thanks. 

[Ojas Rahate]

Hi Eduardo,

I wanted to know a mechanism by which Log rotation can be configured to automatically delete log files after a certain amount of time has elapsed!