tail input didnt catch rotated log

[Evgeny Kornev]

Hello!
Suddenly fluent-bit (works in kubernetes) lost rotated log file, here is debug output:

[2017/09/05 05:06:46] [debug] [in_tail] rotated: /mnt/phone-1304852214-9xpzx_default_phone-a783b693ec49347de033bb7931b65f4bf0903c609ddb4421ffaeb2b99c37031e.log -> /var/lib/docker/containers/a783b693ec49347de033bb7931b65f4bf0903c609ddb4421ffaeb2b99c37031e/a783b693ec49347de033bb7931b65f4bf0903c609ddb4421ffaeb2b99c37031e-json.log.1

[2017/09/05 05:06:46] [debug] [out_fw] 17 entries tag='kube.mnt.phone-1304852214-9xpzx_default_phone-a783b693ec49347de033bb7931b65f4bf0903c609ddb4421ffaeb2b99c37031e.log' tag_len=122

[2017/09/05 05:06:47] [debug] [dyntag tail.0] 0x7efc79a3f0c0 destroy (tag=kube.mnt.phone-1304852214-9xpzx_default_phone-a783b693ec49347de033bb7931b65f4bf0903c609ddb4421ffaeb2b99c37031e.log)

[2017/09/05 05:06:52] [debug] [in_tail] removed /mnt/phone-1304852214-9xpzx_default_phone-a783b693ec49347de033bb7931b65f4bf0903c609ddb4421ffaeb2b99c37031e.log

It is strange also that new file has access permissions 400, while almost every other is 640.
Other 400 files also were not got by tail module after rotation.
Is the problem in docker by any chance?

[Eduardo Silva]

Who is rotating the files? Docker or logrotate tool ?

--
You received this message because you are subscribed to the Google Groups "Fluent-Bit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluent-bit+unsubscribe@googlegroups.com.
To post to this group, send email to fluen...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fluent-bit/aa8143dd-af37-4303-bf20-800feae0e511%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Evgeny Kornev]

Eduardo, docker rotates it by file size

пятница, 8 сентября 2017 г., 4:51:06 UTC+5 пользователь Eduardo Silva написал:

[Eduardo Silva]

What's the docker log max size defined ? And the average of number of lines per file and per second?

--
You received this message because you are subscribed to the Google Groups "Fluent-Bit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluent-bit+unsubscribe@googlegroups.com.
To post to this group, send email to fluen...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/fluent-bit/ffb01cb8-e769-4553-8db7-41b25848e3a6%40googlegroups.com.

[Evgeny Kornev]

It's 200mb limit
rotated file has about 1010250 lines and 24,36 lines per second

пятница, 8 сентября 2017 г., 19:08:32 UTC+5 пользователь Eduardo Silva написал:

[Eduardo Silva]

thanks for the info.

Now how many containers is Fluent Bit monitoring when that log rotation happens ?

--
You received this message because you are subscribed to the Google Groups "Fluent-Bit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluent-bit+unsubscribe@googlegroups.com.
To post to this group, send email to fluen...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/fluent-bit/36a382c3-446c-4dfe-9991-b55ee43a220c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Eduardo Silva

Open Source, Treasure Data

http://www.treasuredata.com/opensource

 

[Evgeny Kornev]

Eduardo, sorry for delay, there is 43 links in /var/log/containers/ headed to links in /var/log/pods


пятница, 8 сентября 2017 г., 20:11:15 UTC+5 пользователь Eduardo Silva написал:

thanks for the info.

Now how many containers is Fluent Bit monitoring when that log rotation happens ?