Fluentd tail plugin: tail all files in a directory

[Abhishek Tiwari]

Fluentd accepts CSV filenames to log. But that too implies a prior knowledge of file-names.

<source>
   type tail
   path /var/log/nginx/*.log
   tag logging
   format /^(?<time>.+) \[(?<level>[^\]]+)\] *(?<message>.*)$/
   time_format %Y/%m/%d %H:%M:%S
</source>

Is there an option or a hack to do something logically equivalent to- path /var/log/*.log

[Masahiro Nakagawa]

Hi,

Could you try tail_ex plugin instead?

https://github.com/yosisa/fluent-plugin-tail-ex

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Abhishek Tiwari]

Sounds like the fix. Thank you. :)
Question- why don't we pull this into the 'real' tail plugin itself?

[Abhishek Tiwari]

@repeatedly- This plugin does pick-up, say all files if specified-

path = /var/log/module/*.log

How do i go about making the .pos files corresponding to each of the files resolved by the plugin above?
Is something like this possible-

pos_path = /var/run/fluentd/*.log.pos

[Abhishek Tiwari]

i realized that a single pos file can account for multiple tiled logs too. :)
Consider this closed.

[Masahiro Nakagawa]

Sorry for the late reply.

> i realized that a single pos file can account for multiple tiled logs too.

Yes.

pos_file contains multiple file informations using offset.

So in_tail can treat multiple files with one pos_file.

[Abhishek Tiwari]

It is kind of you to say that. :)

i have another question- do fluentd and/or the tail_ex plugin dynamically scan the directory for new files? If i mention-- "path /tmp/log_dir/*.log" in the config and initiate fluentd at t=0 and create x.log at t=10sec, will tail_ex tail this new file?

[Masahiro Nakagawa]

Maybe, refresh_interval is for this case.

See README: https://github.com/yosisa/fluent-plugin-tail-ex