Can we dynamically create directories based on the file output plugin?

[Christian Hedegaard]

Based on the timeslice?

 

I’m using the forest output plugin along with file and s3 to store syslog from my machines that come in via fluentd.

 

I’d like to have fluentd store logs like this: /directory/2014/03/05/hostname.timeslice.log

 

I think with the file output plugin and forest, I can accomplish basically everything but the /2014/03/05 dynamically, unless there is a hidden config option I don’t know about.

 

Alternatively, is there another file output plugin that can do this? I don’t want to store 24, hourly log files for 300’ish hosts per day in a single directory if I can otherwise help it.

 

Thanks!

[Kiyoto Tamura]

Looked at the code and it does not look that way.

One way around it is creating different directories for each of the 300 hosts using out_forest? https://github.com/tagomoris/fluent-plugin-forest

Basically, i am suggesting /directory/hostname/yyyyMMdd.log. If this does not meet your needs, let us know.

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Check out Fluentd, the open source data collector for high-volume data streams

[Christian Hedegaard]

I’ve considered doing that, and we’ve done it in the past, but it makes it easier to keep a day’s worth of logs per directory for all hosts since we have a lot of clusters so it makes searches easier.

 

Can I suggest that you add wildcards to the path config in out_file? J

 

It would be nice to do /directory/%Y/%M/%d/blah.log

 

I’m already using forest to store the logs as hostname.timeslice.log.

 

Thanks!

[Kiyoto Tamura]

Hi Christian,

So...I didn't want to take "no" as an answer and after some experimentation, I found a hack. I want to preface this with: "I've never seen anyone use it this way" disclaimer. I explain how first followed by why it works.

Here is what the configuration looks like:

<match **>
  type file
  path /home/vagrant/file_test/*.access.log
  time_slice_format %Y/%m/%d/
  flush_interval 1s
</match>

<source>
  type forward
</source>

And when you run

 echo '{"hoge":100}' | /usr/lib/fluent/ruby/bin/fluent-cat test

you get this file hierarchy:

.:
2014

./2014:
03

./2014/03:
06

./2014/03/06:
_0.access.log

So, the hierarchy is indeed file_text/2014/03/06/_0.access.log

Why this works

This abuses the fact out_file makes the output file location using Ruby's File#mkdir_p (which is mkdir -p). Because my time_slice_format has "/" in it, it actually creates a directory hierarchy.

pro:
  this works without modifying in_tail or relying on a third party library

con:
  the path still looks kinda ugly (but it is subdiretory per day though!)

  because I've never seen this hack, maybe it won't be supported one day when in_tail gets updated.

Kiyoto

[Christian Hedegaard]

Wow… Haha, that’s awesome! Great hack, but I’m afraid to use it now J

[Kiyoto Tamura]

Probably the correct path forward is to allow placeholders in the path field like this

<match **>

  type file

  path /path/to/%Y/%m/%d/access.log
  ...

</match>

actually, there is some work along this line: https://github.com/tagomoris/fluent-plugin-file-alternative Perhaps @tagomoris can see if it supports a case described above.

Kiyoto

[Sanne de Roever]

I was exactly looking for this. The setup makes file management easier. Did you succeed in doing this?