Collect logs from multiple files in a directory using type tail

[Dushyant Dixit]

Can someone help me in collecting logs from multiple file in a directory.

eg.

 

<source>

  type tail

  path /var/log/containers/*.log

  pos_file /var/run/td-agent/pos_files/test.pos

  tag test.*

  read_from_head true

</source> 

Each files in 'path' must get collected on remote with file name as <file_name>.log

PS: I am able to collect logs from all log files to a single file on remote server but not able to collect logs individually like <file_name>.log 

Please help

[Mr. Fiber]

It is hard with fluentd v0.12.

In v0.14 master, master means not relased yet, it can with following configuration.

<source>
  @type tail
  path in/*.log
  path_key tailed_path
  read_from_head true
  tag logs.tail
  <parse>
    @type none
  </parse>
</source>

<filter logs.tail>
  @type record_transformer
  enable_ruby true
  <record>
    # overwrite full path to file name
    tailed_path ${File.basename(record["tailed_path"])}
  </record>
</filter>

<match logs.tail>
  @type file
  path out/${tailed_path}
  append true
  add_path_suffix false

  <buffer time,tailed_path>
    flush_at_shutdown true # for test
  </buffer>
  <format>
    @type single_value
  </format>
</match>

$  ls in/
test.log

% ls out            
${tailed_path}/    test.log.20161114

timestamp is added to seperate files.

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.