In_tail input not working

[Marek]

Hello everyone,
I am trying to setup fluentd. I followed installation guide and manual http input with debug messages works for me. I tried dummy messages and those work too. Problem is when I try very simple config to tail log file I simply can't get it to work. This is my configuration:

<source>

  type tail

  path /var/log/fluentd-debug/log_test.log

  pos_file /var/log/td-agent/log_test.log.pos

  refresh_interval 10s

  tag tail.test

#  format /^(?<data>[*])$/

  format none

</source>

<match tail.test>

  type file

  path /var/log/td-agent/tail

</match>

Log file contents:

cat /var/log/fluentd-debug/log_test.log

"Test_message"

Files from previous examples I mentioned are present but nothing concerning tail. 

ls /var/log/td-agent/

buffer  dbg.20160219.b52c25c0cf09658f3  dummy.20160217.b52bfa63ec9184bee  log_test.log.pos  td-agent.log  td-agent.log-20160218.gz  td-agent.log-20160219

Because systemd is running on server I use
systemctl start|stop|restart td-agent

I omitted rest of configuration file which consists of original config later in file. Any help is appreciated as I can't find any other help or solution to this problem.
Marek

[Mr. Fiber]

How about setting "read_from_head true" ?

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Marek]

Thanks for your reply!

I tried it. Here is td-agent.log since last restart:

2016-02-19 22:57:11 +0000 [info]: reading config file path="/etc/td-agent/td-agent.conf"

2016-02-19 22:57:11 +0000 [info]: starting fluentd-0.12.20

2016-02-19 22:57:11 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.3.1'

2016-02-19 22:57:11 +0000 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'

2016-02-19 22:57:11 +0000 [info]: gem 'fluent-plugin-mongo' version '0.7.12'

2016-02-19 22:57:11 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.4'

2016-02-19 22:57:11 +0000 [info]: gem 'fluent-plugin-s3' version '0.6.5'

2016-02-19 22:57:11 +0000 [info]: gem 'fluent-plugin-scribe' version '0.10.14'

2016-02-19 22:57:11 +0000 [info]: gem 'fluent-plugin-td' version '0.10.28'

2016-02-19 22:57:11 +0000 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.2'

2016-02-19 22:57:11 +0000 [info]: gem 'fluent-plugin-webhdfs' version '0.4.1'

2016-02-19 22:57:11 +0000 [info]: gem 'fluentd' version '0.12.20'

2016-02-19 22:57:11 +0000 [info]: adding match pattern="tail.test" type="file"

2016-02-19 22:57:11 +0000 [info]: adding match pattern="td.*.*" type="tdlog"

2016-02-19 22:57:11 +0000 [info]: adding match pattern="debug.file" type="file"

2016-02-19 22:57:11 +0000 [info]: adding match pattern="debug.**" type="stdout"

2016-02-19 22:57:11 +0000 [info]: adding source type="tail"

2016-02-19 22:57:11 +0000 [info]: adding source type="forward"

2016-02-19 22:57:11 +0000 [info]: adding source type="http"

2016-02-19 22:57:11 +0000 [info]: adding source type="debug_agent"

2016-02-19 22:57:11 +0000 [info]: using configuration file: <ROOT>

  <source>

    type tail

    path /var/log/fluentd-debug/log_test.log

    pos_file /var/log/td-agent/log_test.log.pos

    refresh_interval 10s

    tag tail.test

    read_from_head true

    format none

  </source>

  <match tail.test>

    type file

    path /var/log/td-agent/tail

    buffer_path /var/log/td-agent/tail.*

  </match>

  <match td.*.*>

    type tdlog

    apikey xxxxxx

    auto_create_table 

    buffer_type file

    buffer_path /var/log/td-agent/buffer/td

    <secondary>

      type file

      path /var/log/td-agent/failed_records

      buffer_path /var/log/td-agent/failed_records.*

    </secondary>

  </match>

  <match debug.file>

    type file

    path /var/log/td-agent/dbg

    buffer_path /var/log/td-agent/dbg.*

  </match>

  <match debug.**>

    type stdout

  </match>

  <source>

    type forward

  </source>

  <source>

    type http

    port 8972

  </source>

  <source>

    type debug_agent

    bind 127.0.0.1

    port 24230

  </source>

</ROOT>

2016-02-19 22:57:11 +0000 [info]: following tail of /var/log/fluentd-debug/log_test.log

2016-02-19 22:57:11 +0000 [info]: listening fluent socket on 0.0.0.0:24224

2016-02-19 22:57:11 +0000 [info]: listening dRuby uri="druby://127.0.0.1:24230" object="Engine"

Sadly I am still getting nothing through:

ls

buffer  dbg.20160219.b52c25c0cf09658f3  dummy.20160217.b52bfa63ec9184bee  log_test.log.pos  td-agent.log  td-agent.log-20160218.gz  td-agent.log-20160219

[Mr. Fiber]

Did you remove pos_file before set `read_from_head true` ?

[Marek]

I admit I did not. Removing position file and restarting service helped. I can see my simple message now, though I don't understand why start from head should be implemented. My issue was solved, though if you could provide extra commentary why/how that would be great for me!
Anyway, many thanks.

On Friday, 19 February 2016 23:36:01 UTC+1, Marek wrote: