some problems when i use tail

[cms coo]

Hi,

I used 2 fluentd.

<source>

@type forward

port 24224

bind 0.0.0.0

</source>

<filter **>

@type stdout

</filter>

<match mix.log>

  @type copy

  <store>

        @type file

        path /var/log/fluent/${tag}/access.log

        #  compress gzip

        <buffer tag>

                timekey 1m

                #timekey_use_utc true

                timekey_wait 1m

        </buffer>

        <format>

                @type json

        </format>

  </store>

</match>

<match **>

@type stdout

</match>

This is My first flentd,  It receives json data, and save them in files.It works well.

<source>

  @type tail

  path /var/log/fluent/mix.log/*.*

  pos_file /var/log/fluent/pos.log

  tag mix.log

<parse>

  @type json

</parse>

</source>

<filter **>

@type stdout

</filter>

<match mix.log>

    @type kafka2

    brokers 39.108.95.xxx:9092

    use_event_time false

    # data type settings

    <format>

    @type json

    </format>

    topic_key fluent3

    default_topic fluent3

    get_kafka_client_log true

    # producer settings

    required_acks -1

    compression_codec gzip

</match>

<match **>

@type stdout

</match>

This is my second fluentd service, I hope it tail the files generated by the first fluent, and push data to kafka.

Below is the abnormal situation I encountered :

This is the data in file(PATH: /var/log/fluent/mix.log/access.log_0.log)  

``` 

{"A":0,"B":0}

{"A":1,"B":1}

{"A":2,"B":2}

{"A":3,"B":3}

{"A":4,"B":4}

{"A":5,"B":5}

{"A":6,"B":6}

{"A":7,"B":7}

{"A":8,"B":8}

{"A":9,"B":9}

```

When the file is generated,the second fluentd tell me :

2021-03-31 09:50:46.446632200 +0800 fluent.info: {"message":"detected rotation of /var/log/fluent/mix.log/access.log_0.log; waiting 5 seconds"}

2021-03-31 09:50:46 +0800 [info]: #0 fluent/log.rb:329:info: detected rotation of /var/log/fluent/mix.log/access.log_0.log; waiting 5 seconds

2021-03-31 09:50:46.446632200 +0800 fluent.info: {"message":"detected rotation of /var/log/fluent/mix.log/access.log_0.log; waiting 5 seconds"}

2021-03-31 09:50:46 +0800 [info]: #0 fluent/log.rb:329:info: following tail of /var/log/fluent/mix.log/access.log_0.log

2021-03-31 09:50:46.448034400 +0800 fluent.info: {"message":"following tail of /var/log/fluent/mix.log/access.log_0.log"}

2021-03-31 09:50:46.448034400 +0800 fluent.info: {"message":"following tail of /var/log/fluent/mix.log/access.log_0.log"}

But the second fluentd not read the file(access.log_0.log).Of course,don't push data to kafka.

However,  when I manually modify the file(add one line at the end of the file),  the second fluent works,and tail all data in the file, and push them to kafka successfully.

One more thing,  when I add a new line again, 2nd flunt tail all data,Instead of the line I just add.

I am confused,. Did I miss some configs? please help !  I'm sorry for my bad English

[cosmo09...@gmail.com]

Hi,

Did you use Fluentd v1.12.0 or 1.12.1?

These versions contain some feature breakage of tail plugin.

Could you update Fluentd to v1.12.2 and try your configuration again?

Best,

Hiroshi

2021年3月31日水曜日 11:18:21 UTC+9 cooc...@gmail.com:

[cms coo]

YES I use   Fluentd  v1.12.1,

I will try the new version immediately ! Thanks

[cms coo]

Hi,

I Updated to v1.12.2,   This problem still exists.

In my previous description, there is a description that needs to be corrected

When the file is generated,the second fluentd tell me only this:(Without the first three lines mentioned before) 

```

2021-03-31 09:50:46 +0800 [info]: #0 fluent/log.rb:329:info: following tail of /var/log/fluent/mix.log/access.log_0.log

2021-03-31 09:50:46.448034400 +0800 fluent.info: {"message":"following tail of /var/log/fluent/mix.log/access.log_0.log"}

2021-03-31 09:50:46.448034400 +0800 fluent.info: {"message":"following tail of /var/log/fluent/mix.log/access.log_0.log"}

```

[cosmo09...@gmail.com]

Hi,

Your scenario seems to be a bit messed, but when using Fluentd v1.11.5 still exists your issue?

If yes, Fluentd losts logs on detecting log rotating.

If no, Fluentd v1.12.2 causes some unknown edge issues.

Thanks,

Hiroshi

2021年3月31日水曜日 14:44:50 UTC+9 cooc...@gmail.com:

[Suvelee Sarpotdar]

Hello,

We are trying to implement the same scenario and we followed this stackoverflow suggestion - https://stackoverflow.com/questions/53960655/fluentd-one-source-for-several-filters-and-matches

This solution worked for us. We are using a dockerized version of Fluentd and run only one container. You declare only one <source> tag with tail plugin details and then you copy the event and relabel it. Then you can handle each label as per your needs. 

Hope this helps

Thanks,

Suvelee

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fluentd/f4a6fe65-ecae-4996-8db8-8edf16057bc0n%40googlegroups.com.

[Anirudh Venkatesh]

With pos file, use read_from_head true. Fluentd tries to read from tail of the file by default. Also since you are using a wildcard match in the path, use follow_inodes true to avoid duplicates with log rotation.

Hope this helps.

Thanks,

Anirudh