Fluent bit seems to be ineffective in ignore_Older Configuration Parameter

[Pavan Kumar Nallam]

Fluent bit seems to  be  ineffective in ignore_Older Configuration Parameter, its always shipping old logs when we are rebooting our system.

Can any one please suggest what am i doing wrong ?

The below is my fluent bit startup Log

I am using a DB file to persist the offsets and a parser and ignore older configuration as below 

Fluent Bit v1.7.1
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/02/04 05:58:38] [ info] [engine] started (pid=2685)
[2022/02/04 05:58:38] [ info] [storage] version=1.1.0, initializing...
[2022/02/04 05:58:38] [ info] [storage] in-memory
[2022/02/04 05:58:38] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2022/02/04 05:58:38] [ info] [sp] stream processor started

 cat parsers.conf
[PARSER]
  Name        custom_parser
  Format      json
  Time_Key    timestamp
  Time_Format %Y-%m-%dT%H:%M:%S.%L%z
  Time_Keep    On

Input configuration 

[INPUT]
    Name tail
    Tag  logs_custom
    Read_from_Head false
    Skip_Long_Lines true
    Parser electron_parser
    Ignore_Older 60s
    # Interval Sec
    # ====
    # Read interval (sec) Default: 1
    path somepath
    DB somepath/file_lite.db
    DB.Sync Normal
    # Interval_Sec 1

[Eduardo Silva]

does it happen with the newer version ? 

older versions of ignore older respected the "record timestamp" that depended on a parser, newest versions just take the file date

--
You received this message because you are subscribed to the Google Groups "Fluent Bit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluent-bit+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fluent-bit/d583ab0c-b630-47a4-a140-19d1ce27b6c5n%40googlegroups.com.

--

Eduardo Silva

Calyptia Inc.

https://fluentbit.io

https://twitter.com/edsiper

[Pavan Kumar Nallam]

My fluent bit version was 1.7.1 which has this issue.

can you please suggest which is the stable version which was working with record timestamp ?

Thanks & Regards,
Pavan Kumar Nallam.

[Eduardo Silva]

I don't remember exactly what's the latest one, please check in the release notes:

https://fluentbit.io/announcements/

To view this discussion on the web visit https://groups.google.com/d/msgid/fluent-bit/CAJ4E0-zRXa6QTBLzRkx_u_XAkNasRf-7QUDBO7S2o6Aoo-uXyg%40mail.gmail.com.

[Pavan Kumar Nallam]

I mean if the behaviour is changed to pick the log files rather than individual lines , the logic that we need is not to send older lines again, so I would like to understand which is the stable version which had this functionality.

Thanks & Regards,
Pavan Kumar Nallam.