fluent-bit and multiple files in a directory?

[Christopher Robbins]

I am trying to use fluent-bit in an AWS EKS deployment for monitoring several Magento containers.  I have three input configs that I have deployed, as shown below.  I'm noticing the following when using the tail plugin and testing (focused on  input_magento_report_api.conf and input_magento_report.conf).  These two input plugins monitor for files created with a long numeric string; the log files have no extension.  

* When logfiles are written by Magento, the logs aren't detected by the tail plugin

* When I manually create a log using echo (i.e. - echo "This is a log" > 1234567891234), the log is detected and forwarded to the Elasticsearch cluster properly

* If I try creating the file first (i.e. - touch 1234567891234), then writing to it, this fails to be detected by the tail plugin - I'm guessing this behavior matches what Magento is doing

Is this expected behavior?  Is there a better way to monitor many logs in a single directory?

Config(s)

  fluent-bit.conf:

    [SERVICE]

      Flush          1

      Daemon         Off

      Log_Level      info

      Parsers_File   parsers.conf

    @INCLUDE input_magento_logs.conf

    @INCLUDE input_magento_report_api.conf

    @INCLUDE input_magento_report.conf

    @INCLUDE output_elasticsearch.conf

  input_magento_logs.conf

    [INPUT]

      Name           tail

      Tag            magento_logs

      Path           /magento/var/log/*.log

      Parser         magento

      DB             /var/log/magento.db

      Mem_Buf_Limit  5MB

      Path_Key       On

  input_magento_report_api.conf

    [INPUT]

      Name           tail

      Tag            magento_api_report_logs

      Path           /magento/var/report/api/*

      Parser         magento_report

      DB             /var/log/magento_report_api.db

      Mem_Buf_Limit  5MB

      Path_Key       On

  input_magento_report.conf

    [INPUT]

      Name           tail

      Tag            magento_report_logs

      Path           /magento/var/report/*

      Parser         magento_report

      DB             /var/log/magento_report.db

      Mem_Buf_Limit  5MB

      Path_Key       On

  output_elasticsearch.conf

    [OUTPUT]

      Name            es

      Match           *

      Host            ${FLUENT_ELASTICSEARCH_HOST}

      Port            ${FLUENT_ELASTICSEARCH_PORT}

      Index           ${FLUENT_ELASTICSEARCH_NAMESPACE}

      Include_Tag_Key On

      Tag_Key         tag

  parsers.conf

    [PARSER]

      Name   magento

      Format regex

      Regex (?<Time>[^ ]* [^ ]*) (?<Level>[^ ]*):(?<Log>[^\n]+)\n*

      Time_Key Time

      Time_Format [%Y-%m-%d %H:%M:%S]

    [PARSER]

      Name   magento_report

      Format regex

      Regex  ^".*"

[Eduardo Silva]

Hi Christopher, 

I cannot reproduce the problem, a few questions:

- are you using Fluent Bit v1.0.6 ?

- what about if you add "Refresh_Interval 5" to your Tail input section ? (which means: scan for new files every 5 seconds)

regards, 

--
You received this message because you are subscribed to the Google Groups "Fluent-Bit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluent-bit+...@googlegroups.com.
To post to this group, send email to fluen...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fluent-bit/2d3462b7-0ed4-4f00-ae5d-5a1d85dce853%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Eduardo Silva
Principal Engineer  | Arm
. . . . . . . . . . . . . . . . . . . . . . . . . . . 
m. +506 70138007
Arm.com
Treasuredata.com

   

[Christopher Robbins]

Eduardo - 

I have tried using the latest fluent-bit version and "Refresh_Interval 5" with no change.  

--

Christopher P Robbins
P: 480.336.3043

C: 928.257.6368
Email: robb...@gmail.com