[error] [input chunk] error writing data from tail

[Mike Nikitas]

Hi to all,

I am using OPS Agent 2.9.0 (google cloud monitoring) which contains Fluent Bit v1.8.11 and at random times the logging-module.log keeps growing until no more space left in the server (CentOS Linux 7  with Kernel: Linux 3.10.0-514.10.2.el7.x86_64). logging-module.log contains the following entries:

[2022/02/08 08:49:19] [error] [input chunk] error writing data from tail.2 instance
[2022/02/08 08:49:19] [error] [storage] format check failed: tail.2/16046-1644299393.484592389.flb
[2022/02/08 08:49:19] [error] [storage] format check failed: tail.2/16046-1644299393.484592389.flb
[2022/02/08 08:49:19] [error] [storage] [cio file] file is not mmap()ed: tail.2:16046-1644299393.484592389.flb
[2022/02/08 08:49:19] [error] [input chunk] error writing data from tail.2 instance
[2022/02/08 08:49:19] [error] [storage] format check failed: tail.2/16046-1644299393.484592389.flb
[2022/02/08 08:49:19] [error] [storage] format check failed: tail.2/16046-1644299393.484592389.flb
[2022/02/08 08:49:19] [error] [storage] [cio file] file is not mmap()ed: tail.2:16046-1644299393.484592389.flb

Any ideas?

My issue is critical because Fluent Bit causes our server to stop working when there is no more space available.

Regards,

Mike

[Eduardo Silva]

Hi Mike, 

Can you please attach your full config file and log file? Ideally to republish on GitHub as an issue would be ideal for a more clear troubleshooting:

https://github.com/fluent/fluent-bit

Thanks 

--
You received this message because you are subscribed to the Google Groups "Fluent Bit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluent-bit+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fluent-bit/a131c0c5-044e-46b4-a0c2-f8bb191be95dn%40googlegroups.com.

[Mike Nikitas]

The full log is enormous, 536gb containing similar content.

Configuration is:

# Master configuration file for google-fluentd

# Include any configuration files in the config.d directory.
#
# An example "catch-all" configuration can be found at
# https://github.com/GoogleCloudPlatform/fluentd-catch-all-config
@include config.d/*.conf

# Do not collect fluentd's own logs to avoid infinite loops.
<match fluent.**>
  type null
</match>

# Configure all sources to output to Google Cloud Logging
<match **>
  type google_cloud
  # Set the chunk limit conservatively to avoid exceeding the limit
  # of 10MB per write request.
  buffer_chunk_limit 2M
  flush_interval 5s
  # Never wait longer than 5 minutes between retries.
  max_retry_wait 300
  # Disable the limit on the number of retries (retry forever).
  disable_retry_limit
  # Use multiple threads for processing.
  num_threads 8
</match>

[Eduardo Silva]

that config belongs to Fluentd project, not Fluent Bit

--

Eduardo Silva
https://edsiper.com

Twitter: @edsiper
OSS: https://fluentbit.io