incoming data exceed limit

[s...@getpolymorph.com]

Hey Guys,

I'm considering using fluent-bit as a side car in a kubernetes app and benchmarking things with about 10k requests per second across about 10 pods. For some reason keep getting the following error. If i filter to stdout i can see that a handful of logs make it through but the service is being rejected a majority of the time. I've tried increasing Chunk_Size and Buffer_Size but it doesn't seem to have any affect. Is there some other config value i should use?

Its running as a side car to the logging service so each docker image only deals with total traffic of about 500 to 1000 requests per second.

[2018/04/17 23:25:22] [ warn] [in_fw] fd=56 incoming data exceed limit (32768 bytes)
[2018/04/17 23:25:22] [ warn] [in_fw] fd=55 incoming data exceed limit (32768 bytes)

fluent-bit config

[SERVICE]
    Flush         1
    Daemon        Off
    Log_Level     info
    Parsers_File  parsers.conf


[INPUT]
    Name        forward
    Listen      0.0.0.0
    Port        24224
    Chunk_Size  32
    Buffer_Size 64

[OUTPUT]
    Name             file
    Path             /dev/null

[s...@getpolymorph.com]

Oh, i should also mention that i'm using the komitsu/fluentd cilent https://github.com/fluent/fluentd

[Eduardo Silva]

hi Sam, 

if you are using in_forward or any plugin that behave as a server you have to setup your buffering limits for it. If the incoming data exceed the limits (which means incomplete data) the data will be dropped. For more details please check the options Buffer_Max_Size and Buffer_Chunk_Size  in the plugin documentation:

https://fluentbit.io/documentation/0.12/input/forward.html

let me know how it goes, 

regards, 

--
You received this message because you are subscribed to the Google Groups "Fluent-Bit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluent-bit+unsubscribe@googlegroups.com.
To post to this group, send email to fluen...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/fluent-bit/adeb08e5-f2a7-4d57-9c3e-acec937f4049%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Eduardo Silva

Open Source, Treasure Data

http://www.treasuredata.com/opensource