Will fluent-bit break data in between to keep track of 2MB chunk size for tail plugin

[Chiranjit Datta]

Hi All,

Fluent-bit tail plugin internally chunks incoming data on 2MB size. My query is will fluent-bit take care of "not-to-break" data in between to keep track of that 2MB size? 

I have tested with couple of data, and I have observed if for one character it cross the size limit, fluent-bit will put that line into the next chunk. 

I just want to clarify whether that is the expected behaviour from fluent-bit in all cases?

[Eduardo Silva]

replied in: https://github.com/fluent/fluent-bit/issues/345

1. it is like that will it be possible to variablize the value based on requirement?

That's a hard coded value. If you have a special use case where you need to change this let me know to understand the problem you are trying to fix.

2. Will fluent-bit take care of "not-to-break" data in between to keep track of that 2MB size?

Data will always be split internally in chunks and that chunk is immutable until it reach the output plugin who likely will generate a new modified copy (e.g: elasticsearch requires special JSON format instead of Fluent Bit chunk data representation).

In general when the input plugins "insert" data into the engine, the engine will create it own chunks.

--
You received this message because you are subscribed to the Google Groups "Fluent-Bit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluent-bit+unsubscribe@googlegroups.com.
To post to this group, send email to fluen...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fluent-bit/dc1b19ca-37b6-4d1f-8265-2c165028da81%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Eduardo Silva

Open Source, Treasure Data

http://www.treasuredata.com/opensource