Size of the emitted data exceeds buffer_chunk_limit

[Maciej Małecki]

Hi there,

I have a problem with chunk sizes between client <-> server. I have a client with fluentd with a list of tail type sources:

<source>
  type tail
  format /^(?<foo>.*)$/
  path ...
  pos_file ...
  tag prefix.foo.suffix
</source> 

 [...]

and forward type match:

<match prefix.**>
  type forward
  buffer_chunk_limit 512k
  buffer_queue_limit 1024 # 1024 * 512kb = 512mb in memory buffer
  flush_interval 1s
  <server>
    host IP
    port ...
  </server>
</match> 

I have also a server with running fluentd:

  <source>
    type forward
    bind IP
    chunk_size_warn_limit 1m
  </source> 

  <match prefix.*.suffix>
    type copy
    <store>
      type mongo
      database ...
      buffer_queue_limit 4096
      flush_interval 1s
      tag_mapped
      capped
      capped_size 500m
      remove_tag_prefix prefix.
    </store>
    <store>
      buffer_type memory
      type per_prefix_file
      path ...
      flush_interval 0s
    </store>
  </match>

I suppose that the client should not send a chunk larger than 512kb (right?). Server receives chunks and stores them in per_prefix_file type and in mongo collection. The problem is that I am getting the following error on the server side:

2015-05-07 22:21:47 +0200 [warn]: Input chunk size is larger than 'chunk_size_warn_limit': tag="prefix.foo.suffix" source="host: ..., addr: ..., port: ..." limit=1048576 size=6346265
2015-05-07 22:21:47 +0200 [warn]: Size of the emitted data exceeds buffer_chunk_limit.
2015-05-07 22:21:47 +0200 [warn]: This may occur problems in the output plugins ``at this server.``
2015-05-07 22:21:47 +0200 [warn]: To avoid problems, set a smaller number to the buffer_chunk_limit
2015-05-07 22:21:47 +0200 [warn]: in the forward output ``at the log forwarding server.``

 

I don't get it. Even if it received a 6.35mb chunk, it should still handle it, because the default chunk size limit is 8mb (from docs). I also got some errors from mongo db that it couldn't save a document > 16mb. I changed chunk size for mongo store to 4mb and it works now, but still I am getting an error about the size of emitted data.

[Mr. Fiber]

Sorry for the delay.

> I suppose that the client should not send a chunk larger than 512kb (right?).

There is one exceptional case that records are too large.

in_tail emits max 1000 events in one read.

If the size of emitted events exceeds the buffer_chunk_limit,

output plugin flushes emitted events immediately.

2015-05-07 22:21:47 +0200 [warn]: Size of the emitted data exceeds buffer_chunk_limit.
2015-05-07 22:21:47 +0200 [warn]: This may occur problems in the output plugins ``at this server.``
2015-05-07 22:21:47 +0200 [warn]: To avoid problems, set a smaller number to the buffer_chunk_limit
2015-05-07 22:21:47 +0200 [warn]: in the forward output ``at the log forwarding server.``

There is no this warning in the client side?

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Maciej Małecki]

Probably there is a warning on the client side, but I have a big number of clients so it is hard to track all warnings there. Is there any possiblity to limit the number of events that tail takes at once from file? How can I solve this issue? On the server side I can see that sometimes it receives chunks >16mb.

[Mr. Fiber]

> Is there any possiblity to limit the number of events that tail takes at once from file? 

Currently, in_tail's 1000 limit is fixed. Hmm...

>  On the server side I can see that sometimes it receives chunks >16mb.

Ah, really? It's weird. Because if in_tail's one call is the cause, the size of one record is over 10k...

If not, could you show me your entire client / server side configuration and log example?

I want to try to reproduce the problem.

Masahiro

[Maciej Małecki]

The biggest chunk from the last 12 hours has ~14mb:

> 2015-05-11 10:16:55 +0200 [warn]: Input chunk size is larger than 'chunk_size_warn_limit': tag="prefix.foo.suffix" source="host: ..., addr: ..., port: ..." limit=1048576 size=14195142

The configuration is pasted in the first post. Just keep in mind that there is more than one in_tail source (mostly 3-6) on the client side (with the same cfg -- differed only in the logfile path). I cannot give a sample log entry, because this is a sensitive data, but logs comes from Ruby/Rails application running on Puma and services such as Sidekiq.

I am wondering if I can set a bigger chunk size on the client side. I am afraid of the network traffic in that case, because I have several hundred clients. What do you think?

[Mr. Fiber]

> The biggest chunk from the last 12 hours has ~14mb:

I see. Could you change following line to smaller size on your client side?

From chunk size warning, you know the host that sends large chunks.

https://github.com/fluent/fluentd/blob/a7cda0103e060c8456f19b50bcddb219c5bbad28/lib/fluent/plugin/in_tail.rb#L472

If resolved the problem by using smaller size, I will add parameter to in_tail like max_emit_records or similar.

> I am wondering if I can set a bigger chunk size on the client side. I am afraid of the network traffic in that case, because I have several hundred clients. What do you think?

Yes. This is trade-off of streaming processing. This is why fluentd provides several buffer parameters.

Some companies uses low latency setup with very small flush_interval.

On the other hand, some companies uses high throughput setup with larger chunks,

Masahiro

[Maciej Małecki]

You mean something like this https://gist.github.com/smt116/8df9127ec98beab8eefd ? I will check that later and let you know if helped

[Maciej Małecki]

Didn't help. I set it to 32 and still client sent a 22mb chunk to the server.

[Mr. Fiber]

The code is incorrect.

Could you use following code?

module Fluent
  class LimitedTail < NewTailInput
    Plugin.register_input('limited_tail', self)

    class NewTailInput::TailWatcher
      MAX_LINES_AT_ONCE = 256 # 1000 is by default
    end
  end
end

MAX_LINES_AT_ONCE is in TailWatcher class, not under NewTailInput.

Masahiro

[Maciej Małecki]

Good point, sorry ;). I have set MAX_LINES_AT_ONCE to 32, but still getting chunks >10mb. Anyway looks better now, because the largest chunk from last few hours have 14mb instead of 38mb. Probably there are some events that are really big.

Okay but the server will reject too big chunks after a while and this will not block new ones, right?

[Mr. Fiber]

I wrote a patch: https://github.com/fluent/fluentd/pull/593

> Okay but the server will reject too big chunks after a while and this will not block new ones, right?

fluentd doesn't reject chunks first because it causes data lost.

So above messages are warn, not error or fatal.

fluentd tries to flush chunks to primary output.

If the problem happens, fluentd uses secondary to backup it.

If there is no secondary, chunks are thrown away.

Masahiro

[Maciej Małecki]

Sorry for the late response. Thanks for the patch.

[Mr. Fiber]

Just released v0.12.9 including read_lines_limit.

[Yasin Amadmia]

Hi,

We having same issue and wanted to understand this bit more.

So is the buffer_chunck_limit enforced by output plugin always ?

What if say you have tail_line_limits to say 10 and these 10 lines total count 10m and your buffer_chunck_limit is 5m ? Will it be broken down into 2 x 5m chunks or flushed as 1 x 10m ? If later than there is no way buffer_chunck_limit is enforceable I suppose ?

Thanks,

Yasin.

[Mr. Fiber]

> Will it be broken down into 2 x 5m chunks

fluentd v0.14 will do this similar behaviour.

>  flushed as 1 x 10m 

fluentd v0.12 does this approach.

> If later than there is no way buffer_chunck_limit is enforceable I suppose ?

Yes in v0.12. You should send smaller chunk from forwarder or input side.

This e-mail and any attachments are confidential and intended solely for the use of the addressee only. If you have received this message in error, you must not copy, distribute or disclose the contents; please notify the sender immediately and delete the message.

This message is attributed to the sender and may not necessarily reflect the view of Travis Perkins Trading Company Limited, 733503, Lodge Way House, Lodge Way, Harlestone Road, Northampton, NN5 7UG or its parent company Travis Perkins plc (Registered in England No. 824821; Lodge Way House, Lodge Way, Harlestone Road, Northampton, NN5 7UG; VAT number 408556737) and any of its subsidiaries. Agreements binding Travis Perkins Trading Company Limited may not be concluded by means of e-mail communication.

E-mail transmissions are not secure and Travis Perkins Trading accepts no responsibility for changes made to this message after it was sent. Whilst steps have been taken to ensure that this message is virus free, Travis Perkins Trading accepts no liability for infection and recommends that you scan this e-mail and any attachments.

--

You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

[Yasin Amadmia]

Thanks. 

We have td-agent  2.3.2-0. So I guess it will enforce buffer_chunck_limit ? i.e. send 2 x 5m.

[Mr. Fiber]

> I guess it will enforce buffer_chunck_limit ? i.e. send 2 x 5m.

No. td-agent 2.3.x uses fluentd v0.12.

See: https://docs.treasuredata.com/articles/td-agent-changelog#td-agent2-current-stable

Masahiro

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

[Yasin Amadmia]

Thanks once again. Anyway we could get td-agent version that uses fluentd v0.14 ? Perhaps a development release of td-agent ?

Secondly, in our configuration, we set fluentd_disable_retry_limit = true to avoid loosing/dropping any data. However, this causes us issues as above. If due to any reason, we get a chunk that is larger in size, the chunk will be looping through indefinitely, not letting other smaller chunks go out either as it will be blocking the output plugin. What we want is that after so many tries, the chunk should be written to a file or use some other output plugin instead of it getting trashed. Would this be possible ?

If the above is not possible, will fluentd log (under td-agent.log) before trashing the chunk so at-least we know it has trashed a particular chunk ?

[Mr. Fiber]

> Anyway we could get td-agent version that uses fluentd v0.14 ? Perhaps a development release of td-agent ?

We don't have a release plan for now.

See this thread: https://groups.google.com/forum/#!topic/fluentd/QQM1sPAH0Do

> we set fluentd_disable_retry_limit = true to avoid loosing/dropping any data.

> the chunk should be written to a file or use some other output plugin instead of it getting trashed. Would this be possible ?

Use secondary instead of disable_retry_limit.

See official article: http://docs.fluentd.org/articles/buffer-plugin-overview#secondary-output

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

[Yasin Amadmia]

Thanks.

Will it be file format or the actual format of the plugin that is failing ?

We have something like

<match **.elasticsearch>

    type "aws-elasticsearch-service"

    buffer_type file

    buffer_path /var/log/td-agent/buffer/elasticsearch

    buffer_queue_limit 512m

    disable_retry_limit false

    retry_limit 5

    buffer_chunk_limit 5m

    log_level debug

    retry_wait 10s

    max_retry_wait 120s

    include_tag_key true

    tag_key tag

    logstash_format true

    flush_interval 10s

    <endpoint>

      url https://xxx.amazonaws.com

      region eu-west-1

    </endpoint>

    <secondary>

      type file

      path /var/log/td-agent/fluent-plugin-aws-elasticsearch-service.failed

    </secondary>

</match>

This works with a caveat. After 5 retry limits, it goes to secondary. However, the secondary file's data is not human readable. I suppose it is dumping it in the format as elasticsearch would expect which doesn't make it that useful. Anyway we could get around it ?

[Mr. Fiber]

> However, the secondary file's data is not human readable

The content of seconday depends on primary plugin's format.

If you use elasticsearch plugin, its content is msgpack.

If you use webhdfs or s3 plugin, its content is `format` parameter.

If you want to re-send backuped data to elasticsearch,

you can read data using msgpack library in your language.

Masahiro

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.