Having trouble with sending high volume topics to pubsub using fluentd plugin

[Fere]

Hi,

The plugin works find with low volume topics, but for the high volume its having trouble giving error:

    unexpected error error="The value for message_count is too large. You passed 1001 in the request, but the maximum value is 1000.

I changed the config set the following params:

    buffer_chunk_limit 2k

    flush_interval 1

Even though the buffer chunk limit is 2k, I still see the buffered files having much larger sizes, as if the chunk limit is not having any impact.

I now get the following errors for high volume topics:

      2017-02-04 21:38:08 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-02-04 21:38:03 +0000 error_class="Gcloud::Pubsub::ApiError" error="The value for message_count is too large. You passed 6915 in the request, but the maximum value is 1000." plugin_id="object:3fbb05554ad8"

      2017-02-04 21:38:08 +0000 [warn]: suppressed same stacktrace

      2017-02-04 21:38:09 +0000 [warn]: Size of the emitted data exceeds buffer_chunk_limit.

      2017-02-04 21:38:09 +0000 [warn]: This may occur problems in the output plugins ``at this server.``

      2017-02-04 21:38:09 +0000 [warn]: To avoid problems, set a smaller number to the buffer_chunk_limit

      2017-02-04 21:38:09 +0000 [warn]: in the forward output ``at the log forwarding server.``

In the output forwarder document I don't see any reference to chunk limit param.  And I am not sure how I can enforce smaller chunks to be set to pubsub.  

Any help is appreciated.

Fere

[Fere]

Anyone, any idea how to make the buffer_chunk_limit actually function with the pubsub plugin and keep the chunks within the specified size?

[Mr. Fiber]

> 2017-02-04 21:38:09 +0000 [warn]: Size of the emitted data exceeds buffer_chunk_limit.

> 2017-02-04 21:38:09 +0000 [warn]: This may occur problems in the output plugins ``at this server.``

> 2017-02-04 21:38:09 +0000 [warn]: To avoid problems, set a smaller number to the buffer_chunk_limit

> 2017-02-04 21:38:09 +0000 [warn]: in the forward output ``at the log forwarding server.``

I don't know the your data pipeline but this log saids input plugin sends larger chunks to pubsub plugin.

What the input plugin do you use?

if in_forward, forwarder's out_forward should be set smaller buffer_chunk_limit.

If other input plugin, set smaller parameter to emit releated config.

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.