How can I flush fluentd buffer inside container?

[Paul Harter]

I'm running a project on GKE and fluentd pod is stuck in a bad state. I accidentally flooded it with log entries and it cannot recover.

The pod is up to its memory limit and fluentd is being restarted every 12 seconds. the logs look like this repeatedly:

2017-03-16 03:55:26 +0000 [error]: fluentd main process died unexpectedly. restarting.

2017-03-16 03:55:27 +0000 [warn]: 'block' action stops input process until the buffer full is resolved. Check your pipeline this action is fit or not

2017-03-16 03:55:27 +0000 [warn]: 'block' action stops input process until the buffer full is resolved. Check your pipeline this action is fit or not

2017-03-16 03:55:38 +0000 [error]: fluentd main process died unexpectedly. restarting.

I have tried:

- Deleting the pod.

- Downgrading and upgrading the k8s version on the node.

- Looking in host /var/log/fluentd-buffers for massive files

None of which helped

Is there a way to force the fluentd in the container to flush work again?

It seems like a bug that it can even get into this state.

Thanks

p

[Mr. Fiber]

> The pod is up to its memory limit and fluentd is being restarted every 12 seconds

Does it mean fluentd process is killed by OOM via system?

If the buffer files exist, fluentd is trying to flush existing buffers after restarted.

I'm not sure your configuration but if the problem is memory usage,

there are several approches.

- If you set num_threads is larger than 1, set 1 to avoid parallel flush. It reduces memory usage during buffer flush.

- Use larger memory container

- Set RUBY GC parameter to reduce total memory usage: http://docs.fluentd.org/v0.12/articles/performance-tuning#reduce-memory-usage

For further help, need detailed infromation of the cause.

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.