How to flush files buffer

[Franck]

Dear all,

I have a fluentd server connected via a plugin to my database. Yesterday my DB server crashed and since then my fluentd server buffers the received message into files on its filesystem. 

How do I flush these files ? kill -SIGUSR1 on the process does nothing and I can't (obviously) loose data in these files.

Thanks in advance

Franck

[Kiyoto Tamura]

Hi Franck-

I am assuming that you have already restarted your DB servers but NOT Fluentd. Which output plugin are you using and what does your logs say?

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Check out Fluentd, the open source data collector to unify log management.

[Franck]

Dear Kiyoto

I wrote my own plugin bufferedOutput plugin based on one I found in the available plugins. 

I have nothing in my log file. This morning I have an error message saying "2014-09-11 07:22:20 +0200: forward error: queue size exceeds limit" so I stopped my fluentd server

Franck

[Kiyoto Tamura]

I see. That message means that the buffer queue has gotten larger than its limit. I have a couple of questions:

1. Does your output plugin handle any connection failure? In other words, what's the expected behavior if the DB gets restarted? If it does not re-establish the connection, then it get get stuck as you described in the thread. You should handle connection failure in the plugin code.

2. As for recovery:

i) Restart the DB

ii) Restart Fluentd

iii) Force the buffer to flush with the -USR1 signal. If you are using td-agent, it would be

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kiyoto Tamura]

Sorry I sent that before finishing it off. Let me repeat the recommended course of action.

i) Restart the DB

ii) Restart Fluentd

iii) Force the buffer to flush with the -USR1 signal. If you are using td-agent, it would be

kill -USR1 `cat /var/run/td-agent/td-agent.pid`

and if you are running Fluentd alone,

kill -USR1 <your_fluentd_supervisor_process_id>

Kiyoto