Is there a recommended way of monitoring fluent-logger approaching buffer limit?

[Jae Lee]

Hi all,

Is there a recommended way of monitoring fluent-logger approaching buffer limit?

I am planning to monitor td-agent instead and use monitor_agent to see the current queue size. However I suspect that wouldn't give me the full picture.

Any idea?

I could always use logger that send an alarm upon errors. Then by looking at the error message I can tell whether there was a data loss or not. Although I would have hoped there's a way to see the rate of increase of @pending.bytesize relative to @limit just to give that full context which might be useful to decide what to do.

How everyone else is monitoring fluentd in general?

regards

J

[Jae Lee]

From second thought, I think I might have a look at in_tail plugin instead, given support of pos_file it sounds like it will have less chance of losing data.

any thought?

J

[Masahiro Nakagawa]

Hi,

> How everyone else is monitoring fluentd in general?

Popular approach is to use existence monitoring system with in_monitor_agent, e.g. zabbix, munin, etc.

http://docs.fluentd.org/articles/monitoring

In addition, some users monitor record traffic for more robust operation. See this reply:

https://groups.google.com/forum/?fromgroups#!searchin/fluentd/monitoring/fluentd/A6qDNzykEk4/BlzaNzT07qoJ

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Masahiro Nakagawa]

> I think I might have a look at in_tail plugin instead, given support of pos_file it sounds like it will have less chance of losing data.

Does this mean 'application writes events to local file first. After that, fluentd tails events from local file' ?

If so, Pixiv uses same approach too. They writes json to local file and fluentd processes its file.

--

[Jae Lee]

yes my understanding is when local fluentd daemon has crashed, fluentd-logger will queue up the message internally upto a limit (default to 8M) then drop them?

I was wondering if monitoring the fluentd-logger log for any droppage is the only way to monitor that leg of the journey (i.e. journey between application to fluentd daemon).

Regarding using in_tail plugin, is so that I can reduce chance of message dropping. Although I would need to test how log rotation and modification affects in_tail plugin. I briefly saw its implementation that seems to handle file rotation with inode and pos_file etc.

J

[Masahiro Nakagawa]

> fluentd-logger will queue up the message internally upto a limit (default to 8M) then drop them?

It depends on logger implementation. Yes for Ruby, No for Java.

> I briefly saw its implementation that seems to handle file rotation with inode and pos_file etc.

That's right. And you can use 'rotate_wait' parameter to wait delayed record.

http://docs.fluentd.org/articles/in_tail