How to count daily flow via time header?

[Yasuyuki ENDO]

I want to count my daily flow via "time" header's value.

But fluent-plugin-flowcounter,fluent-plugin-datacounter dose not read "time" header.

How do I count *real* daily flow?

[Satoshi Tagomori]

AFAIK, these's no way to do it.

IMO, counting flow via data (ex: time field) in fluentd is almost nonsense.

If your fluentd cluster delivers data with just little delay, counting via time arrived works well.

If your data have long delay from real time, you should count these data on any batch processing

systems, like hadoop or something else.

2015年9月29日火曜日 14時35分40秒 UTC+9 Yasuyuki ENDO:

[Yasuyuki ENDO]

1. I use fluent-plugin-google-bigquery. It create BigQuery table each day from "time" header.

2. I want to compare BQ table count and fluentd daily flow count.

If your data have long delay from real time, you should count these data on any batch processing
systems, like hadoop or something else.

OK, I will store daily flow to local storage, and count it.

2015年9月29日火曜日 15時40分02秒 UTC+9 Satoshi Tagomori:

[Satoshi Tagomori]

Of course, you can write any plugins to do such thing.

I think that such that plugin will be written in just tens of lines.

Output delay is most important factor, because if your data arrives with 3 hours delay,

you MUST compute events per day at 3am everyday.

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

田籠 聡 (TAGOMORI Satoshi)
Mail: tago...@gmail.com