output bucketing based on event category when centralizing scribe messages
11 views
Skip to first unread message
Vlad
Dec 20, 2010, 3:51:47 PM12/20/10
to Flume Users
Hello,
I have been playing the last couple of days with different
configuration of flume in an attempt to better understand how it
works. Currently i have configured a simple node that intercepts and
centralizes data coming from several scribe servers. I can see that
the category information supplied by scribe is being appended to the
meta-data of every event and I'm trying to figure out now how to
control the collectors node output based on this information and log
events to different files based on this criteria. The user guide has a
reference witch makes me think this is possible "When collecting data
from scribe sources, you want to use its bucket data based on its the
event’s category information" but i have not yet found the way to do
this. Any information that might put me on the right track with this
will be highly appreciated.
Thank you
I have been playing the last couple of days with different
configuration of flume in an attempt to better understand how it
works. Currently i have configured a simple node that intercepts and
centralizes data coming from several scribe servers. I can see that
the category information supplied by scribe is being appended to the
meta-data of every event and I'm trying to figure out now how to
control the collectors node output based on this information and log
events to different files based on this criteria. The user guide has a
reference witch makes me think this is possible "When collecting data
from scribe sources, you want to use its bucket data based on its the
event’s category information" but i have not yet found the way to do
this. Any information that might put me on the right track with this
will be highly appreciated.
Thank you
Vlad
Dec 21, 2010, 7:33:07 AM12/21/10
to Flume Users
I have figured this one out. Using something like this
exec config 'scribe-test' 'scribe(1464)' 'collectorSink("file:///mnt/
logs/flume/scribe/%{scribe.category}/", "scribe-%H%M-")'
did the trick.
Maybe the documentation from output bucketing should point that fields
from event meta-data can also be used to
control the output.
Bye
exec config 'scribe-test' 'scribe(1464)' 'collectorSink("file:///mnt/
logs/flume/scribe/%{scribe.category}/", "scribe-%H%M-")'
did the trick.
Maybe the documentation from output bucketing should point that fields
from event meta-data can also be used to
control the output.
Bye
Jonathan Hsieh
Dec 21, 2010, 11:25:15 AM12/21/10
to Vlad, Flume Users
Vlad,
Glad to see you figured it out. Admittedly, a lot of these attributes are not documented.
I think one thing we may do, is to start using keyword arguments to enable users to set attribute names on their own, and use the "old names" as defaults. This would help readability by making attributes explicitly defined at the same level they are used at in the configuration language.
It would look something like:
scribe(1464, categoryAttr="scribe.category")
Jon.
Reply all
Reply to author
Forward
0 new messages