Set Tag at runtime?

[Amit Saha]

Hi all,

I am trying to achieve a logging setup where I want to use a separate output based on the absence or presence of a certain field in the incoming log. One way to do that would be to update the `Tag` if

my field of interest is present or absent. To put it more concretely, I want to use a different Elastic Search index name for certain category of logs identified by the presence of a certain field for retention purposes.

For example, say my application logs to stdout and can emit two kinds of logs:

1. {"request_path":"/foo/bar", "request_body": {"foo":"bar"}}

2. {"request_path":"/foo/bar"}

I want to now look for the presence of the "request_body" field and tag it differently so that I can use a different index in elastic search.

Is this possible to do? Has somebody else tried something similar?

Thanks,

Amit.

[Eduardo Silva]

hi Amit, 

you can use the stream processor engine as an alternative workaround, you can create your own tag stream like:

CREATE STREAM payloads WITH(tag='requests_payloads') AS SELECT * FROM STREAM:tail.0 WHERE @record.contains(request_body);

CREATE STREAM nopayloads WITH(tag='requests') AS SELECT * FROM STREAM:tail.0 WHERE NOT @record.contains(request_body);

then mark your first tail input section as routable off, then add two OUTPUT sections for tags request_payloads and requests . 

Not so straightforward but it will get the job done, let me know how it goes..

cheers, 

 

--
You received this message because you are subscribed to the Google Groups "Fluent-Bit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluent-bit+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fluent-bit/223c37ea-8077-450b-a49f-a95e99cfa3c5%40googlegroups.com.

--

Eduardo Silva
Principal Engineer  | Arm
. . . . . . . . . . . . . . . . . . . . . . . . . . . 
m. +506 70138007
Arm.com
Treasuredata.com

   

[Amit Saha]

Hi Eduardo,

Thank you. I got it working as follows:
(complete examples:
https://github.com/amitsaha/webapp-exp/tree/master/logging/fluent-bit-streaming)

# fluent-bit conf

[SERVICE]
Flush 1
Log_Level info
Parsers_file parsers.conf
Streams_File stream_processor.conf


[INPUT]
Name Tail
Path ./sample.log
Parser json
routable off


[OUTPUT]
Name null
Match type2

[OUTPUT]
Name stdout
Match type1

# stream-processor.conf

[STREAM_TASK]
Name example
Exec CREATE STREAM payloads WITH(tag='type1') AS SELECT * FROM
STREAM:tail.0 WHERE @record.contains(request_body);

[STREAM_TASK]
Name example1
Exec CREATE STREAM nopayloads WITH(tag='type2') AS SELECT * FROM

STREAM:tail.0 WHERE NOT @record.contains(request_body);

# sample.log
{"date": "22/abr/2019:12:43:51 -0600", "ip": "73.113.230.135", "word":
"balsamine", "country": "Japan", "flag": false, "num": 96}
{"date": "22/abr/2019:12:43:52 -0600", "ip": "242.212.128.227",
"word": "inappendiculate", "country": "Chile", "flag": false, "num":
15}
{"date": "22/abr/2019:12:43:52 -0600", "ip": "85.61.182.212", "word":
"elicits", "country": "Argentina", "flag": true, "num": 73}
{"date": "22/abr/2019:12:43:52 -0600", "ip": "124.192.66.23", "word":
"Dwan", "country": "Germany", "flag": false, "num": 67}
{"date": "22/abr/2019:12:43:52 -0600", "ip": "18.135.244.142", "word":
"chesil", "country": "Argentina", "flag": true, "num": 19,
"request_body": "foo"}



When fluent bit is run, we only see:

{"date": "22/abr/2019:12:43:52 -0600", "ip": "18.135.244.142", "word":
"chesil", "country": "Argentina", "flag": true, "num": 19,
"request_body": "foo"}


Is the way I am defining the `STREAM_TASK` the right way to do it when
there are multiple Exec statements?

Thanks,
Amit.

--
http://echorand.me