copy tags from json logs

[Chitender Kumar]

we have log line something like below:
----------------

{\"log_version\": 1, \"log_type\": \"EVENT\", \"timestamp\": 1616415932304, \"sub_module\": \"users\", \"module\": \"administrative-metadata-management\", \"event_description\": \"goyal, siddharth edited\", \"tenant_id\": \"tenant3\", \"identifier\": {\"@kibana-highlighted-field@actor@/kibana-highlighted-field@\": {\"type\": \"user\", \"value\": \"19462770-74ad-4d83-b144-cd6a25069f03\"}, \"target\": {\"type\": \"user\", \"value\": [\"674a2f25-2262-45c9-a78c-77942e58aaf0\"]}}, \"status\": 0, \"request_id\": \"amm-24fd64c1-7da0-4122-ad2c-afe09fba37bb\", \"meta\": {\"request_path\": \"/users/674a2f25-2262-45c9-a78c-77942e58aaf0/\", \"user_id\": \"19462770-74ad-4d83-b144-cd6a25069f03\", \"status\": \"Edited user\", \"method\": \"PUT\"}, \"event_id\": \"default\"}
-------------------

these are json logs and we have specific field which is log_type and its expected values are EVENT, AUDIT.
now we want to push logs which has log_type: EVENT to event index on Elasticsearch and log_type: AUDIT to audit index on elasticsearch.
for this i tried below config:
------------------------

# <filter event.**>
# @type rewrite_tag_filter
# <rule>
# key $.log.log_type
# pattern ^(.+)$
# tag $1
# </rule>
# </filter>

------------------

so that we can route on the basis of tags. but we were unable to do it and getting below error:

------------------

2021-03-22 13:50:02 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not matched with data '2021-03-22 13:49:59 +0000 [warn]: #0 unknown placeholder `${record[\"log\"][\"log_type\"]}` found\n'" location=nil tag="event.var.log.containers.audit-report-fluentd-2c8nd_logging_fluentd-faea24e0af60e0d4f45f936bf99a33a0972a7d2deeeed88aea4b816dc8cd8e78.log" time=2021-03-22 13:49:59.265421841 +0000 record={"log"=>"2021-03-22 13:49:59 +0000 [warn]: #0 unknown placeholder `${record[\"log\"][\"log_type\"]}` found\n"}

-----------------------------

any help or guidance on this?

[Kentaro Hayashi]

Hi,

If {"log": "{\"log_version\": 1, ...}"} is given in filter event.* , you may need to parse "log" field explicitly.

Then, rewrite tag by rewrite_tag_filter with key $.log_type.

Regards,

2021年3月23日火曜日 0:13:04 UTC+9 Chitender Kumar:

[Chitender Kumar]

thanks for your response @kentaro Hayashi. i realised it sometime later and implemented the same.

--
You received this message because you are subscribed to a topic in the Google Groups "Fluentd Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fluentd/Wa9AMzc8QpY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fluentd+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fluentd/09aa7bee-ca70-4e0a-a57a-eb361bbe9503n%40googlegroups.com.