Mapper parsing exception in Filebeat.

[Кирилл Новогран]

Today I add to my logcollection config a new logfile in JSON format. But unfortenately alerts generated from this log messages didn't appear in events.
I found warnings with 'mapper_parsing_exception' in filebeat log.

~~~

2023-03-24T13:56:01.655+0300    WARN    [elasticsearch] elasticsearch/client.go:408     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xc0ff7e502343a73c, ext:71419312570, loc:(*time.Location)(0x42417a0)}, Meta:{"pipeline":"filebeat-7.10.2-wazuh-alerts-pipeline"}, Fields:{"agent":{"ephemeral_id":"347de08e-2bee-4877-b3fd-11d16fdfd288","hostname":"vm-wazuh","id":"22554303-b6fe-4bad-9906-6b106dbefdc7","name":"vm-wazuh","type":"filebeat","version":"7.10.2"},"ecs":{"version":"1.6.0"},"event":{"dataset":"wazuh.alerts","module":"wazuh"},"fields":{"index_prefix":"wazuh-alerts-4.x-"},"fileset":{"name":"alerts"},"host":{"name":"vm-wazuh"},"input":{"type":"log"},"log":{"file":{"path":"/var/ossec/logs/alerts/alerts.json"},"offset":701265774},"message":"{\"timestamp\":\"2023-03-24T13:56:00.315+0300\",\"rule\":{\"level\":5,\"description\":\"S2box log event.\",\"id\":\"105000\",\"firedtimes\":255,\"mail\":false,\"groups\":[\"local\",\"syslog\",\"json\"]},\"agent\":{\"id\":\"007\",\"name\":\"box1.*****.com\",\"ip\":\"10.***.***.2\"},\"manager\":{\"name\":\"vm-wazuh\"},\"id\":\"1679655360.545355037\",\"full_log\":\"{\\\"app\\\":\\\"s2\\\",\\\"action_type\\\":\\\"s2_deals_card_open\\\",\\\"action_datetime\\\":\\\"2023-03-24T10:55:57Z\\\",\\\"user_id\\\":100415,\\\"user_email\\\":\\\"*****@*****.com\\\",\\\"user_fio\\\":\\\"***** *****\\\",\\\"request_id\\\":\\\"3a355355-8ffc-40f7-b508-bc13e2b0d105\\\",\\\"log_id\\\":\\\"b3599eca-cb75-4abb-b15c-a50048de7ff8\\\",\\\"request_ip\\\":\\\"192.168.172.147\\\",\\\"request_user_agent\\\":\\\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36\\\",\\\"request_url\\\":\\\"https://talent.******.com/deals/7177139\\\",\\\"request_method\\\":\\\"GET\\\",\\\"request_body\\\":{},\\\"payload\\\":{\\\"deal_id\\\":7177139,\\\"deal_name\\\":\\\"***** ******\\\",\\\"contact_ids\\\":[11602182]}}\",\"decoder\":{\"parent\":\"json\",\"name\":\"json\"},\"data\":{\"app\":\"s2\",\"action_type\":\"s2_deals_card_open\",\"action_datetime\":\"2023-03-24T10:55:57Z\",\"user_id\":\"100415\",\"user_email\":\"*******@*********.com\",\"user_fio\":\"********* *******\",\"request_id\":\"3a355355-8ffc-40f7-b508-bc13e2b0d105\",\"log_id\":\"b3599eca-cb75-4abb-b15c-a50048de7ff8\",\"request_ip\":\"192.168.172.147\",\"request_user_agent\":\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36\",\"request_url\":\"https://talent.*******.com/deals/7177139\",\"request_method\":\"GET\",\"payload\":{\"deal_id\":\"7177139\",\"deal_name\":\"******* ********\",\"contact_ids\":[11602182]}},\"location\":\"/box/swarm/fluentd/data/kafka-adt.s2-audit/s2-adt-event.20230324.log\"}","service":{"type":"wazuh"}}, Private:file.State{Id:"native::3015266-64768", PrevId:"", Finished:false, Fileinfo:(*os.fileStat)(0xc00004e4e0), Source:"/var/ossec/logs/alerts/alerts.json", Offset:701267545, Timestamp:time.Time{wall:0xc0ff7e3e601f3c5c, ext:366594266, loc:(*time.Location)(0x42417a0)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x2e0262, Device:0xfd00}, IdentifierName:"native"}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"object mapping for [data.app] tried to parse field [app] as object, but found a concrete value"}

~~~ 

In my "wazuh-alerts" index I already have "app.name" field from other logs.
Could you please help me with this filebeat error?

[Fabian Ruiz]

Hi, thanks for using Wazuh!

To clarify this question a bit, could you please send us how you configured the new log collector in your ossec.conf configuration file?

You can use this documentation to configure a new log collector in wazuh:

https://documentation.wazuh.com/current/user-manual/capabilities/log-data-collection/log-data-configuration.html

[Fabian Ruiz]

Hi, Кирилл Новогран

I have investigated your case and this happens because the [app] field was mapped with object type but the value of the [app] field sent is not of object type, The elasticsearch can not match the field type and throws a mapper_parsing_exception. You must make sure that the value of the [app] field is an object.

I remain attentive to your comments.

Thanks for using Wazuh!

[Кирилл Новогран (KNovogran)]

Thank you for quick reply.

I read some filebeat manuals and found that I could try renaming the field “app” with processors:

https://www.elastic.co/guide/en/beats/filebeat/7.17/rename-fields.html

I add this lines at the beginning of /etc/filebeat/filebeat.yml config:

processors:
- rename:
    ignore_missing: true
    fields:
      - from: "data.app"
        to: "data.application"


Unfortunately, it didn’t work. Could you help me with this config?

суббота, 25 марта 2023 г. в 02:12:50 UTC+3, Fabian Ruiz:

[Fabian Ruiz]

Hi, Кирилл Новогран.

Could you please send us the configuration file ossec.conf, filebeat configuration file, the rule of the alert, with that I could help you transform the data or use a different field.

I remain attentive to your comments.

Thanks for using Wazuh!

[Кирилл Новогран (KNovogran)]

For logcollection on the agent I use shared configuration:

~~~

<agent_config>
<!-- Shared agent configuration here -->
<syscheck>
<disabled>yes</disabled>
</syscheck>
<localfile>
<location>/box/swarm/fluentd/data/kafka-adt.s2-audit/s2-adt-event.*.log</location>
<log_format>json</log_format>
<only-future-events>no</only-future-events>
</localfile>
</agent_config>
~~~

My rule for detecting events:

~~~

<group name="local,syslog,json,">      
  <rule id="105000" level="5">
    <decoded_as>json</decoded_as>
    <description>S2box log event.</description>
  </rule>
</group>

~~~

filebeat.yml file attached.

I will appreciate it if you help me.

среда, 29 марта 2023 г. в 15:02:09 UTC+3, Fabian Ruiz:

[Fabian Ruiz]

Hi, Кирилл Новогран.

You could try to convert the value to the indicated value if possible.

https://www.elastic.co/guide/en/beats/filebeat/7.17/convert.html#convert

You can update the mapping for the data.app field to accept a string valuue insted of an object, you may need to reindex your data or recreate the index for these changes to take effect.

Send me the complete configurations so I can check well, you can hide the sensitive data, with that I could organize a lab to test your problem, also I would need the pipeline.yaml of the filebeat and an example of the value of [app.name] in the wazuh-alerts index.

https://documentation.wazuh.com/current/user-manual/elasticsearch/configure-indices.html

[Кирилл Новогран (KNovogran)]

When I've seen your suggestion I found that my method has a compatibility issue in my current configuration. Thank you for your help. I will try another approach to my problem.

пятница, 31 марта 2023 г. в 20:39:51 UTC+3, Fabian Ruiz: