Filebeat "Cannot index event publisher.Event"

[Frank]

Hello once again,

After configuring decoders recently noticed that I was missing alerts, after checking Filebeat I noticed warnings, example (shortened):

Mar 23 12:06:01 host filebeat[4553]: 2022-03-23T12:06:01.222+0200        WARN        [elasticsearch]        elasticsearch/client.go:408        Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xc086da222649db18, ext:3877549838983, loc:(*time.Location)(0x42417a0)}, Meta:{"pipeline":"filebeat-7.10.2-wazuh-alerts-pipeline"}, Fields:{"agent":{"ephemeral_id":"dbd129b7-805c-4d0f-b2a2-fb24da28b56d"

...

(status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [data.message] of type [keyword] in document with id '6qI9tn8BO-8Eo70pi__4'. Preview of field's value: '{length=4427, id=cekrc2MFl1iz0EHPTKy0OlItN}'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:292"}}

Also after checking Index Patterns (not sure if related):

Any ideas how I could solve this? 

Thank you in advance 

[Javier Castro]

Hello Frank,

this is due to a conflict in your mapping.

If you take a look at the log it says failed to parse field [data.message] of type [keyword] [...]  "reason":"Can't get text on a START_OBJECT, which means that, at some point, one of the indexed events contained a field named data.message mapped as an object, but at a later point in time, a different event contained a field named data.message mapped as a keyword (a type of string).

This can happen because of two main scenarios:

• A single source of information includes objects and strings under the field decoded as data.message.
• Multiple sources of information, some of them with an object, some others with a string, decode a particular field as data.message.

The best way you can solve this is to understand which events include a field that is decoded as object under data.message and modify the decoder so it is stored under a different field (maybe data.message_object).

You can learn more about conflicts here: https://medium.com/@rkaur05/handling-conflicts-of-elastic-search-1efc593358d2

Hope this helps!

Javier.

[Frank]

Hello,

Changing message fields to different names seemed to solved my issue and relevant events are visible in kibana

I just don't understand why kibana reported about 2 Mapping conflicts ( data.client and data.event), but not data.message one in Stack Management -> Index patterns -> conflicts. Conflicts with data.client and data.event are present in 45 indices though. To solve this, from what I understand, I will require fix them 1 by 1 (fix mapping, create backup, reindex, delete) - ouch.

Anyway, thank you a bunch for the help.

Take care

[Javier Castro]

Hello Frank,

I'm sorry for the late reply here.

The conflict appears in Kibana only if two existing indices have different types of data under the same field name. Maybe in data.message that never got to happen because the first alert with data.message in it always had the same type, but you still had the problem in Elasticsearch.

Regarding the conflicts themselves, you are correct, you will need to fix the mapping, backup, reindex and delete, but you shouldn't fix them 1 by 1, but instead do it index by index (fixing all of the conflicts in every index at once).

Keep in mind that if you reindex the information you are basically moving it from one index to another, it is not going through the Wazuh manager again, so for you to transform the data your reindex request should accommodate manually for field name changes (aside from changes in Wazuh rules and decoders to prevent this in the future).

Hope that helps!

Regards,

Javier.