Can't get text on a START_OBJECT at 1:1194 for data.status(keyword)
1,556 views
Skip to first unread message
M Jones
Jul 12, 2022, 6:39:29 AM7/12/22
to Wazuh mailing list
Hi,
Jul 12 10:25:39 dev filebeat[816]: 2022-07-12T10:25:39.169Z#011WARN#011[elasticsearch]#011elasticsearch/client.go:414#011Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2022, time.July, 12, 10, 25, 38, 145721597, time.Local), Meta:{"pipeline":"filebeat-7.17.4-wazuh-alerts-pipeline"}, Fields:{"agent":{"ephemeral_id":"123qwe45-12a4-4a15-9jht-x4318gvd051l","hostname":"dev","id":"12345678-41e3-4d5a-1234-e7cbvdfe1478","name":"dev","type":"filebeat","version":"7.17.4"},"ecs":{"version":"1.12.0"},"event":{"dataset":"wazuh.alerts","module":"wazuh"},"fields":{"index_prefix":"wazuh-alerts-4.x-"},"fileset":{"name":"alerts"},"host":{"name":"dev"},"input":{"type":"log"},"log":{"file":{"path":"/var/ossec/logs/alerts/alerts.json"},"offset":242342308},"message":"{\"timestamp\":\"2022-07-12T10:25:33.706+0000\",\"rule\":{\"level\":3,\"description\":\"Azure: AD \",\"id\":\"87802\",\"firedtimes\":19,\"mail\":false,\"groups\":[\"azure\"]},\"agent\":{\"id\":\"000\",\"name\":\"dev\"},\"manager\":{\"name\":\"devl"},\"id\":\"1657621533.436382052\",\"decoder\":{\"name\":\"json\"},\"data\":{\"id\":\"d88aklx8-1171-4740-87b1-61d2219a2133\",\"createdDateTime\":\"2022-07-11T09:55:24Z\",\"userDisplayName\":\"redactedname\",\"userPrincipalName\":\"reda...@domain.com\",\"userId\":\"ddsdq48k-8g70-a264-4red-f120dede082d\",\"appId\":\"c44b4083-fds2-65dq-b47d-1234567891011e\",\"appDisplayName\":\"Azure Portal\",\"ipAddress\":\"85.174.174.147\",\"clientAppUsed\":\"Browser\",\"correlationId\":\"b6662517-74fe-496d-a590-gf5432seru78\",\"conditionalAccessStatus\":\"success\",\"isInteractive\":\"true\",\"riskDetail\":\"hidden\",\"riskLevelAggregated\":\"hidden\",\"riskLevelDuringSignIn\":\"hidden\",\"riskState\":\"none\",\"riskEventTypes\":[],\"riskEventTypes_v2\":[],\"resourceDisplayName\":\"Windows Azure Service Management API\",\"resourceId\":\"mn764rd3-ba00-4fd7-ba43-vsd5431qhjky\",\"status\":{\"errorCode\":\"0\",\"failureReason\":\"Other.\",\"additionalDetails\":\"MFA requirement satisfied by claim in the token\"},\"deviceDetail\":{\"operatingSystem\":\"Windows 10\",\"browser\":\"Chrome 103.0.0\",\"isCompliant\":\"false\",\"isManaged\":\"false\"},\"location\":{\"city\":\"London\",\"state\":\"London\",\"countryOrRegion\":\"GB\",\"geoCoordinates\":{\"altitude\":\"null\",\"latitude\":\"22.212040\",\"longitude\":\"-32.786120\"}},\"appliedConditionalAccessPolicies\":[],\"azure_tag\":\"azure-ad-graph\",\"azure_aad_tag\":\"azure-active_directory\"},\"location\":\"Azure\"}","service":{"type":"wazuh"}}, Private:file.State{Id:"native::1552056-64513", PrevId:"", Finished:false, Fileinfo:(*os.fileStat)(0xc0008f1a00), Source:"/var/ossec/logs/alerts/alerts.json", Offset:242343839, Timestamp:time.Date(2022, time.July, 12, 10, 1, 52, 137407350, time.Local), TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x17aeb8, Device:0xfc01}, IdentifierName:"native"}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [data.status] of type [keyword] in document with id 'Fsbx8YEBh1bsAwwzaTpR'. Preview of field's value: '{failureReason=Other., errorCode=0, additionalDetails=MFA requirement satisfied by claim in the token}'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:1194"}}, dropping event!
Im having some issues with some events from AzureAD Graph and looking for some guidance. Filebeats seems to be dropping the logs for the data.status field(keyword) but not too sure how to fix this for future logs so i can alert on them.
Any help would be great,thanks
Log -
Jul 12 10:25:39 dev filebeat[816]: 2022-07-12T10:25:39.169Z#011WARN#011[elasticsearch]#011elasticsearch/client.go:414#011Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2022, time.July, 12, 10, 25, 38, 145721597, time.Local), Meta:{"pipeline":"filebeat-7.17.4-wazuh-alerts-pipeline"}, Fields:{"agent":{"ephemeral_id":"123qwe45-12a4-4a15-9jht-x4318gvd051l","hostname":"dev","id":"12345678-41e3-4d5a-1234-e7cbvdfe1478","name":"dev","type":"filebeat","version":"7.17.4"},"ecs":{"version":"1.12.0"},"event":{"dataset":"wazuh.alerts","module":"wazuh"},"fields":{"index_prefix":"wazuh-alerts-4.x-"},"fileset":{"name":"alerts"},"host":{"name":"dev"},"input":{"type":"log"},"log":{"file":{"path":"/var/ossec/logs/alerts/alerts.json"},"offset":242342308},"message":"{\"timestamp\":\"2022-07-12T10:25:33.706+0000\",\"rule\":{\"level\":3,\"description\":\"Azure: AD \",\"id\":\"87802\",\"firedtimes\":19,\"mail\":false,\"groups\":[\"azure\"]},\"agent\":{\"id\":\"000\",\"name\":\"dev\"},\"manager\":{\"name\":\"devl"},\"id\":\"1657621533.436382052\",\"decoder\":{\"name\":\"json\"},\"data\":{\"id\":\"d88aklx8-1171-4740-87b1-61d2219a2133\",\"createdDateTime\":\"2022-07-11T09:55:24Z\",\"userDisplayName\":\"redactedname\",\"userPrincipalName\":\"reda...@domain.com\",\"userId\":\"ddsdq48k-8g70-a264-4red-f120dede082d\",\"appId\":\"c44b4083-fds2-65dq-b47d-1234567891011e\",\"appDisplayName\":\"Azure Portal\",\"ipAddress\":\"85.174.174.147\",\"clientAppUsed\":\"Browser\",\"correlationId\":\"b6662517-74fe-496d-a590-gf5432seru78\",\"conditionalAccessStatus\":\"success\",\"isInteractive\":\"true\",\"riskDetail\":\"hidden\",\"riskLevelAggregated\":\"hidden\",\"riskLevelDuringSignIn\":\"hidden\",\"riskState\":\"none\",\"riskEventTypes\":[],\"riskEventTypes_v2\":[],\"resourceDisplayName\":\"Windows Azure Service Management API\",\"resourceId\":\"mn764rd3-ba00-4fd7-ba43-vsd5431qhjky\",\"status\":{\"errorCode\":\"0\",\"failureReason\":\"Other.\",\"additionalDetails\":\"MFA requirement satisfied by claim in the token\"},\"deviceDetail\":{\"operatingSystem\":\"Windows 10\",\"browser\":\"Chrome 103.0.0\",\"isCompliant\":\"false\",\"isManaged\":\"false\"},\"location\":{\"city\":\"London\",\"state\":\"London\",\"countryOrRegion\":\"GB\",\"geoCoordinates\":{\"altitude\":\"null\",\"latitude\":\"22.212040\",\"longitude\":\"-32.786120\"}},\"appliedConditionalAccessPolicies\":[],\"azure_tag\":\"azure-ad-graph\",\"azure_aad_tag\":\"azure-active_directory\"},\"location\":\"Azure\"}","service":{"type":"wazuh"}}, Private:file.State{Id:"native::1552056-64513", PrevId:"", Finished:false, Fileinfo:(*os.fileStat)(0xc0008f1a00), Source:"/var/ossec/logs/alerts/alerts.json", Offset:242343839, Timestamp:time.Date(2022, time.July, 12, 10, 1, 52, 137407350, time.Local), TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x17aeb8, Device:0xfc01}, IdentifierName:"native"}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [data.status] of type [keyword] in document with id 'Fsbx8YEBh1bsAwwzaTpR'. Preview of field's value: '{failureReason=Other., errorCode=0, additionalDetails=MFA requirement satisfied by claim in the token}'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:1194"}}, dropping event!
Manuel Pedro Gomez Castro
Jul 12, 2022, 9:38:54 AM7/12/22
to Wazuh mailing list
Hello, thank you for reaching out to us!
Whenever an event is indexed that was not present in that idex's mapping, it is assigned a type dependent on the data present. You can check your current index mapping under the index management section.
In this case, it seems like the first time the data.status field appeared, it was assigned the keyword type, but on the event you shared with us, it is an object "status\":{\"errorCode\":\"0\",\"failureReason\":\"Other.\",\"additional...
In order to prevent these kinds of alerts from happening in the future, I would recommend the index mapping template is updated so future events are handled correctly as objects.
To do this, you can edit the file /etc/filebeat/wazuh-template.json in your manager's machine. Add a mapping under the data.properties object for "status" with the type "object" or "flattened"
You can learn more about field data types on this URL https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html
I hope this is helps!
Whenever an event is indexed that was not present in that idex's mapping, it is assigned a type dependent on the data present. You can check your current index mapping under the index management section.
In this case, it seems like the first time the data.status field appeared, it was assigned the keyword type, but on the event you shared with us, it is an object "status\":{\"errorCode\":\"0\",\"failureReason\":\"Other.\",\"additional...
In order to prevent these kinds of alerts from happening in the future, I would recommend the index mapping template is updated so future events are handled correctly as objects.
To do this, you can edit the file /etc/filebeat/wazuh-template.json in your manager's machine. Add a mapping under the data.properties object for "status" with the type "object" or "flattened"
You can learn more about field data types on this URL https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html
I hope this is helps!
M Jones
Jul 21, 2022, 2:15:39 AM7/21/22
to Wazuh mailing list
Hi Manuel,
Thanks for getting back to me but im still struggling to get this to work, ichnaged the mapping to object(Had another issue with data.location) but seems to break filebeats. Do i need to add a new mapping but if so can one filed have two mappings?
Reply all
Reply to author
Forward
0 new messages