Blank Visualizations in Wazuh

[Michael Hodge]

Hi,

We were recently getting the error below and the visualizations on the dashboards were saying "No results found", so we did a reindex.  Now we do not get the error or that saying.  We just get blank visualizations.  Events still come in and are clear to read.  Any reason why the visualizations are just blank?

App Version 4.0.4 

App Revision 4016

Error:

illegal_argument_exception

Request:

{ "aggs": { "2": { "terms": { "field": "agent.id", "order": { "_count": "desc" }, "size": 5 }, "aggs": { "3": { "terms": { "field": "agent.name", "order": { "_count": "desc" }, "size": 5 }, "aggs": { "4": { "terms": { "field": "syscheck.uname_after", "order": { "_count": "desc" }, "size": 1 } } } } } } }, "size": 0, "stored_fields": [ "*" ], "script_fields": {}, "docvalue_fields": [ { "field": "data.aws.createdAt", "format": "date_time" }, { "field": "data.aws.end", "format": "date_time" }, { "field": "data.aws.resource.instanceDetails.launchTime", "format": "date_time" }, { "field": "data.aws.service.eventFirstSeen", "format": "date_time" }, { "field": "data.aws.service.eventLastSeen", "format": "date_time" }, { "field": "data.aws.start", "format": "date_time" }, { "field": "data.aws.updatedAt", "format": "date_time" }, { "field": "data.timestamp", "format": "date_time" }, { "field": "data.vulnerability.published", "format": "date_time" }, { "field": "data.vulnerability.updated", "format": "date_time" }, { "field": "syscheck.mtime_after", "format": "date_time" }, { "field": "syscheck.mtime_before", "format": "date_time" }, { "field": "timestamp", "format": "date_time" } ], "_source": { "excludes": [ "@timestamp" ] }, "query": { "bool": { "must": [], "filter": [ { "match_all": {} }, { "match_phrase": { "manager.name": { "query": "WazPrem01.INF.INTERNAL" } } }, { "match_phrase": { "rule.groups": { "query": "syscheck" } } }, { "range": { "timestamp": { "gte": "2022-02-22T07:00:00.000Z", "lte": "2022-02-23T04:00:00.000Z", "format": "strict_date_optional_time" } } } ], "should": [], "must_not": [] } } }

{ "took": 27, "timed_out": false, "_shards": { "total": 31, "successful": 29, "skipped": 29, "failed": 2, "failures": [ { "shard": 0, "index": "wazuh-alerts-3.x-2022.02.22", "node": "56Pjj478R1Oa_oUfZ4SOww", "reason": { "type": "illegal_argument_exception", "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [agent.id] in order to load field data by uninverting the inverted index. Note that this can use significant memory." } }, { "shard": 0, "index": "wazuh-alerts-3.x-2022.02.23", "node": "56Pjj478R1Oa_oUfZ4SOww", "reason": { "type": "illegal_argument_exception", "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [agent.id] in order to load field data by uninverting the inverted index. Note that this can use significant memory." } } ] }, "hits": { "total": 0, "max_score": 0, "hits": [] } }  

[Miguel Keane]

Hello Michael, 

It seems like there is an issue with your index mapping. Before further troubleshooting, I would recommend deleting your index pattern and, when going back to Wazuh, it should create back again. 

To do so, go to Stack Management --> Index Patterns

Then, click on the remove icon: 

Now, go back to the Wazuh Kibana App. It should generate the index pattern back again, and hopefully, the issues will disappear. 

Rest assured, this process is safe and you will not delete any data. The index pattern just contains a list of all fields and their corresponding value types. Where it seems there is a mismatch for some reason. 

Let me know if this helped. 

Otherwise we can continue troubleshooting. 

Best regards, 
Miguel Keane

[Michael Hodge]

Hi,

I followed your instructions, deleted the index pattern.  When I went back into the Wazuh app, it said it could not find the default index pattern so it is creating a new one.  Now we are back to having the "illegal_argument_exception" error when loading anything with visualizations.

Posting the error and the log for the Wazuh App below:

/logs/wazuhapp.log

2022/03/04 09:35:50 INFO Default index pattern not found, creating it... 2022/03/04 09:41:38 ERROR Not Found

{ "aggs": { "2": { "terms": { "field": "agent.name", "order": { "_count": "desc" }, "size": 5 } } }, "size": 0, "stored_fields": [ "*" ], "script_fields": {}, "docvalue_fields": [ { "field": "data.aws.createdAt", "format": "date_time" }, { "field": "data.aws.end", "format": "date_time" }, { "field": "data.aws.resource.instanceDetails.launchTime", "format": "date_time" }, { "field": "data.aws.service.eventFirstSeen", "format": "date_time" }, { "field": "data.aws.service.eventLastSeen", "format": "date_time" }, { "field": "data.aws.start", "format": "date_time" }, { "field": "data.aws.updatedAt", "format": "date_time" }, { "field": "data.timestamp", "format": "date_time" }, { "field": "data.vulnerability.published", "format": "date_time" }, { "field": "data.vulnerability.updated", "format": "date_time" }, { "field": "syscheck.mtime_after", "format": "date_time" }, { "field": "syscheck.mtime_before", "format": "date_time" }, { "field": "timestamp", "format": "date_time" } ], "_source": { "excludes": [ "@timestamp" ] }, "query": { "bool": { "must": [], "filter": [ { "match_all": {} }, { "match_phrase": { "manager.name": { "query": "WazPrem01.INF.INTERNAL" } } }, { "match_phrase": { "rule.groups": { "query": "syscheck" } } }, { "range": { "timestamp": { "gte": "2022-03-04T15:35:40.063Z", "lte": "2022-03-04T15:50:40.063Z", "format": "strict_date_optional_time" } } } ], "should": [], "must_not": [] } } } 

  { "took": 36, "timed_out": false, "_shards": { "total": 31, "successful": 30, "skipped": 30, "failed": 1, "failures": [ { "shard": 0, "index": "wazuh-alerts-3.x-2022.03.04", "node": "7Ic8UH-CQyCklf7Ag6LrRg", "reason": { "type": "illegal_argument_exception", "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [agent.name] in order to load field data by uninverting the inverted index. Note that this can use significant memory." } } ] }, "hits": { "total": 0, "max_score": 0, "hits": [] } }  

[Miguel Keane]

Hello Michael, 

you mentioned that you are using Wazuh 4.0.4. But the issue I am seeing is on the index "wazuh-alerts-3.x-2022.03.04". This is likely the issue. The index should be named: "wazuh-alerts-4.x-2022.03.04"

Assuming that Wazuh, Elastic and Filebeat are in the mentioned versions and the Wazuh App is also on the compatible version. Please run the following commands to ensure the correct index is being generated:

curl -so /etc/filebeat/wazuh-template.json https://raw.githubusercontent.com/wazuh/wazuh/v4.0.4/extensions/elasticsearch/7.x/wazuh-template.json
chmod go+r /etc/filebeat/wazuh-template.json

 

curl -s https://packages.wazuh.com/4.x/filebeat/wazuh-filebeat-0.1.tar.gz | sudo tar -xvz -C /usr/share/filebeat/module

filebeat setup --index-management -E output.logstash.enabled=false

systemctl daemon-reload

systemctl enable filebeat

systemctl start filebeat

Also, make sure there is no index-pattern called: wazuh-alerts-3.x-*   

If there is, you should remove it. The pattern from version 4.x and above is: wazuh-alerts-*

If this didn't fix it either. The issue is likely due to some version mismatch somewhere. Instead of spending efforts troubleshooting it, I would recommend carefully upgrading Wazuh and Elastic to the latest version following our documentation to the dot: https://documentation.wazuh.com/current/upgrade-guide/index.html

Let me know if this helped with your issue. I will be happy to assist you until you get everything back to working as expected. 

Best regards, 
Miguel Keane

 

Miguel Keane
IT Security Engineer — Wazuh, Inc.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/bbd12312-f35b-4606-9a68-4d36f06c0a7cn%40googlegroups.com.