Update 4.9 dashboard error "search_phase_execution_exception"

[lionel]

Hi,

I've just updated my server in 4.9 version all things seems to work except primary dashboard. Dashboard is empty and print popup with an alert.

Some one can help me?

[WazuhError]: search_phase_execution_exception: [illegal_argument_exception] Reason: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [manager.name] in order to load field data by uninverting the inverted index. Note that this can use significant memory.; [illegal_argument_exception] Reason: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [manager.name] in order to load field data by uninverting the inverted index. Note that this can use significant memory.; [illegal_argument_exception] Reason: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [manager.name] in order to load field data by uninverting the inverted index. Note that this can use significant memory.; [illegal_argument_exception] Reason: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [manager.name] in order to load field data by uninverting the inverted index. Note that this can use significant memory.; [illegal_argument_exception] Reason: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [manager.name] in order to load field data by uninverting the inverted index. Note that this can use significant memory.; [illegal_argument_exception] Reason: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [manager.name] in order to load field data by uninverting the inverted index. Note that this can use significant memory.; [illegal_argument_exception] Reason: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [manager.name] in order to load field data by uninverting the inverted index. Note that this can use significant memory.; [illegal_argument_exception] Reason: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [manager.name] in order to load field data by uninverting the inverted index. Note that this can use significant memory.; [illegal_argument_exception] Reason: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [manager.name] in order to load field data by uninverting the inverted index. Note that this can use significant memory.; [illegal_argument_exception] Reason: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [manager.name] in order to load field data by uninverting the inverted index. Note that this can use significant memory.; [illegal_argument_exception] Reason: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [manager.name] in order to load field data by uninverting the inverted index. Note that this can use significant memory.; [illegal_argument_exception] Reason: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [manager.name] in order to load field data by uninverting the inverted index. Note that this can use significant memory.; [illegal_argument_exception] Reason: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [manager.name] in order to load field data by uninverting the inverted index. Note that this can use significant memory.; [illegal_argument_exception] Reason: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [manager.name] in order to load field data by uninverting the inverted index. Note that this can use significant memory.; [illegal_argument_exception] Reason: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [manager.name] in order to load field data by uninverting the inverted index. Note that this can use significant memory.; [illegal_argument_exception] Reason: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [manager.name] in order to load field data by uninverting the inverted index. Note that this can use significant memory.; [illegal_argument_exception] Reason: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [manager.name] in order to load field data by uninverting the inverted index. Note that this can use significant memory.; [illegal_argument_exception] Reason: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [manager.name] in order to load field data by uninverting the inverted index. Note that this can use significant memory.; [illegal_argument_exception] Reason: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [manager.name] in order to load field data by uninverting the inverted index. Note that this can use significant memory.; [illegal_argument_exception] Reason: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [manager.name] in order to load field data by uninverting the inverted index. Note that this can use significant memory.; [illegal_argument_exception] Reason: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [manager.name] in order to load field data by uninverting the inverted index. Note that this can use significant memory.; [illegal_argument_exception] Reason: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [manager.name] in order to load field data by uninverting the inverted index. Note that this can use significant memory.; [illegal_argument_exception] Reason: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [manager.name] in order to load field data by uninverting the inverted index. Note that this can use significant memory.; [illegal_argument_exception] Reason: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [manager.name] in order to load field data by uninverting the inverted index. Note that this can use significant memory.

[Lamya Imam]

Hello lionel,

This seems like there is a mapping issue of manager which is different from our template. (https://raw.githubusercontent.com/wazuh/wazuh/v4.9.0/extensions/elasticsearch/7.x/wazuh-template.json)
You should use the expected template.

For now, I would suggest you to follow the guidelines below:

- Stop filebeat:
# systemctl stop filebeat

- Download the alerts template again and set the permission accordingly:
# curl -so /etc/filebeat/wazuh-template.json https://raw.githubusercontent.com/wazuh/wazuh/v4.9.0/extensions/elasticsearch/7.x/wazuh-template.json
# chmod go+r /etc/filebeat/wazuh-template.json

- Upload the new Wazuh template and pipelines for Filebeat:
# filebeat setup --pipelines
# filebeat setup --index-management -E output.logstash.enabled=false

- Restart Filebeat:
# systemctl restart filebeat

Next, I would recommend you to re-map your index. This means you should create a new index using the data from the corrupted index and apply the template. Once you are done, you can replace the corrupted index with the well-mapped index. Basically, re-index the affected index with manager.name mapped as keyword.
Reference: https://documentation.wazuh.com/current/user-manual/wazuh-indexer/re-indexing.html#re-indexing

Let me know if you need further assistance on this!

[lionel]

Thank you for your answer.

I followed your guidelines, but I got an error during reindexing. In fact, the index grew for a long time and then I got an error on the console. Now the original index is nearly 1GB, while the reindexed one is nearly 4GB.

Error in dev_tools:

{
  "error": {
    "root_cause": [
      {
        "type": "rejected_execution_exception",
        "reason": "rejected execution of coordinating operation [coordinating_and_primary_bytes=104817822, replica_bytes=0, all_bytes=104817822, coordinating_operation_bytes=2991479, max_coordinating_and_primary_bytes=107374182]"
      }
    ],
    "type": "rejected_execution_exception",
    "reason": "rejected execution of coordinating operation [coordinating_and_primary_bytes=104817822, replica_bytes=0, all_bytes=104817822, coordinating_operation_bytes=2991479, max_coordinating_and_primary_bytes=107374182]"
  },
  "status": 429
}

Error in log wazuh-cluster.log, but it does not seem to be linked:

Json Mapping Error: Cannot invoke "java.lang.Long.longValue()" because "this.cacheMaxSize" is null (through reference chain: org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector$CacheMaxSizeStatus["Cache_MaxSize"])

Thank you for helping.

[lionel]

Hi

I don't know if it's relevant, but today the dashboard isn't working either, even though it should be partially functional since a new index has been created with the “right template”.

[lionel]

Hi i tried to delete old indices, and wazuh work again.

So i backup my old indice (30 days rolling) in case of need .
thanks.

[Lamya Imam]

Hello lionel,

I am glad to hear that your issue has been resolved! Thank you for letting me know.

Regards,