Upgraded 4.2.0 to 4.2.5 now getting a shard error

[nbent...@gmail.com]

Hi,

Over the weekend I upgraded from 4.2.0 to 4.2.5 and I ended up getting one shard with an error:

illegal_argument_exception at shard 0index wazuh-alerts-3.x-2022.01.10node 2qHt7vRNSiKaaIwQxvADCg

Type: illegal_argument_exception

Reason: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [rule.mitre.technique] in order to load field data by uninverting the inverted index. Note that this can use significant memory.

Should I be concerned? Will this error persist in each new shard or is this a one time issue?

[mayte...@wazuh.com]

Hi,
 
It is odd that the index is named as wazuh-alerts-3.x-yyyy.mm.dd. Since Wazuh 4.0.0 version the alert indices should be named as wazuh-alerts-4.x-yyyy.mm.dd (You can check here the Wazuh template: wazuh-template.json - v4.0.0).
Maybe some steps were missing when performing the upgrade to v4.2.0 and still affect the v4.2.5.
 
Regarding the error you mention, it is related to the missing Wazuh template since it seems there is some issue with the mapping. If the Wazuh template is applied properly, the rule.mitre.technique field should be keyword instead of text (rule.mitre.technique field - v4.2.5)

You can check which indices did not apply the Wazuh template by running the following query in your Elasticsearch server: curl -k -u <user>:<pass> "https://localhost:9200/wazuh-alerts*/_mapping/field/rule.mitre.technique?pretty"
All those indexes that do not have the rule.mitre.technique as keyword have not applied the Wazuh template.

Could you perform the following query in your Elasticsearch server to get your current templates?: curl -k -u <user>:<pass> 'https://localhost:9200/_cat/templates'
Check if the wazuh template is properly loaded and matches the desired indices (we use the wazuh template to define the mapping for wazuh-alerts-4.x-* and wazuh-archives-4.x-* indices)
 
If not, as I said before, maybe some steps were missing when performing the migration. Please follow this guide: upgrading-filebeat
 
Once the Wazuh template is properly loaded, you should reindex the indices created without using the Wazuh template (in order to apply the proper template) or delete them (this second option will result in the data loss for those indexes without the possibility of restoring it unless you make a backup beforehand).
 
Keep us updated!

Best regards,
Mayte Ariza

[nbent...@gmail.com]

You hit the nail on the head Mayte,

After I posted I double checked my work from over the weekend and found filebeat was accidently added to the Elastic server and not my wazuh instance... I upgraded the Wazuh server's filebeat (uninstalled the filebeat from the elastic server) and I now see wazuh-alerts-4.x-yyyy.mm.dd  templates being used. As for the reindexing I'll investigate that to fix the old Shard: wazuh-alerts-3.x-2022.01.10 to see if it can be reindexed to wazuh-alerts-4.x-2022.01.10.

[mayte...@wazuh.com]

Hi,
 
I am glad you managed to solve the issue!
 
You may use the following command in your Elasticsearch server to reindex the wazuh-alerts-3.x-2022.01.10 index:
 
curl -k -u <user>:<pass> -X POST "https://localhost:9200/_reindex" -H 'Content-Type: application/json' -d'
{
  "source": {
    "index": "wazuh-alerts-3.x-2022.01.10"
  },
  "dest": {
    "index": "wazuh-alerts-4.x-2022.01.10"
  }
}
'
(It may take a while)
 
If the wazuh-alerts-4.x-2022.01.10 index is already created, I recommend you to change the name (e.g. wazuh-alerts-4.x-2022.01.10-0) in order not to taint the index and avoid documents duplicated if the reindex process does not work as expected.
 
Also, before deleting the wazuh-alerts-3.x-2022.01.10 index, make sure that the reindex process was performed properly to prevent losing any data. You may check if the number of documents in both indices match.
 
Best regards,
Mayte Ariza