Issues with Wazuh (possible Filebeat) after upgradring from 4.4.2 to 4.5.2

[Oliver Olsen]

Hi there,

Just upgraded my Wazuh cluster from 4.4.2 to 4.5.2 follwing the upgrade procedure in the docs. No errors during the upgrade, but I don't see anything on the Wazuh dashboard, just "There are no results for selected time range. Try another one."

If I extend the time range I surely see my old alerts. I tried looking for obvious issues in logs, and one that looks to have issues is Filebeat (7.10.2). The /var/log/filebeat/filebeat looks like this

2023-09-12T12:27:38.114+0200 ERROR [elasticsearch] elasticsearch/client.go:224 failed to perform any bulk index operations: 400 Bad Request: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}

2023-09-12T12:27:38.249+0200 ERROR [publisher_pipeline_output] pipeline/output.go:180 failed to publish events: 400 Bad Request: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}

2023-09-12T12:27:38.249+0200 INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(elasticsearch(\`https://<my-ip>:9200``))`

2023-09-12T12:27:38.249+0200 INFO [publisher] pipeline/retry.go:213 retryer: send wait signal to consumer

2023-09-12T12:27:38.249+0200 INFO [publisher] pipeline/retry.go:217 done

For the sake of it I did upgrade a lab-cluster as well, and got the same issues. Despite this, I do get a few Slack notifications for some of the alerts that I have configured, so it's not totally broken...

For the reference, my cluster consists of 3 indexer-nodes, 3 manger-nodes, 1 dashboard, plus an nginx LB-setup for the manager nodes. All updated Debian 11 VMs. Been working perfectly since May, so hopefully there is an easy fix for this.

Thanks!

[Oliver Olsen]

Forgot to mention that

# filebeat test output

have no errors regarding DNS lookup, Elasticsearch or TLS

[Gonzalo Acuña]

Hi, Oliver.
1. Have you tried restarting Filebeat?
2. Can you share the output of the filebeat test output command?

3. Please verify the compatibility.override_main_response_version option is set to true in /etc/wazuh-indexer/opensearch.yml (Wazuh indexer). If not, set it to true, then restart the Wazuh indexer nodes and test again.

Regards.
Gonzalo.

[Oliver Olsen]

Hi Gonzalo,

You're a genious ;-)

Surely it was the opensearch.yml file that didn't have the  compatibility.override_main_response_version set to true. Changed it, restarted the indexers and BANG :-) Not sure how I missed this setting in the first place, but it actually worked fine with the 4.4.x release so I guess I simply forgot about it. 

Anyhow - thank you so much for your prompt response. Saved my day as I am going to present Wazuh as a PoC for my team and manager later this week, and having a working setup is always preferred for those moments... ;-)

Best regards,

Oliver

[Gonzalo Acuña]

Hi,
Glad to hear that! Good luck with the PoC.

Regards.
Gonzalo.