wazuh-alerts-3.x index is created in wazuh 4.x

[Slava G]

Hi,

I recently noticed that my elasticsearch cluster is yellow, I found that there's an index created 

wazuh-alerts-3.x-2021.04.18 every time I'm refreshing events view in the Wazuh app in Kibana. While I'm using wazuh 4.1.4 and index wazuh-alerts-3.x-2021.04.18  also exists. So, why is the index wazuh-alerts-3.x-2021.04.18 created ? 

Thanks

[Victor Moreno Jimenez]

Hi @slavago,
If your Elasticsearch node is creating wazuh-alerts-3.x indices means that you have loaded the old wazuh-template.json. To check which templates your Elasticsearch node have, you could use this API call (remember to replace admin:admin with your credentials and localhost with your Elasticsearch node IP):

[root@centos7 filebeat]# curl -k -u'admin:admin' -XGET https://localhost:9200/_cat/templates 
wazuh    [wazuh-alerts-4.x-, wazuh-archives-4.x-] 0 1  
wazuh-agent [wazuh-monitoring-*]

If you have there wazuh-alerts-3.x you could delete it using the API as well:

[root@centos7 filebeat]# curl -k -u'admin:admin' -XDELETE https://localhost:9200/_index_template/wazuh-alerts-3.x

Also, check if you have that template in your filebeat configuration and remove it. Check the template you are using in your filebeat node. In /etc/filebeat/filebeat.yml make sure you have configured the 4.x template.

setup.template.json.path: '/etc/filebeat/wazuh-template.json'

You could get more info about opendistro API and templates here https://opendistro.github.io/for-elasticsearch-docs/docs/elasticsearch/index-templates/
Hope it helps!
Víctor.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/CAP6s8-rMdqWAhSe56vLO%2BZGSaUeV%3Dvemnzo0%3Dif54EfW4RSohA%40mail.gmail.com.

[Slava G]

Hi Victor,

I checked templates in Elastic:

wazuh                             [wazuh-alerts-4.x-*, wazuh-archives-4.x-*]   0          1  

I also checked in filebeat:

"index_patterns": [
    "wazuh-alerts-4.x-*",
    "wazuh-archives-4.x-*"
  ]

So, nothing contains wazuh-alerts-3.x

But they're continued to be created.

Thanks

[Victor Moreno Jimenez]

Hi Slava,
Please, if you upgraded Wazuh from 3.x make sure you have followed our guide https://documentation.wazuh.com/current/upgrade-guide/elasticsearch-kibana-filebeat/upgrading-open-distro.html.

Pay special attention to the 12th step where the index pattern is removed:

curl 'https://<kibana_ip>:<kibana_port>/api/saved_objects/index-pattern/wazuh-alerts-3.x-*' -X DELETE  -H 'Content-Type: application/json' -H 'kbn-version: 7.10.0' -k -uadmin:admin

Hope it helps!

Víctor.

[Slava G]

Hi Victor,

Well, it doesn't :(

{"statusCode":404,"error":"Not Found","message":"Saved object [index-pattern/wazuh-alerts-3.x-*] not found"}

Thanks

[Slava G]

In addition, 

Seems that index is created without template, as it is created with default replica 1, that is not true in the template. Indexes for Wazuh 4 created with replica 0.

Thanks

[Victor Moreno Jimenez]

Hi Slava,
Could you please check if everything is correct in the Wazuh filebeat module? It is located in /usr/share/filebeat/module/wazuh. Please check that your file: /usr/share/filebeat/module/wazuh/alerts/manifest.yml looks like:

[root@wazuh-manager-master alerts]# cat /usr/share/filebeat/module/wazuh/alerts/manifest.yml 
module_version: 0.1 

var: 
  - name: paths 
    default: 
      - /var/ossec/logs/alerts/alerts.json 
  - name: index_prefix 
    default: wazuh-alerts-4.x- 

input: config/alerts.yml 

ingest_pipeline: ingest/pipeline.json

Greetings,
Víctor.

[Slava G]

Hi Victor.

This is tec content of the file:

------------------

module_version: 0.1

var:
  - name: paths
    default:
      - /var/ossec/logs/alerts/alerts.json
  - name: index_prefix
    default: wazuh-alerts-4.x-

input: config/alerts.yml

ingest_pipeline: ingest/pipeline.json

---------------------

Thanks

[Slava G]

Hi Victor,

Is there's anything else I can check to fix this issue ?

Thanks

[Slava G]

As of now, after upgrade to 4.1.4 (since April 18) wazuh-alerts-4.x- stopped to be produced and only wazuh-alerts-3.x- indexes are produced now

but they provides and exception in Wazuh app of "invalid argument" and wazuh actually malfunctioning for 2 days already.

Please advise.

[Slava G]

Well, I think I found the trouble maker, it was a logstash that left after upgrading to 4.x but it was not running, I did a restart before upgrading to 4.1.4 and it's started to run.

Now need to understand why wazuh-alerts-4.x stopped to produce. Any suggestions?

Thanks

[Alberto Rodriguez]

Hello 

Let's try to clarify the scenario. You did an upgrade (I assume that it was following the guide https://documentation.wazuh.com/current/upgrade-guide/index.html). 
If an old Logstash was working and pointing to the Elasticsearch node after the upgrade, it's normal that the node has a 3.x index. I think that you don't need it, so you can remove it by

curl -k -u'admin:admin' -XDELETE https://localhost:9200/wazuh-alerts-3.x-*

Then, we need to investigate why the new alerts are not sent to Elasticsearch. We know that Elasticsearch is working, so it's possible that:

- Filebeat is not working. Could you please check the command: filebeat test output ?  Please check also the logs in /var/logs/filebeat/filebeat in order to determine if filebeat is sending correctly alerts to Elasticsearch. systemctl status filebeat could help also. 

- Wazuh manager is not creating alerts in alerts.json. Please check with systemctl status wazuh-manager and let's see if there is any problem. 

The checks above are required in the Wazuh server.