SIEM Internal Server error at login

[Matt Colucci]

Hello! I just started getting this error message at login on my SIEM. Just noticed this morning too that I am no longer getting recent events. Any help on resolving this error is much appreciated.

_construct@https://siem/36136/bundles/core/core.entry.js:6:4859
Wrapper@https://siem/36136/bundles/core/core.entry.js:6:4249
_createSuperInternal@https://siem/36136/bundles/core/core.entry.js:6:3388
HttpFetchError@https://siem/36136/bundles/core/core.entry.js:6:6016
_callee3$@https://siem/36136/bundles/core/core.entry.js:6:59535
tryCatch@https://siem/36136/bundles/plugin/opendistroQueryWorkbenchKibana/opendistroQueryWorkbenchKibana.plugin.js:1:32004
invoke@https://siem/36136/bundles/plugin/opendistroQueryWorkbenchKibana/opendistroQueryWorkbenchKibana.plugin.js:1:35976
defineIteratorMethods/</prototype[method]@https://siem/36136/bundles/plugin/opendistroQueryWorkbenchKibana/opendistroQueryWorkbenchKibana.plugin.js:1:33129
fetch_asyncGeneratorStep@https://siem/36136/bundles/core/core.entry.js:6:52652
_next@https://siem/36136/bundles/core/core.entry.js:6:52992

[Damian Nicastro]

Hello @mcolucci

I hope you are fine.

Could you tell me from where did you get this JS error? This does not say much about the problem.

Please, check for errors in Elastic and Kibana and send me the results:

# less /var/log/elasticsearch/<node-name>.log | grep -iE 'WARN|ERROR'

# journalctl -u kibana --no-pager | grep -iE 'WARN|ERROR'

I hope this helps.

Thanks

[Matt Colucci]

Hello Damian,

Thanks for reaching out. I get this error whenever I login to the wazuh dashboard. I've attached two TXT files with the results from the commands you provided. Please let me know what else you need from me. Thanks again for your help.

[Damian Nicastro]

Hi @mcolucci:

I hope you are fine.

The reason because you are not getting new information in the Wazuh-dashboard is because the disk occupation in the Wazuh-indexer machine has exceed the water mark:
Aug 18 15:19:59 SIEM kibana[343]: {"type":"log","@timestamp":"2022-08-18T19:19:59Z","tags":["error","elasticsearch","data"],"pid":343,"message":"[cluster_block_exception]: index [.kibana_1] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];"} Aug 18 15:19:59 SIEM kibana[343]: { ResponseError: index [.kibana_1] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];: cluster_block_exception Aug 18 15:19:59 SIEM kibana[343]: name: 'ResponseError',

This means that the limit of 85% of disk occupation has been reached. From that point, the wazuh-indexer stops accepting events from filebeat service and keep running as read only. This is used to prevent disk flooding.

You can confirm this, login in to the wazuh-dashboard, clicking on "three lines" menu, then on "Dev_tools". There you can type this API query:
GET /cat/allocation?v

This watermark can be removed from wazuh-indexer configuration, but this is not recommended because the disk can be overloaded and many processes in the machine will stop.

The solution is to delete old indices and/or indices that are normally not used like "wazuh-statistics", for instance in "DEv_tools":

DELETE wazuh-alerts-4.x-2022.06.*

DELETE wazuh-statistics-*

Once you get enough space, you can recover the information that was not inserted using the recovery script and follow in detail the instructions:

https://wazuh.com/blog/recover-your-data-using-wazuh-alert-backups/

In order to prevent this, you should implement the retention policies in order to automatically delete indices after certain period of time:

Wazuh index management · Wazuh · The Open Source Security Platform

Please, note that in this document, the first part is for Elasticsearch (ILM) and the second part is for Opendistro/Opensearch (ISM).

In the case that your disk is not big enough and you want to migrate the wazuh-indexer data to another FileSystem, you can stop the wazuh-indexer service, copy all the data (inidces info) and logs and configure the wazuh-indexer to save the information in the new FileSystem:

# systemctl stop wazuh-indexer

# cp -rp /var/lib/wazuh-indexer/* /path/to/new_fs/

# cp -rp /var/log/wazuh-indexer/* /path/to/new_fs2/

# vi /etc/wazuh-indexer/opensearch.yml

...

path.data: /path/to/new_fs/
path.logs: /path/to/new_fs2/

...

# systemctl start wazuh-indexer

[Matt Colucci]

Hello Damian,

This is great information! Thank you very much! I will adjust my SIEM based on your recommendations. Thank you again for all your help!

[Damian Nicastro]

Hello mcolucci:

I am glad I could help you out!

Have a nice weekend!

Thanks