Wazuh Indexer - Circuit Breaker Errors

[sang thanh]

Hi teams,

I using Wazuh which have version 4.3.9, everything working pretty smooth, well done team.

Btw I just wanna ask about the Circuit Breaker Errors which belong to the Wazuh Indexer when I visualize the data from Grafana, the full logs I put in the attachment file.

Changed the total heap space from ~ 2GB to 4GB via /etc/wazuh-indexer/jvm.options file:

# -Xms1931m

# -Xmx1931m

-Xms4g

-Xmx4g

And also changed the indices.breaker.total.limit to 4GB but still not resolved my issue.

Do you guys have any idea for this issue?

Thanks.

[Gabriel Diaz Lopez de la Llave]

Hello!

Wazuh-indexer is based on opensearch. There is a nice guide about how to tune your environment to avoid these exceptions in https://aws.amazon.com/premiumsupport/knowledge-center/opensearch-circuit-breaker-exception/

also, the guide at https://documentation.wazuh.com/current/user-manual/elasticsearch/elastic-tuning.html is applicable too for this case.

From the error message you posted, it doesn't look like the jvm.options are in effect. I guess you restarted the service after changing the jvm.options right? How much memory does the server have? 

To double-check what's the actual heap of the indexer, you can use the Dev tools, from the left menu in the wazuh dashboards, and run the query:

GET _cat/nodes?h=heap*&v

This should give you an answer like:

heap.current heap.percent heap.max
       4.6gb           58    7.8gb

But with the heap size you specified.

You can also check it wirh a curl command like:

$ curl -k -u admin  "https://localhost:9200/_cat/nodes?h=heap*&v"
Enter host password for user 'admin':
heap.current heap.percent heap.max
     686.5mb            8    7.8gb

Another way to check it could be to inspect the gc.log file like:

# grep -i "gc,init" /var/log/wazuh-indexer/gc.log
[2023-03-15T14:15:43.414+0000][4573][gc,init] Version: 17.0.5+8 (release)
[2023-03-15T14:15:43.414+0000][4573][gc,init] CPUs: 8 total, 8 available
[2023-03-15T14:15:43.415+0000][4573][gc,init] Memory: 16008M
[2023-03-15T14:15:43.415+0000][4573][gc,init] Large Page Support: Disabled
[2023-03-15T14:15:43.415+0000][4573][gc,init] NUMA Support: Disabled
[2023-03-15T14:15:43.415+0000][4573][gc,init] Compressed Oops: Enabled (Zero based)
[2023-03-15T14:15:43.415+0000][4573][gc,init] Heap Region Size: 4M
[2023-03-15T14:15:43.415+0000][4573][gc,init] Heap Min Capacity: 8004M
[2023-03-15T14:15:43.415+0000][4573][gc,init] Heap Initial Capacity: 8004M
[2023-03-15T14:15:43.415+0000][4573][gc,init] Heap Max Capacity: 8004M
[2023-03-15T14:15:43.415+0000][4573][gc,init] Pre-touch: Enabled
[2023-03-15T14:15:43.415+0000][4573][gc,init] Parallel Workers: 8
[2023-03-15T14:15:43.415+0000][4573][gc,init] Concurrent Workers: 2
[2023-03-15T14:15:43.415+0000][4573][gc,init] Concurrent Refinement Workers: 8
[2023-03-15T14:15:43.415+0000][4573][gc,init] Periodic GC: Disabled

Let us know if this is helpful! 

[sang thanh]

Hello Gabriel Diaz Lopez de la Llave,

Thanks for your kind response, the edit in jvm.options is worked. Before that I make a mistake when changing in another Wazuh server and check result in another.

One more time, thanks a lot.

Vào lúc 15:59:39 UTC+7 ngày Thứ Năm, 16 tháng 3, 2023, Gabriel Diaz Lopez de la Llave đã viết: