Error 429 too many requests

[Wazuh | Mailing List]

Hi,
While on wazuh-dashboard, I had an error of too many request and now I can't access it anymore.
I test the connection with "filebeat test output" and I have this output :

root@wazuh-2:/usr/share/wazuh-dashboard/data# filebeat test output
elasticsearch: https://10.0.43.196:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 10.0.43.196
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... OK
  version: 7.10.2
elasticsearch: https://10.0.43.197:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 10.0.43.197
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... ERROR 429 Too Many Requests: {"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [1024672040/977.2mb], which is larger than the limit of [1020054732/972.7mb], real usage: [1024672040/977.2mb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=18317/17.8kb, in_flight_requests=1000988/977.5kb]","bytes_wanted":1024672040,"bytes_limit":1020054732,"durability":"TRANSIENT"}],"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [1024672040/977.2mb], which is larger than the limit of [1020054732/972.7mb], real usage: [1024672040/977.2mb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=18317/17.8kb, in_flight_requests=1000988/977.5kb]","bytes_wanted":1024672040,"bytes_limit":1020054732,"durability":"TRANSIENT"},"status":429}

Can you help me please ?
Thanks in advance

[Maxray]

I could launch wazuh dashboard but the error is still there :

Version: 2.6.0 Build: 45202 Error: Too Many Requests at Fetch._callee3$ (https://10.0.43.196/45202/bundles/core/core.entry.js:15:584549) at tryCatch (https://10.0.43.196/45202/bundles/plugin/customImportMapDashboards/customImportMapDashboards.plugin.js:13:760622) at Generator.invoke [as _invoke] (https://10.0.43.196/45202/bundles/plugin/customImportMapDashboards/customImportMapDashboards.plugin.js:13:764638) at Generator.next (https://10.0.43.196/45202/bundles/plugin/customImportMapDashboards/customImportMapDashboards.plugin.js:13:761817) at fetch_asyncGeneratorStep (https://10.0.43.196/45202/bundles/core/core.entry.js:15:577641) at _next (https://10.0.43.196/45202/bundles/core/core.entry.js:15:577957)

[Wazuh | Mailing List]

Hello Maxray,

The circuit_breaking_exception is a mechanism used to prevent operations from causing an OutOfMemoryError. The error seems like Elasticsearch was using most of the JVM heap configured, and the total memory required for all operations was superior to the memory available, so the operation you requested was aborted.

In this case, we should suggest increasing the heap size as Elasticsearch forums/Wazuh Docu suggest:
https://documentation.wazuh.com/current/user-manual/elasticsearch/elastic-tuning.html#memory-locking

Note: If you already have 50% of the available RAM defined, it is not recommended to continue increasing it. In this situation, one of the following options should be followed:

- Consider scaling the cluster to obtain more JVM memory to support your workload. This means adding more nodes to the Wazuh-indexer/Elasticsearch cluster.
- If cluster scaling isn't possible, try reducing the number of shards by deleting old or unused indices. Because shard metadata is stored in memory, reducing the number of shards can reduce overall memory usage.
- Change the shards configuration for indices: https://www.elastic.co/es/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster

Regards.

[Wazuh | Mailing List]

Thank you for your answer, I will check that.
Right now, I have 3 wazuh-indexer and we will probably have, in the end, by 1800 workstations.
We used the table in the doc to calculate what we need. But it seems that's not good, do you think we should add more wazuh indexer so ?

[Maxray]

I'm using Wzuh central component and not elasticsearch. I don't have /etc/elasticsearch/elasticsearch.yml so...
Should I do something else ?

[Wazuh | Mailing List]

Problem solved, I edit the ram in /etc/wazuh-indexer/jvm.options to 50%

Thanks for your help