Elasticsearch error

[Mark Rafa]

Hello, 

I know it is a basic question, but I am new in wazuh and elasticsearch.

I cant see logs on kibana due to a elasticsearch error.

I have these logs below in my elasticsearch.log:

Elasticsearch is not opening and I have this log below in elasticsearch.log:

GC did not bring memory usage down, before 

[2021-01-13T12:45:54][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [elastic] attempting to trigger G1GC due to high heap usage 

Can anyone please help me how to solve this situation.

Thanks in advance,

 

[Miguel Casares]

Hello Markrafa,

The Circuit Breaker is a mechanism used to prevent operations from causing an OutOfMemoryError. It seems like at the moment you made your query, Elasticsearch was using most of the JVM heap configured, and the total memory required for all operations was superior to the memory available, so the operation you requested was aborted.

The parent-level breaker can be configured with the following settings:

indices.breaker.total.use_real_memory Static setting determining whether the parent breaker should take real memory usage into account (true) or only consider the amount that is reserved by child circuit breakers (false). Defaults to true. indices.breaker.total.limit Starting limit for overall parent breaker defaults to 70% of JVM heap if indices.breaker.total.use_real_memory is false. If indices.breaker.total.use_real_memory is true, defaults to 95% of the JVM heap.

Source: https://www.elastic.co/guide/en/elasticsearch/reference/master/circuit-breaker.html

If you are using the default configuration, it means that when you ran your query, the total memory required exceeded 95% of the JVM heap.

Taking all this into account, the conclusion is that there are two possibilities:

• The Elasticsearch machine does not have enough resources to run all your queries.

• The JVM heap configuration is not optimal.

Please, could you share the following information with me?

• The output of the commands free -h and ps aux | grep elasticsearch.

• Your configuration files /etc/elasticsearch/elasticsearch.yml and /etc/elasticsearch/jvm.options.

Reference: https://documentation.wazuh.com/4.0/user-manual/elasticsearch/elastic_tuning.html

Additionally, I would check the number of shards in your environment. The Elasticsearch performance is strictly related to the number of shards and the data that your servers are handling so if you can provide information about that we can determine if it is a healthy environment. More information: https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster 

Let us know if you have any questions.

Regards,

Miguel Casares

[Mark Rafa]

Hello Miguel,

Sorry for late response.

Thanks for your interest and guidance. 

Attached you can find the files you have asked for. 

I have not changed anything after installing elasticsearch.

My environment: 

Wazuh 3.13

Centos 8 wazuh 4vcpu, 8gb ram

Centos 8 elasticsearch, kibana  8vcpu, 16 gb ram

According to shards question, how can i find how many shards do i have in my environment?

Hope these informations help to solve the problem.

Regards, 

13 Ocak 2021 Çarşamba tarihinde saat 20:40:42 UTC+3 itibarıyla miguel....@wazuh.com şunları yazdı:

[Mark Rafa]

Hi,

Does anyone have a idea how to solve this issue?

I need to fix this.

Regards

18 Ocak 2021 Pazartesi tarihinde saat 10:55:58 UTC+3 itibarıyla Mark Rafa şunları yazdı:

[Miguel Casares]

Hello Mark,

Sorry for the late response.

As you can see in the configuration file provided, you only have configured 1 GB of memory for Elasticsearch. This is not the proper configuration and you should change it up to 7 GB which is based on the total memory of your environment.

You need to change this:

# Xms represents the initial size of total heap space 

# Xmx represents the maximum size of total heap space 

-Xms1g 

-Xmx1g

To this:

# Xms represents the initial size of total heap space 

# Xmx represents the maximum size of total heap space 

-Xms7g 

-Xmx7g

in the /etc/elasticsearch/jvm.options file and then restart Elasticsearch:

systemctl restart elasticsearch

It will solve the issue.

Reference: https://documentation.wazuh.com/4.0/user-manual/elasticsearch/elastic_tuning.html

To check the number of shards, you can use the Elasticsearch API: https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-shards.html

I hope this helps. Let me know if you need anything else.

Regards,

Miguel Casares

[Mark Rafa]

Thanks a lot Miguel!

Regards, 

2 Şubat 2021 Salı tarihinde saat 19:21:49 UTC+3 itibarıyla miguel....@wazuh.com şunları yazdı:

[Miguel Casares]

Hello Mark,

You're welcome!

Don't hesitate to contact us should you need it.

Regards,

Miguel Casares

[Majid Ibrahim]

Thanks, Miguel.

This has helped me also.

[Faruk Altun]

buda bir çözüm ama muhtemelen jvm bellek yetersizliği ile de uğraşıyorsunuz. 

jvm belleğini 2 gb ve üzerine çıkarttığınızda bütün sorunlar çözülecektir. 

3 Şubat 2021 Çarşamba tarihinde saat 18:43:57 UTC+3 itibarıyla majid....@gmail.com şunları yazdı: