Disk usage by /var/lib/wazuh-indexer

[Łukasz Podgórski]

First, thank You for developing Wazuh.

I'm facing a problem that /var/lib/wazuh-indexer is taking too much space. Is there a way to limit it someway, or rotate it? Sure I can enlarge partition to make some space, but I can't  do it forever.

Thank in advance

[Marcos Darío Buslaiman]

Hi Lukas,
Thank you for using Wazuh!, 
The path  /var/lib/wazuh-indexer contains the indexes with the information of the alerts ( wazuh-alerts-4.x-YYYY.MM.DD  ) and other internal information like statistics and monitoring of process etc.

You can define a period of retention of these indexes to manage the disk space, on the following blog you have a complete explanation of this index management.
https://wazuh.com/blog/wazuh-index-management/
One important thing to mention is that you always need to manage the indexes through the API management or Wazuh-indexer, never should be deleted or modified from the CLI console of the server, due to could generate some inconsistencies.
If you need to list the indexes and check the size you can execute the following from Menu --> Management --> Dev Tool

To list all indeces 
GET /_cat/indices?v=true&s=index

Or if you need only the wazuh-alerts

GET /_cat/indices/wazuh-alerts-*?v=true&s=index

[Marcos Darío Buslaiman]

Sorry, I sent the email without finishing it.

To list all indices 
GET /_cat/indices?v=true&s=index

Or if you need only the wazuh-alerts

GET /_cat/indices/wazuh-alerts-*?v=true&s=index

To delete one execute the following
DELETE wazuh-alerts-4.x-2023.02.06
 
But I recommend you define policies to delete according to a period of retention.
Please let me know if you have any doubts or questions, and I'll be glad to help you.

Marcos

[Łukasz Podgórski]

I've tried to configure retention  of indexes (https://wazuh.com/blog/wazuh-index-management/), but I can't find Index Lifecycle Policies. I can't find Elastic Search web console.

[Marcos Darío Buslaiman]

Hi  Łukasz Podgórski,
Sorry for the delay, I would like to know more about your environment, Could you share your version of the Wazuh-indexer and manager?

To check that you can go to the CLI console and execute the following according to your SO.
 yum list installed | grep -i wazuh
Or 
apt list | grep -i  wazuh

To access the Index Lifecycle Policies, you need to go to the menu and select Index Management and then you will have the Index Policies access, 
 Please check the following image 

Please just let me know for any other questions.
Regards.

[Łukasz Podgórski]

Thank You for the reply.

I'm using latest Wazuh version installed on Ubuntu, self hosted.

wazuh-agent/stable 4.3.10-1 amd64
wazuh-dashboard/stable,now 4.3.10-1 amd64 [installed]
wazuh-indexer/stable,now 4.3.10-1 amd64 [installed]
wazuh-manager/stable,now 4.3.10-1 amd64 [installed]

I do have index management console, it does looks just like in Your screenshot. I just can't set set index policies with this tutorial: https://wazuh.com/blog/wazuh-index-management/ <---- I don't have such options.

I have so much more questions (including multitenency), but right now I'm facing the avalanche of errors like these in attached screenshots. Is there anything I can do about it? I've tried clearing cache in browser, using Wazuh in incognito, etc. Sometimes only restarting wazuh-dashboard helps, but only for moment. Also after machine reboot wazuh-indexer service fails to start on it's own, and must be started manually. Is my wazuh install broken?

Thank You for Your help.

[Łukasz Podgórski]

I'll provide some more info:

My install is ~5 months old, I have ~200 agents, Wazuh is installed on Proxmox VE (virtual machine ~500gb ssd, 8 gb ram, 8 cores, ubuntu 22).

[Marcos Darío Buslaiman]

Hi Lukasz,
I will be checking your logs error, but in the meantime, could you show me the screen of your index policy page, to check which options you have?
Regards

[Łukasz Podgórski]

It just looks different (from blog post)

[Marcos Darío Buslaiman]

Hi Lukasz,
On that page that you have, you may select create policy, this is a different UI than the blog but is the same functionality.
So, when you select create the policy you will get the option to create using a Visual Editor or Json Editor.

If you select the Json Editor, you can use the template that is under the topic "Configuration of Index State Policies" on the blog.
You can paste the template on your policy definition, and modify it according to your needs, taking into account some concepts:
About states:

• Hot—​the index is actively being updated and queried.
• Warm—​the index is no longer being updated, but is still being queried.
• Cold—​the index is no longer being updated and is seldom queried. The information still needs to be searchable, but it’s okay if those queries are slower.
• Delete—​the index is no longer needed and can safely be deleted.

You can check this for more information about it:
https://www.elastic.co/guide/en/elasticsearch/reference/7.0/index-lifecycle-management.html
https://opendistro.github.io/for-elasticsearch-docs/docs/im/ism/policies/

If you select the Visual editor it is the same, but you will need to create the states (Hot, cold,) and set the transitions to the delete action.

Once you have your policy created you need to apply this policy to your index, and follow the next steps of the blog.

Please let me know if you have some doubts or questions regarding this.
I will continue checking your issue regarding the UI error

Regards

[Marcos Darío Buslaiman]

Hi Lukasz

Regarding the issue of "circuit_breaking_exception", I found that this exception is a mechanism used to prevent operations from causing an OutOfMemoryError. It seems like Wazuh-Indexer was using most of the JVM heap configured, and the total memory required for all operations was superior to the memory available, so the operation you requested was aborted.
Please, check our documentation to verify this:

• https://documentation.wazuh.com/current/user-manual/elasticsearch/elastic-tuning.html

The steps for Wazuh-Indexer are very similar to the above documentation, and are the followings:

1. Edit /etc/wazuh-indexer/opensearch.yml file, add or edit the following value:

• bootstrap.memory_lock: true

2. Create the following directory:

• mkdir -p /etc/systemd/system/wazuh-indexer.service.d

3. Create this file and add the following value with this command:

cat > /etc/systemd/system/wazuh-indexer.service.d/wazuh-indexer.conf << EOF 

[Service] 

LimitMEMLOCK=infinity 

EOF

4. Edit /etc/wazuh-indexer/jvm.options with the memory values you desire, you have to modify the -Xms and -Xmx values and remember that the min and max values should be the same.:

# Xms represents the initial size of total heap space 

# Xmx represents the maximum size of total heap space 

-Xms4g 

-Xmx4g

5. Reload and restart the wazuh-indexer service

• systemctl daemon-reload
• systemctl restart wazuh-indexer

6. Check that the memory lock is set, (use your IP and credentials)

• curl -k -u admin:{password} "https://{indexer_ip}:9200/_nodes?filter_path=**.mlockall&pretty"

You should see something like this:

{ 

   "nodes" : { 

        "Mu8NLTv3SqKeWtisKU8Y6A" : { 

              "process" : { 

                    "mlockall" : true 

               } 

         } 

    } 

}

Please, let me know if you solve the error with this procedure.

Regards.

[Łukasz Podgórski]

Rising -Xmx and -Xms values in  /etc/wazuh-indexer/jvm.options solved my problems with Wazuh. Now it works really great.

Also, I managed to set Index Polices for time range I desired.

Thank You very much Marcos, for Your time and help.

[Marcos Darío Buslaiman]

Hi Lukasz,
I'm glad to help you.
Please let us know for any other doubts or questions.
Regards!