Indexing question & Index issues

[Chad Linderman]

Good Morning,

I have a couple of problems with my Wazuh appliance I hope someone can help me out please.

First my health is all green, so that is good.

However, Wazuh has not been creating any new indices for a week now. I have run the recovery.py script and that worked in the past. However, this is not working for these missing indices. I don't get any errors when I check the log file for the recovery script. It seems as though it's not getting consumed.

I also noticed that the previous Wazuh admin didn't set up any index management. The network Wazuh runs on it small (less than 8 end points), but we're not allowed to purge any data. My question would be, if I do an Index management plan can I either not enable a delete phase or just not create that portion at all?

Lastly, the missing data seems to be an ongoing monthly occurrence. It doesn't seem to me that the recovery.py script should be used so often. Would an index management plan resolve this issue?

Thank you,

Chad

[John Soliani]

Hello,

Thank you for posting in our community and using Wazuh.

It’s highly likely that you have run out of shards. There are two things really important regarding the DB when it comes to Elasticsearch. On the one hand, DISK SPACE, data needs space on the disk, nothing new here but on the other hand, we have SHARDS, since Elasticsearch is oriented to be used in a cluster environment, it uses shards (fragments) to store data in the nodes and by default, each node has a maximum limit of 1000 shards. If you’ve reached the max limit, FB won’t be able to parse new data in Elasticsearch, no matter if it comes from the alerts.json (wazuh-manager) or the recovery.json (recovery script).

• curl -k -u <UserName>:<Password> https://<ElasticIP>:9200/_cat/allocation?v
  You will get something like this:

  shards disk.indices disk.used disk.avail disk.total disk.percent host           ip             node
    93       50.4mb    10.2gb      9.3gb     19.5gb           52 192.168.11.100 192.168.11.100 inode-1
  

The first collum is the shards used per node, one line per node in the cluster, in my case I have only 1 elastic node and 93 shards active and in use, bear in mind that unassigned shards (in the last row, if there’s any, I don’t have any) also consume shards in the node. If this is the case, you can free shards by deleting old indices, for instance:

To delete ALL indices from January of 2021, we can use:

• curl -k -u <User>:<Password> -X DELETE https://<ElasticIP>:9200/*2021.01*
  To delete ALL indices from 2021:
• curl -k -u <User>:<Password> -X DELETE https://<ElasticIP>:9200/*2021*

If you need to check indices before deleting them, use:

• curl -k -u <User>:<Password> -X GET https://<ElasticIP>:9200/_cat/indices/*2021.*?v
  You’ll see something like this in return:

  health status index                        uuid                   pri rep docs.count docs.deleted store.size pri.store.size
  green  open   wazuh-alerts-4.x-2021.05.13  819gIjqIQCSdnvvWO8E4DQ   1   0        468            0    616.2kb        616.2kb
  green  open   wazuh-alerts-4.x-2021.05.14  ajB_eIaSTEWhNtaF7GgyRA   1   0          1            0       12kb           12kb
  green  open   wazuh-alerts-4.x-2021.05.17  k2T_PLg0SPmtxBJlIfIU5Q   1   0         87            0    169.9kb        169.9kb
  green  open   wazuh-alerts-4.x-2021.06.29  H7YFXIzBRzSg0vjs_axtNg   1   0          4            0     31.1kb         31.1kb
  green  open   wazuh-alerts-4.x-2021.05.18  YuoQ4WRVRF2ycKrneUWTyA   1   0          1            0       12kb           12kb
  green  open   wazuh-alerts-4.x-2021.08.09  h7qPTOB2Qsy0PL362eEBrg   1   0          9            0     69.7kb         69.7kb
  

  Also, you could increase the max_limit of shards per node BUT THIS IS NOT RECOMMENDED as a real solution to the problem and could lead to an unstable performance in the DB and even to data loss, so do not increase this value over a 20/30 percent.

• curl -k -u <User>:<Password> -XPUT https://<ElasticIP>:9200/_cluster/settings -H "Content-Type: application/json" -d '{"persistent":{"cluster.max_shards_per_node":"1200"}}'

After this is done, you should automate the DB cleaning with a retention policy, otherwise, the DB will store data till there’s no more available space left on the disk or you reached the max shards limit again. We have a blog that will help you in this process, make sure you follow the guidelines for Elasticsearch Stack (the first one), called ILM (Index Lifecycle Management), or the guide for OpenDistro/Wazuh-Indexer (called ISM - Index State Management), the second guide. Here’s the link.

• Wazuh Index Management.

Finally, if you could share with me how many Elastic nodes you have and send me one sample of one wazuh-alerts-X index, I could probably help you to optimize the shards usage in your environment.
A sample like this one: green open wazuh-alerts-4.x-2021.05.13 819gIjqIQCSdnvvWO8E4DQ 1 0 468 0 616.2kb 616.2kb

Let me know if this helped!
John.-

​

[Chad Linderman]

Thank you for your help & warm welcome John.

Currently this system is in an air-gapped network. So I will have to print out a log & scan it. Due to security, we cannot bring any data out.

I also am unable to delete any data due to security. I wouldn't think there's that much data, since we only have less then 8 end-points. However, I will check. If I delete an index, won't that delete data too?

I have the Index Management document printed out. However, I had a question about it: If I do an Index management plan can I either not enable a delete phase or just not create that portion at all?

Thank you,

Chad

[John Soliani]

Hi Chad,

I see, no problem. Yes, deleting old indices means deleting data, there are three things you can do to increase the available shards in your Elastic, delete indices to free shards, expand the resources horizontally in the Elastic cluster (by adding one or more nodes) or increase the max_shards_per_node value (Elasticsearch does not recommend doing so, do it under your own risk).

Maybe I wasn’t clear enough, the amount of data is one thing, another is the shards you are using to store indices in Elastic. One index could use 3 shards and 1 MB of disk space or 1 shard and 10 GB of disk space, the amount of data you are sending will affect the space on the disk, not the shards.

You could have an ILM/ISM policy without a deletion phase, but remember, you’ll be storing data forever and you have limits on the DB usage, so by doing this, eventually, you’ll fall to the point you are now, out of shards or disk space.

Regards,
John.-

​

[Chad Linderman]

Thank you so much for you help John. I see a lot of great information going around, but some of it is a bit difficult to follow or skim through for a complete novice like myself.

It doesn't look like space is an issue. I checked my HDD Space and it was fine. Additionally, I am showing a free space in the about of ~14GB.

Thank you,

Chad

[John Soliani]

Hey Chad,

Glad to help!

Space is not the issue … now, it’s a really small partition though, only 40 GB so keep an eye on it. The problem here is shards, we can see you already increased the maximum to 1500, this is not recommended by the vendor.
To free shards, you can delete old indices or add more nodes to the elastic cluster.
To check your max shards settings run this:

• curl -k -u <User>:<Password> https://<ElasticIP>:9200/_cluster/settings

NOTE: If you don’t have shards available in Elastic, you won’t be able to add new data to the DB.

Regards,
John.-

​

[Chad Linderman]

Thank you John. This is what is returned when I run that command (If there's a typo, it's because I typed it from the print out):

{"persistent":{"cluster":{"max_shards_per_node":"1500"}}, "transient":{}}[wazuh@wazuh-manager /]

One thing I did find in the Wazuh logs is an Index error. From what I can see, this looks exactly like what you're getting at; I don't have enough shards.

Sep 7, 2022 @ 08:15:00 ERROR Could not create wazuh-monitoring-2022.09.07 index on elasticsearch due to validation_exception
Sep 7, 2022 @ 08:15:00 ERROR index_not_found_exception

Is there any issue with adding another node on a single cluster? I found these instructions: https://documentation.softwareag.com/webmethods/compendiums/v10-5/C_API_Management/index.html#page/api-mgmt-comp/ta-add_node_to_cluster.html

Or would it be better to just go ahead and create & execute an Index Management Plan? I have to do one anyways. The system I inherited it from, the gentleman did not ever create one.

Thank you,

Chad

[Chad Linderman]

Scratch the idea of creating an Index Maintenance Plan. It says I don't have enough Shards to even do that. :(

[Chad Linderman]

Hi John,

Right now we only have one Node in our Wazuh environment. It's also not clustered. Is there any issue with just adding a new node if I have the space?

Thank you,

Chad

On Wednesday, September 7, 2022 at 6:55:16 AM UTC-7 john.s...@wazuh.com wrote:

[Cyberguy21793]

Chad, 

Just went through this.... and yes it is kind of a circle of pain, but it is really easy to correct once you figure out the sequence. One thing that will help you out is to understand what a shard is and what it is not. 

In Splunk data is held in the indexes if you delete them the information is gone. In wazuh, the base information (LOGS) are not held in the indexes (Shards). Indexes in Wazuh are the base logs/info contextualized and "marked up" or prettified I guess you could say making them searchable. So what does that mean? it means you can safely delete old indexes (enough of them) to get into the Gui and implement some index management policies. 

I don't know what version you have so this part will sound a little generic. What I did was ssh in and use the cli commands to delete all indexes from 2021. This was 2 weeks ago so I wasn't too concerned that anyone was actually searching that data. That deleted enough indexes(shards) to get me in to the GUI. I then created 3 index management policies for the montoring indexes, statistics indexes and Alerts indexes. I copied the example in the documentation and edited it just a little. For the monitoring and statistics I set the indexes to go to cold after 180 and delete after 270 days. For the alerts I set policy to move them to cold after 180 days and delete after 365. Then assigned the policies to the indexes and let wazuh manager to figure out and manage the rest for me.

so from the command line using the wazuh admin user. Personally I kept getting errors using any other ones. I cheated and looked in the filebeat.yml and found the user and password....I used that one and it worked. 8^)

curl -k -u <User>:<Password> -X DELETE https://<ElasticIP>:9200/wazuh-statistic -2021* like that for alerts, statistics, and montoring. That should free up enough shards to do what you need. 

I am assuming that you are using a single node and you have under 100 agents reporting. If my assumptions are wrong then you need additional help. Over 100 agents, you need another node. Already have more than 1 node then verify that what I told you to do won't mess up your cluster.

Hope that helps,

Rex