Cannot see new logs and charts in Kibana

[Max Kalachov]

Hi guys,

About 36 hours ago Kibana stopped showing new logs and charts on the dashboard.

The app version is 4.0.3 and all agents have the same client version 4.0.3

I don't see any errors in the ossec.log.  Following another forum thread, I added wazuh_db.debug = 2 and syscheckd.debug = 2  to /var/ossec/etc/local_internal_options.conf but don't see anything useful so far. I also tried to restart the system.

Wazuh continues sending email notifications about events with the configured alert level 8. So I suppose it's still receiving logs.

Thank you

Regards,

Max

 

[Alejandro Cuellar]

Hello,

To check what may be happening, we are going to carry out the following checks:

- We are going to see if the alerts are arriving correctly at Kibana, to do this, in the container where the manager is hosted, execute the following command (with which we will obtain the last alert):

tail -1 /var/ossec/logs/alerts/alerts.json

Then we will go to the Discover of Kibana and check that precisely that same alert appears.

Then, let's check the data flow between Wazuh and Elasticsearch. It is possible that alerts information is not being sent correctly from Filebeat to Elasticsearch, so you should first check if you have Filebeat correctly configured and alerts are being sent to Elasticsearch.

For this, run these commands:

Check that the filebeat service is running:

systemctl status filebeat

Check if Filebeat has communication with Elasticsearch:

filebeat test output

Check for any possible error in the elasticsearch log:

grep -i error /var/log/elasticsearch/elasticsearch.log

Could you share the output of the above commands (please hide possible sensitive data if any)?

[Max Kalachov]

Hi Alejandro,

I tried to post the reply with all the requested information 5 times. Each time I post the message, it's being deleted.

Can we do something about it?

Thank you

Regards,

Max

[Alejandro Cuellar]

Hi Max,
Thank you very much for your feedback, it seems that Google groups does not allow certain symbols, but I have managed to see all the messages you have sent me.
We see the error clearly in elasticsearch.log that you have passed to us. With the error message "Unable to index audit log". This means you reached the shards limit count (1000 by default in the node)
To fix this issue, you have multiple options:

• Delete indices. This frees shards. You could do it with old indices you don't want/need. Or even, you could automate it with ILM policies to delete old indices as explained in this post: https://wazuh.com/blog/wazuh-index-management.
• Add more nodes to your Elasticserach cluster.
• Increment the max shards per node (not recommneded). But if you do this option, make sure you do not increase it too much, as it could provoke inoperability and performance issues in your Elasticsearch cluster. To do this:

curl -k -u admin:admin -XPUT ELASTICSEARCH_HOST_ADDRESS/_cluster/settings -H "Content-Type: application/json" -d '{ "persistent": { "cluster.max_shards_per_node": "MAX_SHARDS_PER_NODE" } }'

replace the placeholders, where:
- ELASTICSEARCH_HOST_ADDRESS : Elasticsearch host address. Include the protocol https if needed.
- MAX_SHARDS_PER_NODE : Maximum shards by node. Maybe you could try with 1200 o something like that, depending of your case.

[Max Kalachov]

Hi Alejandro,

Thank you very much for the provided solutions. I have some challenges to implement them.

So I followed the Index Management manual, in my case the corresponding part is OpenDistro ISM. When I try to create the policy I get the error: [validation_exception] Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open

I decided to temporarily increase the maximum number of shards, just to create the policy and then change it back. Unfortunately, I'm unable to use the API command to connect to Elastic.

Initially, to deploy this  I used "all in one" unattended installation. Elastic seems to be listening on 127.0.0.1 on the following ports 9200 and 9300 with PID 1186.

[root@wazuh-02-pr1 ~]# systemctl status elasticsearch.service

● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-04-29 09:14:07 PDT; 5 days ago
     Docs: https://www.elastic.co
 Main PID: 1186 (java)
   CGroup: /system.slice/elasticsearch.service
           └─1186 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfi...

[root@wazuh-02-pr1 ~]# netstat -tupan | grep LISTEN

tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1168/sshd           
tcp        0      0 0.0.0.0:55000           0.0.0.0:*               LISTEN      1977/python3        
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1423/master         
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      851/node            
tcp        0      0 127.0.0.1:199           0.0.0.0:*               LISTEN      1179/snmpd          
tcp        0      0 0.0.0.0:1514            0.0.0.0:*               LISTEN      2135/ossec-remoted  
tcp        0      0 0.0.0.0:1515            0.0.0.0:*               LISTEN      2027/ossec-authd    
tcp6       0      0 127.0.0.1:9300          :::*                    LISTEN      1186/java           
tcp6       0      0 :::22                   :::*                    LISTEN      1168/sshd           
tcp6       0      0 ::1:25                  :::*                    LISTEN      1423/master         
tcp6       0      0 127.0.0.1:9600          :::*                    LISTEN      850/java            
tcp6       0      0 127.0.0.1:9200          :::*                    LISTEN      1186/java           
tcp6       0      0 :::9650                 :::*                    LISTEN      850/java

[root@wazuh-02-pr1 ~]# netstat -tupan | grep 1186
tcp6       0      0 127.0.0.1:9300          :::*                    LISTEN      1186/java           
tcp6       0      0 127.0.0.1:9200          :::*                    LISTEN      1186/java  

I tried to connect with and without the HTTP/S, and also tried other ports that the server is listening on (just as a precaution, to make sure I'm not missing anything).

Unfortunately in most cases, I get the same CURL error 7:

Example: curl: (7) Failed connect to 172.0.0.1:9200; Connection timed out

What am I missing or doing wrong?

Thank you

Regards,

Max

[Alejandro Cuellar]

Hi Max,

Well, it seems to me that it can be due to two things, the first, in the curl error that you show us, the ip is 172.0.0.1, when it should be 127.0.0.1, could it be an error when it was written quickly?

We advise you to make the request  from elasticsearch host, so instead of using an IP you can use localhost, which can avoid future errors.

Regards,
Alejandro

[Max Kalachov]

Hi Alejandro,

Thank you very much for pointing to the problem. I actually had this typo in all connection attempts.

So I managed to connect to the API and temporarily change the shards from 1000 to 1200. After that using the provided guide I created a policy, with some time interval changes, 30 and 90 days instead of 30 and a year. The policy has been applied to all existing and future indices and shards were changed from 1200 back to 1000.

Once applied filebeat setup --index-management I got the following message:

ILM policy and write alias loading not enabled.
Index setup finished.

Is it Ok?

I already see charts and logs in Kibana, however, I assume it's due to shards number change because all the old Indices are still there (older than 90 days).

That makes me wonder, how can I see the shards' usage? And how frequently the Index Policies are applied to Indices?

Thank you

Regards,

Max

[Alejandro Cuellar]

Hi Max,

The output message is normal, and the installation was successful.

To see the use of shards, you will have to go to kibana's DevTools and in it, do the following query: GET _cluster/health, which I show you in the screenshot.


As for the policies, if you apply them on an index, they are instantaneous and go cold as modified at 30 days and at 90 days, it would delete the index. Then in the case of using a template, which is applied to all wazuh-alerts- * as soon as the index appears, the policy is already assigned to it. You can also check the life cycle of the IML.

Regards,
Alejandro

[Max Kalachov]

Hi Alejandro,

Thank you for your help.

So since my last post, several things have happened. The day after my post, the charts disappeared from Kibana again. To me, it seemed (it's speculation) like the system used the temporarily added shards rather than the policy. I was seeing the old indices and in the "Managed Indices" I had many failed jobs. Perhaps it's important to mention that I have the same 30/90 days policy applied to wazuh-alerts* , wazuh-alerts* , and wazuh-monitoring*

In one of the old topics in the group, I found API calls to retrieve all the indices:

curl -k -u admin:password https://127.0.0.1:9200/_cat/indices/wazuh-alerts*
curl -k -u admin:password https://127.0.0.1:9200/_cat/indices/wazuh-statistics*
curl -k -u admin:password https://127.0.0.1:9200/_cat/indices/wazuh-monitoring*

So I deleted most of the indices ( wazuh-alerts* , wazuh-alerts* , and wazuh-monitoring* ) older than 90 days, and restarted the failed jobs, and it restored the charts on the dashboard, as well as I see no new failed jobs.

curl -k -u admin:password -XDELETE https://127.0.0.1:9200/wazuh-alerts-4.x-2020.12.*

etc...

Thank you

Regards,

Max

[Alejandro Cuellar]

Hi Max,

As we mentioned, increasing the number of shards was a temporary measure, just as deleting old indices helps you free up the number of shards per node as well.
The policy you created eliminates the indexes 90 days after creating them, therefore, up to 3 months from now, the shards will only continue to increase.
Perhaps lowering the frequency of the policy could help you not have to worry about the node's shards arriving again.

Any other question, do not hesitate to contact us.
Regards,
Alejandro.

[Max Kalachov]

Hi Alejandro,

What do you mean by "lowering the frequency of the policy", changing 30/90 to something bigger like 30/180?

Regards,

Max

[Alejandro Cuellar]

Hi Max,

Yes, sorry, by reducing the frequency of the policy, I wanted to put that frequency every less as 10/30, for example, to try to avoid creating more shards than will fit.

Regards,
Alejandro

[Max Kalachov]

Hi Alejandro,

Understood. The problem though, that many security compliances require storing the logs for 3 months - ~90 days.

Perhaps I could reduce the transition time from "hot" to "cold", but I still need to store those logs for as minimum as 90 days... Are there any other approaches that could improve the situation?

As of now, the shards status is the following:

"cluster_name" : "elasticsearch",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 546,
  "active_shards" : 546,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 255,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 68.16479400749064

Thank you

Regards,

Max

[Alejandro Cuellar]

Hi Max,

It all depends on whether the amount of shards that are generated in a 90-day period for your use case is more or less than the limit of available shards.

If not the use case you have does not give you for all the shards that are generated, you would have to find how to reduce the number of shards with some of this (apart from the index handling policies):

- Setting up generated indexes to use fewer shards

- Adding another node to the Elasticsearch cluster Increasing the maximum shards of the nodes that you have (this if you make it a little and try because it could affect the performance and response of Elasticsearch)

Depending on your use case / needs, it may require more than a policy for index management.

Any other questions, do not hesitate to contact us again.

Regards, Alejandro.