WAZUH not collecting logs

[Olajide Olatunji]

I have Wazuh working well in my environment for the past 5months plus but all of a sudden last week no logs were collected from any of the agents even though all the devices are connected and live

Below is the ossec logs I keep getting from all the connected agents

2021/12/07 00:00:10 ossec-agent: INFO: Starting new log after rotation.
2021/12/07 00:43:46 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2021/12/07 00:43:50 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2021/12/07 01:08:32 ossec-agent: ERROR: Could not get message for (Application)
2021/12/07 01:43:46 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2021/12/07 01:43:50 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2021/12/07 02:43:46 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2021/12/07 02:43:50 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2021/12/07 03:43:46 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2021/12/07 03:43:50 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2021/12/07 04:43:46 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2021/12/07 04:43:51 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2021/12/07 05:43:46 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2021/12/07 05:43:52 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2021/12/07 06:07:59 ossec-agent: ERROR: Could not get message for (Application)
2021/12/07 06:43:46 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2021/12/07 06:43:51 wazuh-modulesd:syscollector: INFO: Evaluation finished.

[mayte...@wazuh.com]

Hi,

It would be helpful if you could answer the following questions in order to find the cause of the problem 

• Which Wazuh version are you using? (agents and server)
• Did you perform any changes last week?
• Did the agents stop sending logs for some specific devices or do they not send any logs at all?

If the agents stop sending logs for some specific devices it would be useful to enable the debug mode in one of the agents to troubleshoot the issue. The settings should be changed in the internal_options.conf file (Do not forget to restart the agent afterwards to  apply the changes)

If no log is showing in the WUI, the issue may be in the manager or the communication with Elasticsearch may be the one that is not working properly. If so:

- Could you check if the /var/ossec/logs/alerts/alerts.json file is being populated? (You can use the command tail -f /var/ossec/logs/alerts/alerts.json or check the last alerts using tail -n 5 /var/ossec/logs/alerts/alerts.json)

- If the /var/ossec/logs/alerts/alerts.json file is being populated, check if Filebeat is sending the logs to Elasticsearch and if there is communication between Filebeat and Elasticsearch by running: filebeat test output. Also check the filebeat status by running the command service filebeat status

Also, it may be helpful to check the following in your Elasticsearch server:

- Elasticsearch indices: curl -k -u <user>:<pass> https://localhost:9200/_cat/indices?s=index

- Elasticsearch cluster health: curl -k -u <user>:<pass> https://localhost:9200/_cluster/health?pretty

 

Please keep us updated!

Best regards,

Mayte Ariza

[Olajide Olatunji]

Hi,

Thanks for your response, my response to your enquires below;

Wazuh version are you using? (agents and server)

Server :  App Version 4.1.5 

Agent: 4.1.5

Did you perform any changes last week?

No changes performed

 
Did the agents stop sending logs for some specific devices or do they not send any logs at all?

All agents stopped sending logs

- Elasticsearch indices: curl -k -u <user>:<pass> https://localhost:9200/_cat/indices?s=index

Elasticsearch cluster health: curl -k -u <user>:<pass> https://localhost:9200/_cluster/health?pretty

I ran this command "tail -n 5 /var/ossec/logs/alerts/alerts.json" and got this output which shows that logs are still being collected by the WAZUH Manager but the issue now is that its not displaying on the web url

--
You received this message because you are subscribed to a topic in the Google Groups "Wazuh mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/K0GAN_FZjyM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/72968eb5-a744-4bd2-b528-5fddb2e5fef8n%40googlegroups.com.

[mayte...@wazuh.com]

Hi,

Run the following command in your Elasticsearch server to check the wazuh-alerts indices: curl -k -u <user>:<pass> https://localhost:9200/_cat/indices/wazuh-alerts*?s=index
If your indices are created daily, it allows us to check if your alerts are still indexed in Elasticsearch or not.

If the /var/ossec/logs/alerts/alerts.json file is being populated but no indices are created in Elasticsearch, check if Filebeat is sending the logs to Elasticsearch and if there is communication between Filebeat and Elasticsearch by running the command: filebeat test output
Also, check if Filebeat is running: service filebeat status

Please keep us updated!

Best regards,
Mayte Ariza

[Olajide Olatunji]

Hi Mayte,

I dont have filebeat running alongside my wazuh setup in my environment, I only have wazuh manager, kibana and elasticsearch running for the past 6months and its been working fine till last week when i stop seeing logs on the WAZUH dashboard or kibana discovery.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/9caf1aae-85bd-43ec-aa72-71ac70770a4dn%40googlegroups.com.

[mayte...@wazuh.com]

Hi,

Filebeat is the tool on the Wazuh server that securely forwards alerts and archived events to Elasticsearch. You should have Filebeat installed on the same server your Wazuh manager is running.

Check if there is communication between Filebeat and Elasticsearch by running the command: filebeat test output

Also, check if Filebeat is running: service filebeat status

Please keep us updated!

Best regards,
Mayte Ariza

[Olajide Olatunji]

Hi Mayte,

Find the output below for Filebeat test output

Service filebeat status

Looking forward to your update

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/222a9835-1c9e-4eb0-833e-11aa349c2624n%40googlegroups.com.

[Olajide Olatunji]

Is anyone with a solution on how to resolve this issue?

[Julio José Reyes Hurtado]

Hi,

In the screenshot that you shared about the filebeat status, the beginning of a WARNING log may be causing that you are losing the events for some reason. By default, filebeat sends the logs to syslog output, if you are using ubuntu or ubuntu based distro the path would be /var/ossec/log/syslog or if you are in a centos/RHEL os you can check at /var/log/messages you can use cat command combined with grep -i filebeat | grep -iv info.

Alternatively, you can stop the service and use the debug modes in the filebeat documentation it runs the service in foreground and you can see the logs in the standard output.

Looking forward your update

Regards, Julio Reyes

[Black Fish]

check the default limit for the number of open index it would be like 1000, if the number of open index is  equal to limit , no more index will be created resulting in no indexing of alerts.

Virus-free. www.avast.com

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/548df503-1f14-4d8c-8aef-bdb4dbaa2c18n%40googlegroups.com.

Virus-free. www.avast.com

[mayte...@wazuh.com]

Hi,

 
Sorry, I have been OOO for a while. Are you still facing the same issue?

Filebeat test output and status seem correct. As Julio José said it might be interesting to check the Filebeat logs in case events are being discarded for some reason. You could use the following command cat /var/log/filebeat/filebeat | grep -v "INFO" to check warning and error logs.

Run the following command in your Elasticsearch server to check the wazuh-alerts indices: curl -k -u <user>:<pass> https://localhost:9200/_cat/indices/wazuh-alerts*?s=index

If your indices are created daily, it allows us to check if your alerts are still indexed in Elasticsearch or not. Please share the output with us.

Also, take a look at the Elasticsearch logs. They should be located in /var/log/elasticsearch/<your_cluster_name>.log file

Regarding what was said about the cluster.max_shards_per_node setting, by default the shard limit per node is 1000, since you had 864 active shards that should not be the problem, unless the value of that setting has been modified (cluster-shard-limit).

Please keep us updated!

Best regards,
Mayte Ariza

[Le Sok]

Does anyone solve this problem yet ? now I meet the same problem 
please help how to solve this problem