Agents alerts and AWS alerts are not showing on the dashboard

[shyamkanth ganesan]

Dear Team,

Dear Team I have Integrated the wazuh with 2 server, 1 Firewall, and AWS nginx and postgres

Recently I have added a cronjob to take the backup of the alert file in /var/ossec/logs/alerts/2023/Apr if the root storage is full , But even though the logs was received

Recently for 10 days the alert logs are not Showing in the dashboard but the agent logs are properly coming to /var/ossec/logs/alerts/2023/Apr and AWS logs are receiving in the path where we mentioned on the ossec.conf

Below error was showing in the dashboard while select the filebeat index :

null_pointer_exception
Cannot invoke "org.opensearch.search.aggregations.InternalAggregations.getSerializedSize()" because "reducePhase.aggregations" is null
Error: Internal Server Error
    at Fetch._callee3$ (https://localhost/44101/bundles/core/core.entry.js:15:584549)
    at tryCatch (https://localhost/44101/bundles/plugin/customImportMapDashboards/customImportMapDashboards.plugin.js:13:760622)
    at Generator.invoke [as _invoke] (https://localhost/44101/bundles/plugin/customImportMapDashboards/customImportMapDashboards.plugin.js:13:764638)
    at Generator.next (https://localhost/44101/bundles/plugin/customImportMapDashboards/customImportMapDashboards.plugin.js:13:761817)
    at fetch_asyncGeneratorStep (https://localhost/44101/bundles/core/core.entry.js:15:577641)
    at _next (https://localhost/44101/bundles/core/core.entry.js:15:577957)

I have replaced my ip with localhost in the above error note

Additional information : I have enough disk space in root dirc 34/50G 

[Cedrick Foko]

Hello, 

Thank you for using Wazuh.

Based on the information you provided,it appears that the alerts are received and indexed but not shown on the dashboard.

Run the following command to check that filebeat is communicating with opensearch: filebeat test output

Also, check the filebeat configuration ensuring that reducePhase.aggregations parameter is properly set.

If the issue persists, kindly share with me the output of the following commands for further investigations: 

• cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -E "err|warn"

• cat /var/log/filebeat/filebeat | grep -i -E "err|warn"

Looking forward to your feedback.

Regards,

[shyamkanth ganesan]

Thanks for your immediate response,

Please find the below screenshot of the command which you have provided

"Also, check the filebeat configuration ensuring that reducePhase.aggregations parameter is properly set." - Where to check this?
Please provide the path or conf filename please

Thanks and Regards,

Shyam kanth Ganesan

Associate Security Engineer L2
Auriseg Consulting PVT LTD

Chennai

--
You received this message because you are subscribed to a topic in the Google Groups "Wazuh mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/KtNQJoP9Of0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/322462ae-b140-486d-8b1d-9de356c28eb8n%40googlegroups.com.

[Cedrick Foko]

Hello,

The filebeat configuration file is located at /etc/filebeat/filebeat.yml 

Filebeat logs don't seem to provide information regarding the issue. Please share with me the output of the commands: 

• filebeat test output

• journalctl -u filebeat | grep -iE "error|warn|crit"
• journalctl -u wazuh-dashboard | grep -iE "error|warn|crit"
  

On the other hand, I can see you have this warning in wazuh-cluster logs: high disk watermark [90%] stating you have only 4.6GB free on your system. The more recommended option is to delete old indices that are no longer used so you can free some space.

You can use the Wazuh-indexer API to delete old indices, you can use the Dev Tool option that can be reached by clicking on the "Hamburguer menu" Icon on the top left corner of the UI and then scrolling down to said option:
DELETE <index_name>
You can use wildcards on the index name but be careful not to delete unwanted indices. For example if you no longer need the alerts information from 2022, you can execute something like this:
DELETE wazuh-alerts-4.x-2022.*
To list the indices before deciding what to delete you can use:
GET /_cat/indices

To automate this, a retention policy can be implemented, this can help you delete old indices that are no longer needed, thus freeing up space and making the shards used by them available for new information. You can find more information about this in the following documentation:  https://wazuh.com/blog/wazuh-index-management/

I hope you find this helpful.

[shyamkanth ganesan]

Hi Cedrick,

I have deleted the old indices. Thanks for the useful information

Please find the screenshot attached for journalctl -u wazuh-dashboard | grep -iE "error|warn|crit"

while running journalctl -u filebeat | grep -iE "error|warn|crit" this command no output came.

I am still unable to find the reducePhase.aggregations in the filebeat.yml

Still the same error comes like a pop up 'Internal server error' while selecting the alert or filebeat index. When open the error showing the same message which was pasted in the 1st conversation

Thanks and Regards,

Shyam kanth Ganesan

Associate Security Engineer L2
Auriseg Consulting PVT LTD

Chennai

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/f1437e8e-45b5-414a-a856-7fa1b2b92590n%40googlegroups.com.

[shyamkanth ganesan]

Thanks and Regards,

Shyam kanth Ganesan

Associate Security Engineer L2
Auriseg Consulting PVT LTD

Chennai

[Cedrick Foko]

Hello Shyamkanth,

I can see you have an error in your wazuh-dashboard logs stating: ECONNREFUSED 10.1.1.13:9200 (The dashboard cannot connect with the indexer)

We also have the following error: ERR_SSL_TLSV1_ALERT_UNKNOWN_CA

From those two errors, we can derive that the dashboard cannot connect with indexer due to certificate error. You may need to deploy new certificates for your environment. You can do so following the guide in our documentation:  Certificates deployment - User manual · Wazuh documentation

Let me know if this helps.

Regards,

[shyamkanth ganesan]

Dear Cedrick,

I have updated the certificate but still the same error please find the attachment

null_pointer_exception Cannot invoke "org.opensearch.search.aggregations.InternalAggregations.getSerializedSize()" because "reducePhase.aggregations" is null Error: Internal Server Error at Fetch._callee3$ (https://10.1.1.13/44101/bundles/core/core.entry.js:15:584549) at tryCatch (https://10.1.1.13/44101/bundles/plugin/customImportMapDashboards/customImportMapDashboards.plugin.js:13:760622) at Generator.invoke [as _invoke] (https://10.1.1.13/44101/bundles/plugin/customImportMapDashboards/customImportMapDashboards.plugin.js:13:764638) at Generator.next (https://10.1.1.13/44101/bundles/plugin/customImportMapDashboards/customImportMapDashboards.plugin.js:13:761817) at fetch_asyncGeneratorStep (https://10.1.1.13/44101/bundles/core/core.entry.js:15:577641) at _next (https://10.1.1.13/44101/bundles/core/core.entry.js:15:577957)

Thanks and Regards,

Shyam kanth Ganesan

Associate Security Engineer L2
Auriseg Consulting PVT LTD

Chennai

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/8b061ad5-ea00-46b1-95d7-33541ae86b79n%40googlegroups.com.

[Cedrick Foko]

Hello  Shyamkanth,

Can you show me the full error by clicking on "See the full error" in Search error notification on the dashboard?
I'm suspecting not enough free disk space on elasticsearch data nodes. Let's check the free disk space: df -h

Looking forward to your feedback.

[shyamkanth ganesan]

Thanks for your response back Cedrick,

While check from ssh it showing free space 37/50G but while get into server in GUI there is a popup warning showing free up disk space

This is the full error

null_pointer_exception Cannot invoke "org.opensearch.search.aggregations.InternalAggregations.getSerializedSize()" because "reducePhase.aggregations" is null Error: Internal Server Error at Fetch._callee3$ (https://10.1.1.13/44101/bundles/core/core.entry.js:15:584549) at tryCatch (https://10.1.1.13/44101/bundles/plugin/customImportMapDashboards/customImportMapDashboards.plugin.js:13:760622) at Generator.invoke [as _invoke] (https://10.1.1.13/44101/bundles/plugin/customImportMapDashboards/customImportMapDashboards.plugin.js:13:764638) at Generator.next (https://10.1.1.13/44101/bundles/plugin/customImportMapDashboards/customImportMapDashboards.plugin.js:13:761817) at fetch_asyncGeneratorStep (https://10.1.1.13/44101/bundles/core/core.entry.js:15:577641) at _next (https://10.1.1.13/44101/bundles/core/core.entry.js:15:577957)

Thanks and Regards,

Shyam kanth Ganesan

Associate Security Engineer L2
Auriseg Consulting PVT LTD

Chennai

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/06747b09-3985-41f5-8082-9efb0d07a4adn%40googlegroups.com.

[Cedrick Foko]

Hello Shyamkanth, 

If you get a popup warning asking you to free disk space, it means the output provided by df -h command in not updated. The information is updated after a reboot.

Can you try to get disk space information in GUI?

Otherwise, you can reboot the server and check again.

Let me know the result.

Regards,

[shyamkanth ganesan]

Hi Cedrick,

 I have reboot it manually and checked df -h which i was attached in previous mail,

If the disk space is full usually the API will stop working it right but here like that no error

Did you see any findings on see full error in the internal server error pop up on the dashboard 

Thanks and Regards,

Shyam kanth Ganesan

Associate Security Engineer L2
Auriseg Consulting PVT LTD

Chennai

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/bf72db96-2da3-4929-8884-11a7722ff833n%40googlegroups.com.

[Cedrick Foko]

Hello, 

Please let me know which Wazuh version you are using and the number of agents you have.

Regards,

[shyamkanth ganesan]

Hi Cedrick,

We recently updated the wazuh components and Agent to 4.4V from the previous version, but not updated the opensearch or elasticsearch.

Totally we have 2 server - Agent

1 Firewall (Syslog)

AWS nginix,postgres logs will receive via path we defined in our SIEM server - Filebeat

Thanks and Regards,

Shyam kanth Ganesan

Associate Security Engineer L2
Auriseg Consulting PVT LTD

Chennai

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/2d13355f-cd5e-4ceb-9472-0297beb1db93n%40googlegroups.com.

[shyamkanth ganesan]

Hi Cedrick,

Will this log will help you to solve the issues

Thanks and Regards,

Shyam kanth Ganesan

Associate Security Engineer L2
Auriseg Consulting PVT LTD

Chennai

[Cedrick Foko]

Hello Shyamkanth,

Thank you for your participation. The logs regarding services world-inaccessible could not be the reason of the issue here. 

However, the one regarding opensearch-dashboards may be relevant. Kindly send me the full log as it is cut in the screenshot.

Regards,

[Cedrick Foko]

Hello Shyamkanth,

Since the connection is refused, perhaps your firewall is blocking connection attempts to the indexer. Please check this one and let me know.

Also, let's verify that the port 9200 on the indexer is effectively open and accepting connections. You can use the following command to check processes using that port: netstat -ntp | grep 9200

Looking forward to your feedback.

[shyamkanth ganesan]

Hi Cedrick,

I have uninstalled the wazuh and reinstalled it and now everything working fine. Thanks for your help it teach me to check the connectivity issue

Thanks and Regards,

Shyam kanth Ganesan

Associate Security Engineer L2
Auriseg Consulting PVT LTD

Chennai

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/2f4ff607-3bba-4577-8ef2-caa37a993706n%40googlegroups.com.

[Cedrick Foko]

Hello Shyam,

Thank you for your feedback.

Please don't hesitate to contact us should you need any further help.

Kind regards,