Problemas de visualizacion de wazuh-monitoring-* y wazuh-statistics-*

[Sebastian Cuadro]

Gente de la comunidad, como estan?

Tengo el siguiente tema, no puedo visualizar los eventos de wazuh-monitoring-* y wazuh-statistics-*
Las las alertas funcionan perfectamente al igual que los archivos.

Cual podrías ser el inconveniente?
Dejo print de pantalla.

[German DiCasas]

Hola Sebastian,

Como estas? tengo el mismo problema. ya no me estan saliendo los indices que indicas.. lo pudo solucionar?

Saludos

German

[Dario Menten]

Hola Sebastian,

Te comento que para que todos nos podamos entender sin problemas, es un requerimiento postear las consultas en Ingles.
Por este motivo, te voy a responder en Ingles y si necesitas responderme, porfavor continuemos en Ingles.

First, you need to know if the indices are being created:

curl -k -u user:pass -XGET "https://localhost:9200/_cat/indices/wazuh-monitoring*" curl -k -u user:pass -XGET "https://localhost:9200/_cat/indices/wazuh-statistics*"

The next step is checking the Wazuh Dashboard and Wazuh App logs, looking for any error:

journalctl -u wazuh-dashboard --no-pager | grep -iE "error|warn|fail" grep -iE "error|warn|fail" /usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log

With this troubleshooting, you will have more visibility of what is happening.
You can post the results in a response if you need help with your findings.

​

[Sebastian Cuadro]

Dear Darío,
I apologize for not replying in your preferred language. I am sending you the requested information:

wazuh-indexer:

root@wazuh-indexer-1:/home/ubuntu# curl -k -u admin:XXXXXXXXXXX -XGET "https://localhost:9200/_cat/indices/wazuh-statistics*"
green open wazuh-statistics-2024.16w Cx-uThXtTcqNEgNeE634Yw 1 0 306 0 461kb 461kb
root@wazuh-indexer-1:/home/ubuntu# curl -k -u admin:XXXXXXXXXXX -XGET "https://localhost:9200/_cat/indices/wazuh-monitoring*"
green open wazuh-monitoring-2024.16w NXKLDPHjTISiZNR_71L82g 1 0 1 0 18.7kb 18.7kb

wazuh-dashboard:

grep -iE "error|warn|fail" /usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log

{"date":"2024-04-17T16:14:50.635Z","level":"error","location":"APIUserAllowRunAs:check","message":"Request failed with status code 401"}
{"date":"2024-04-17T16:14:51.451Z","level":"error","location":"monitoring:getApiInfo","message":"Request failed with status code 401"}
{"date":"2024-04-17T16:15:02.933Z","level":"error","location":"APIUserAllowRunAs:check","message":"Request failed with status code 401"}
{"date":"2024-04-17T16:15:04.108Z","level":"error","location":"APIUserAllowRunAs:check","message":"Request failed with status code 401"}
{"date":"2024-04-17T16:15:04.110Z","level":"error","location":"APIUserAllowRunAs:check","message":"Request failed with status code 401"}
{"date":"2024-04-17T16:15:04.111Z","level":"error","location":"APIUserAllowRunAs:check","message":"Request failed with status code 401"}
{"date":"2024-04-17T16:15:04.533Z","level":"error","location":"APIUserAllowRunAs:check","message":"Request failed with status code 401"}
{"date":"2024-04-17T16:15:05.401Z","level":"error","location":"monitoring:getApiInfo","message":"Request failed with status code 401"}
{"date":"2024-04-17T16:20:02.663Z","level":"error","location":"APIUserAllowRunAs:check","message":"Request failed with status code 401"}
{"date":"2024-04-17T16:20:03.058Z","level":"error","location":"APIUserAllowRunAs:check","message":"Request failed with status code 401"}
{"date":"2024-04-17T16:20:03.851Z","level":"error","location":"APIUserAllowRunAs:check","message":"Request failed with status code 401"}
{"date":"2024-04-17T16:20:03.853Z","level":"error","location":"APIUserAllowRunAs:check","message":"Request failed with status code 401"}
{"date":"2024-04-17T16:25:03.277Z","level":"error","location":"APIUserAllowRunAs:check","message":"Request failed with status code 401"}
{"date":"2024-04-17T16:25:03.279Z","level":"error","location":"APIUserAllowRunAs:check","message":"Request failed with status code 401"}
{"date":"2024-04-17T16:25:04.057Z","level":"error","location":"APIUserAllowRunAs:check","message":"Request failed with status code 401"}
{"date":"2024-04-17T16:25:04.059Z","level":"error","location":"APIUserAllowRunAs:check","message":"Request failed with status code 401"}
root@wazuh-dashboard:/home/ubuntu# date
mié 17 abr 2024 20:56:57 UTC

journalctl -u wazuh-dashboard --no-pager | grep -iE "error|warn|fail"

abr 17 20:55:37 wazuh-dashboard opensearch-dashboards[9719]: {"type":"log","@timestamp":"2024-04-17T20:55:37Z","tags":["warning","config","deprecation"],"pid":9719,"message":"It is not recommended to disable xsrf protections for API endpoints via [server.xsrf.whitelist]. Instead, supply the \"osd-xsrf\" header."}
abr 17 20:55:37 wazuh-dashboard opensearch-dashboards[9719]: {"type":"log","@timestamp":"2024-04-17T20:55:37Z","tags":["warning","config","deprecation"],"pid":9719,"message":"\"opensearch.requestHeadersWhitelist\" is deprecated and has been replaced by \"opensearch.requestHeadersAllowlist\""}
root@wazuh-dashboard:/home/ubuntu# date
mié 17 abr 2024 20:59:20 UTC

[Dario Menten]

Hello Sebastian,
It seems the indices are being created, since they exist, but judging it by it size, I can say it is not being populated.
It could be related to a miscommunication between the Wazuh App and the Wazuh Manager’s API, or a product issue.
If you are able to, I would recommend upgrading your Wazuh to the latest version (v4.7.3).
If not, please check the api.log in the Wazuh Manager master node:

grep -iE "error|warn|fail" /var/ossec/logs/api.log

Also, you can check the ossec.conf file for any related issue:

grep -iE "error|warn|fail" /var/ossec/logs/ossec.log

Please let me know if you need any help with your findings.

​

[Sebastian Cuadro]

Hello Dario.

Since the production server is exhibiting the same behavior, we created a new one for testing in the lab.
I have attached the version we are currently implementing and the logs I requested.
It consists of 2 workers, 1 manager (cluster), 2 indexers (cluster), and a dashboard.
Additionally, I want to let you know that we are now on version 4.7.3.

Wazuh-Manager:

root@wazuh-manager:/home/ubuntu# grep -iE "error|warn|fail" /var/ossec/logs/ossec.log
root@wazuh-manager:/home/ubuntu#
root@wazuh-manager:/home/ubuntu#
root@wazuh-manager:/home/ubuntu#

root@wazuh-manager:/home/ubuntu# grep -iE "error|warn|fail" /var/ossec/logs/api.log
root@wazuh-manager:/home/ubuntu#

[Dario Menten]

Hello Sebastián,

If I do no understand wrong, by using 4.7.3 the problem is gone, sin in the last screenshot we can see the statistics are showing up.

In that case, please consider upgrading the conflictive environments in order to solve the issue.

Thank you.

--
You received this message because you are subscribed to the Google Groups "Wazuh | Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/f967e26e-346b-494c-8d04-f8e3058fc944n%40googlegroups.com.

[Sebastian Cuadro]

Hello, dear. 

The statistics still aren't working, what I attached are the versions I'm using. The statistics I did are a screenshot of the Wazuh cluster, those always worked.

[Natalia Castillo]

Hi Sebastian!

Considering the indices exist but aren't populating, it could indeed be related to how the index patterns are set up in the dashboard. You might need to create a custom index pattern to properly channel the events from the monitoring rules into the correct indices. This guide should help you set that up: Creating Custom Index Patterns.

Check out this GIF from the documentation for a quick visual on how it's done:
https://documentation.wazuh.com/current/_images/create-custom-alerts-index-pattern1.gif

Let's start with these steps and see if we can resolve the issue. Keep me posted on your progress!

[Sebastian Cuadro]

Hello,

But will creating custom indexes work for Wazuh's statistics and monitoring indexes?
Today, the alerts from connected devices are working fine.

[Sebastian Cuadro]

Additionally, I would like to mention that in Wazuh Alerts, I can see all the alerts from the agents, but I'm not able to see the ones that should be generated by the same manager nor the ones from the workers.

Connected agent:


Filtered by manager ID:

[Natalia Castillo]

Hi Sebastian,

I'm sorry for the delayed response. Are you still experimenting this issue?

For not being able to see the alerts that should be generated by the same manager nor the ones from the workers, you can try restarting the managers, both master and worker, that causes alerts to be generated from each of them. If you can see the alerts, it means that everything is OK on the deployment side.

[Sebastian Cuadro]

How are you? Indeed, I have completely restarted the server, restarted only the services, and the same thing happens. I see alerts for everything except for the workers and the manager.

[Natalia Castillo]

Hi!

Let's narrow down the problem, the main issue is that the alerts are not showing in the manager and workers, is that correct?

This is the alert generation flow:

• The agent captures an event
• The agent sends the event to the manager
• The manager generates an alert if the event matches a rule and stores it in alerts.json file
• The Filebeat component reads the file and sends the new alert to the Wazuh Indexer
• The Wazuh Dashboard is able to see the new alert reading the alerts' index

You are being able to see the agents so let's check the following file to make sure that alerts are being generated in the manager: /var/ossec/logs/alerts/alerts.json

If you see news alerts being generated on the alerts.json file, it may be a problem with Filebeat. Check the output of the command filebeat test output in the manager to check the state of Filebeat.

If everything is alright with the above,  let's check the indexer. To do that run the following query in your Dev Tools console to get more information

GET _cluster/health 

With the above steps we will be able to better identify where the problem lies. Let me see how it goes!

 

[Sebastian Cuadro]

Hello Natalia,

I'm sending you what you requested. Additionally, I want to clarify that this issue only occurs with the manager and the workers, not with the agents installed on clients. The strange thing is that I can see the alerts in alerts.json when connected via terminal.

filebeat test output

elasticsearch: https://172.19.167.11:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 172.19.167.11
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... OK
  version: 7.10.2
elasticsearch: https://172.19.167.12:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 172.19.167.12
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... OK
  version: 7.10.2
 
GET _cluster/health:

  "cluster_name": "indexer-soc-wazuh",
  "status": "green",
  "timed_out": false,
  "number_of_nodes": 2,
  "number_of_data_nodes": 2,
  "discovered_master": true,
  "discovered_cluster_manager": true,
  "active_primary_shards": 277,
  "active_shards": 337,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 0,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 100
}

/var/ossec/logs/alerts/alerts.json

"agent":{"id":"000","name":"wazuh-manager"},"manager":{"name":"wazuh-manager"},"id":"1715567364.1414","cluster":{"name":"soc-wazuh","node":"wazuh-manager.test-aws.interno"},"full_log":"May 13 02:29:23 wazuh-manager sudo: pam_unix(sudo:session): session opened for user root(uid=0) by ubuntu(uid=1000)","predecoder":{"program_name":"sudo","timestamp":"May 13 02:29:23","hostname":"wazuh-manager"},"decoder":{"parent":"pam","name":"pam"},"data":{"srcuser":"ubuntu","dstuser":"root(uid=0)","uid":"1000"},"location":"/var/log/auth.log"}

[Natalia Castillo]

Hi Sebastian,

All seems correct,  let's check if the filebeat is discarding some logs, or having some issues sending the logs to the Wazuh Indexer. Please share the output of the following commands:

• journalctl -u filebeat --no-pager | grep -iE "error|warn|fail"
• cat /var/log/filebeat/filebeat* | grep -iE "error|warn|fail"

Finally we can see the Wazuh Indexer logs to check if there is something wrong there:
grep -iE "error|warn|fail|caused" /var/log/wazuh-indexer/<cluster_name>.log

Let me know how it goes!

[Sebastian Cuadro]

Hello Noelia. I am sending you what you requested. The only errors I see, and I'm not sure if they are related to the main issue, you will tell me. Could it be a permissions issue? Because it's strange that their own alerts are generated in each worker and manager but do not go to the indexer. On the other hand, the alerts from connected workstations are seen in the indexers.

• journalctl -u filebeat --no-pager | grep -iE "error|warn|fail"
• cat /var/log/filebeat/filebeat* | grep -iE "error|warn|fail"

Manager:

may 12 00:00:24 wazuh-manager filebeat[84492]: 2024-05-12T00:00:24.272Z        ERROR        [publisher_pipeline_output]        pipeline/output.go:180        failed to publish events: temporary bulk send failure
may 13 02:29:01 wazuh-manager filebeat[84492]: 2024-05-13T02:29:01.624Z        ERROR        [publisher_pipeline_output]        pipeline/output.go:180        failed to publish events: temporary bulk send failure
may 17 01:38:53 wazuh-manager filebeat[84492]: 2024-05-17T01:38:53.466Z        ERROR        [publisher_pipeline_output]        pipeline/output.go:180        failed to publish events: temporary bulk send failure
may 23 00:00:37 wazuh-manager filebeat[84492]: 2024-05-23T00:00:37.374Z        ERROR        [publisher_pipeline_output]        pipeline/output.go:180        failed to publish events: temporary bulk send failure
may 26 00:00:26 wazuh-manager filebeat[84492]: 2024-05-26T00:00:26.718Z        ERROR        [publisher_pipeline_output]        pipeline/output.go:180        failed to publish events: temporary bulk send failure

root@wazuh-manager:/home/ubuntu# cat /var/log/filebeat/filebeat* | grep -iE "error|warn|fail"
root@wazuh-manager:/home/ubuntu#

Indexers:

grep -iE "error|warn|fail|caused" /var/log/wazuh-indexer/<cluster_name>.log

root@wazuh-indexer-1:/home/ubuntu# grep -iE "error|warn|fail|caused" /var/log/wazuh-indexer/indexer-soc-wazuh.log
root@wazuh-indexer-1:/home/ubuntu#

[Sebastian Cuadro]

Hello. Is there any news on the case? Or has anyone experienced something similar?