Wazuh dashboard all logs disappeared

[Dmitry Mikheev]

Wazuh dashboard all logs disappeared from 3:00.

1. Agents are online
2. The logs themselves are received and processed:

/var/ossec/logs/alerts/
-rw-r----- 2 wazuh wazuh 549885841 huhti 7 08:17 alerts.json
-rw-r----- 2 wazuh wazuh 707061409 huhti 7 08:17 alerts.log

But nothing is displayed in the dashboard

How to fix and restore the display?

[Stuti Gupta]

Hi  Dmitry 

This happens when filebeat or the indexer is not working properly. Can you please share the following details so we can investigate the issue accordingly?

Please share the output of the following commands:
filebeat test output
curl -XGET -k -u user:pass "https://<WAZUH_INDEXER_IP>::9200/_cluster/health"

Please let us know your current Wazuh version and OS details 
Is this is All-in-one deployment or a distributed one?
Also, please share the number of agents and other devices.

Please share the Indexer and filebeat logs:
cat /var/log/filebeat/filbeat
cat /var/log/wazuh-indexer/wazuh-cluster.log

Looking forward to your response

[Dmitry Mikheev]

Hi, Stuti,

# curl -u admin:passwd -k "https://localhost:9200/_cluster/health"
{"cluster_name":"wazuh-cluster","status":"green","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"discovered_master":true,"discovered_cluster_manager":true,"active_primary_shards":1000,"active_shards":1000,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}

# cat /var/log/filebeat/filebeat
last
2025-04-07T10:59:55.697+0300    WARN    [elasticsearch] elasticsearch/client.go:408     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xc1f4ff5ea8f440f7, ext:5675061440133, loc:(*time.Location)(0x42417a0)}, Meta:{"pipeline":"filebeat-7.10.2-wazuh-alerts-pipeline"}, Fields:{"agent":{"ephemeral_id":"acc97b2f-594e-41e8-946d-93a0c7770cae","hostname":"Wazuh04","id":"beb6d55f-1f15-4b64-a663-32151c10facb","name":"Wazuh04","type":"filebeat","version":"7.10.2"},"ecs":{"version":"1.6.0"},"event":{"dataset":"wazuh.alerts","module":"wazuh"},"fields":{"index_prefix":"wazuh-alerts-4.x-"},"fileset":{"name":"alerts"},"host":{"name":"Wazuh04"},"input":{"type":"log"},"log":{"file":{"path":"/var/ossec/logs/alerts/alerts.json"},"offset":714556909},"message":"{\"timestamp\":\"2025-04-07T10:59:54.494+0300\",\"rule\":{\"level\":3,\"description\":\"FWA-SRX Juniper create know traffic.\",\"id\":\"160102\",\".....

#cat /var/log/wazuh-indexer/wazuh-cluster.log
last
[2025-04-07T11:01:52,702][WARN ][o.o.p.c.u.JsonConverter  ] [node-1] Json Mapping Error: Cannot invoke "java.lang.Long.longValue()" because "this.cacheMaxSize" is null (through reference chain: org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector$CacheMaxSizeStatus["Cache_MaxSize"])
[2025-04-07T11:01:57,703][WARN ][o.o.p.c.u.JsonConverter  ] [node-1] Json Mapping Error: Cannot invoke "java.lang.Long.longValue()" because "this.cacheMaxSize" is null (through reference chain: org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector$CacheMaxSizeStatus["Cache_MaxSize"])

# systemctl status filebeat
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
     Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; preset: enabled)
     Active: active (running) since Mon 2025-04-07 09:25:20 EEST; 1h 32min ago
       Docs: https://www.elastic.co/products/beats/filebeat

# systemctl status wazuh-indexer
● wazuh-indexer.service - wazuh-indexer
     Loaded: loaded (/usr/lib/systemd/system/wazuh-indexer.service; enabled; preset: enabled)
     Active: active (running) since Mon 2025-04-07 09:23:57 EEST; 1h 42min ago
       Docs: https://documentation.wazuh.com
   
# filebeat test output
elasticsearch: https://127.0.0.1:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 127.0.0.1
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.2
    dial up... OK
  talk to server... OK
  version: 7.10.2
 
# filebeat test config
Config OK

since it happened at 0:00:00 GMT I think maybe these are certificates? But I don't understand which ones and where...

[Dmitry Mikheev]

Hi, Stuti,

These are configuration files. They have not been changed.

#cat /etc/filebeat/filebeat.yml
# Wazuh - Filebeat configuration file
output.elasticsearch.hosts:
        - 127.0.0.1:9200

output.elasticsearch:
  protocol: https
  username: ${username}    <-- these are real lines. there is no name and password
  password: ${password}

  ssl.certificate_authorities:
    - /etc/filebeat/certs/root-ca.pem
  ssl.certificate: "/etc/filebeat/certs/wazuh-server.pem"
  ssl.key: "/etc/filebeat/certs/wazuh-server-key.pem"
setup.template.json.enabled: true
setup.template.json.path: '/etc/filebeat/wazuh-template.json'
setup.template.json.name: 'wazuh'
setup.ilm.overwrite: true
setup.ilm.enabled: false

filebeat.modules:
  - module: wazuh
    alerts:
      enabled: true
    archives:
      enabled: false

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644

logging.metrics.enabled: false

seccomp:
  default_action: allow
  syscalls:
  - action: allow
    names:
    - rseq


# cat /etc/wazuh-indexer/opensearch.yml
network.host: "127.0.0.1"
node.name: "node-1"
cluster.initial_master_nodes:
- "node-1"
cluster.name: "wazuh-cluster"

node.max_local_storage_nodes: "3"
path.data: /var/lib/wazuh-indexer
path.logs: /var/log/wazuh-indexer

plugins.security.ssl.http.pemcert_filepath: /etc/wazuh-indexer/certs/wazuh-indexer.pem
plugins.security.ssl.http.pemkey_filepath: /etc/wazuh-indexer/certs/wazuh-indexer-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: /etc/wazuh-indexer/certs/root-ca.pem
plugins.security.ssl.transport.pemcert_filepath: /etc/wazuh-indexer/certs/wazuh-indexer.pem
plugins.security.ssl.transport.pemkey_filepath: /etc/wazuh-indexer/certs/wazuh-indexer-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/wazuh-indexer/certs/root-ca.pem
plugins.security.ssl.http.enabled: true
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.transport.resolve_hostname: false
plugins.security.ssl.http.enabled_ciphers:
  - "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"
  - "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"
  - "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256"
  - "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"
plugins.security.ssl.http.enabled_protocols:
  - "TLSv1.2"
plugins.security.authcz.admin_dn:
- "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.nodes_dn:
- "CN=indexer,OU=Wazuh,O=Wazuh,L=California,C=US"
plugins.security.restapi.roles_enabled:
- "all_access"
- "security_rest_api_access"

plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]

### Option to allow Filebeat-oss 7.10.2 to work ###
compatibility.override_main_response_version: true

[Dmitry Mikheev]

I found that shards ran out and increased them by 2 times

PUT /_cluster/settings
{
"persistent": {
"cluster.max_shards_per_node": 2000
}
}

It is not clear how to download logs for the period of inactivity.

alerts.json contains them, but how to load them from it?

[Stuti Gupta]

Instead of increasing the number of shards, we recommend the following solution. Because increasing the shard can can lead to performance issues, resource exhaustion, and can slow your environment

Delete the indices manually
It is necessary to delete old indices, if they are of no use. It is necessary to check what the indices stored in the environment. The following API call can help:
GET _cat/indices
Then, it is necessary to delete indices that are not needed or older indices. Bear in mind that this cannot be retrieved unless there are backups of the data either using snapshots or Wazuh alert backups.
The API call to delete indices is:
DELETE <index_name>
Or CLI command
 # curl -k -u admin:admin -XDELETE https://<WAZUH_INDEXER_IP>:9200/wazuh-alerts-4.x-YYYY.MM.DD
You can use wildcards (*) to delete more indices in one query.

Index management policies:
Since storage space has a cost and a limit, you may have to delete old data to ensure you can maintain the retention period that you need.
Alerts generated by Wazuh are sent to an Elasticsearch daily index named wazuh-alerts-4.x-YYYY.MM.DD by using the default configuration. You can create policies that govern the lifecycle of the indices based on different phases. You can follow the steps mentioned in this document https://documentation.wazuh.com/current/user-manual/wazuh-indexer/wazuh-indexer-tuning.html. You can also take snapshots of the indices that automatically back up your Wazuh indices in local or Cloud-based storage and restore them at any given time. To do so please refer to https://wazuh.com/blog/index-backup-managementhttps://wazuh.com/blog/wazuh-index-management/

You can add a indexer node.
Thus maximizing the potential of Wazuh to effectively monitor and protect your growing IT infrastructure. Adding a new node to the Wazuh Indexer cluster can enhance the capacity and resilience of the security monitoring infrastructure. https://documentation.wazuh.com/current/user-manual/upscaling/adding-indexer-node.html


To recover, you can refer to https://documentation.wazuh.com/current/migration-guide/restoring/wazuh-central-components.html#restoring-old-logs

Let me know if you need any further assistance!