Security Events not showing up in Wazuh dashboard

[Bill Green]

Good morning,

Thank you for this group - it is a huge help.  I apologize if this topic has been addressed previously, sometimes my search capabilities are lacking.

I had to rebuild my Wazuh server recently. It is an "all-in-one" deployment as I'm monitoring only about 70 systems.  Once I completed the rebuild, all the agents began to show up in the portal.  However, although data is coming in based on the tail -f on the alerts.log in /var/ossec/logs/alerts, when I attempt to look at Security events in the dashboard it indicates "There are no results for the selected time range. Try another one." (see attached screengrab)

I didn't perform the setup on the initial server, so this may have been an issue originally but I'm uncertain how the engineer resolved it.

Below are the config files for Elasticsearch, Kibana, and Filebeat but I'm happy to provide additional configs or info if needed.

Elasticsearch.yml:

network.host: xxx.xxx.32.33

node.name: node-1

cluster.initial_master_nodes: node-1

opendistro_security.ssl.transport.pemcert_filepath: /etc/elasticsearch/certs/elasticsearch.pem

opendistro_security.ssl.transport.pemkey_filepath: /etc/elasticsearch/certs/elasticsearch.key

opendistro_security.ssl.transport.pemtrustedcas_filepath: /etc/elasticsearch/certs/root-ca.pem

opendistro_security.ssl.transport.enforce_hostname_verification: false

opendistro_security.ssl.transport.resolve_hostname: false

opendistro_security.ssl.http.enabled: true

opendistro_security.ssl.http.pemcert_filepath: /etc/elasticsearch/certs/elasticsearch_http.pem

opendistro_security.ssl.http.pemkey_filepath: /etc/elasticsearch/certs/elasticsearch_http.key

opendistro_security.ssl.http.pemtrustedcas_filepath: /etc/elasticsearch/certs/root-ca.pem

opendistro_security.nodes_dn:

- CN=node-1,OU=Docu,O=Wazuh,L=California,C=US

opendistro_security.authcz.admin_dn:

- CN=admin,OU=Docu,O=Wazuh,L=California,C=US

opendistro_security.audit.type: internal_elasticsearch

opendistro_security.enable_snapshot_restore_privilege: true

opendistro_security.check_snapshot_restore_write_privileges: true

opendistro_security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]

cluster.routing.allocation.disk.threshold_enabled: false

node.max_local_storage_nodes: 3

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

Filebeat.yml:

# Wazuh - Filebeat configuration file

output.elasticsearch:

  hosts: ["xxx.xxx.32.33:9200"]

  protocol: https

  username: "xxx"

  password: "xxx"

  ssl.certificate_authorities:

    - /etc/filebeat/certs/root-ca.pem

  ssl.certificate: "/etc/filebeat/certs/filebeat.pem"

  ssl.key: "/etc/filebeat/certs/filebeat.key"

setup.template.json.enabled: true

setup.template.json.path: '/etc/filebeat/wazuh-template.json'

setup.template.json.name: 'wazuh'

setup.ilm.overwrite: true

setup.ilm.enabled: false

filebeat.modules:

  - module: wazuh

    alerts:

      enabled: true

    archives:

      enabled: false

Kibana.yml:

server.host: xxx.xxx.32.33

server.port: 443

elasticsearch.hosts: https://xxx.xxx.32.33:9200

elasticsearch.ssl.verificationMode: certificate

elasticsearch.username: xxxxxxxxxx

elasticsearch.password: xxxxxxxxxx

elasticsearch.requestHeadersWhitelist: ["securitytenant","Authorization"]

opendistro_security.multitenancy.enabled: false

opendistro_security.readonly_mode.roles: ["kibana_read_only"]

server.ssl.enabled: true

server.ssl.key: "/etc/kibana/certs/kibana.key"

server.ssl.certificate: "/etc/kibana/certs/kibana.pem"

elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/certs/root-ca.pem"]

server.defaultRoute: /app/wazuh

Thank you in advance for your help - it's much appreciated.

Bill

[Juan Emiliano Fontana]

Hello Bill, hope you are doing well and thanks for using Wazuh!

 

If the agent status is Active in Kibana but you do not see any alerts generated for the agent it could mean that the data is not reaching the Elasticsearch to be indexed there. Can you please run the nexts command and share the results:

 

# systemctl status filebeat
 

# filebeat test output 

And also can you check your Index Management from the Management tab of Kibana to check if your index has actually a size different from 0? And please check the date on the Wazuh-Manager and see if it is approximately the same that you have on your system (sysmon).

You can check the date on the Wazuh-Manager with the next command:

 

# date

 

Thanks

[Bill Green]

Can't thank you enough for the assistance - I've not used Wazuh for very long but really like it (until I self-inflict issues LOL).

So, it looks like a certificate issue from the commands you provided but not certain how to resolve those off-hand.  Here is the info from the commands (and I've also attached a screenshot).

systemctl status filebeat

root@secon:/home/bgreen# systemctl status filebeat

● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.

     Loaded: loaded (/lib/systemd/system/filebeat.service; enabled; vendor preset: enabled)

     Active: active (running) since Tue 2021-04-06 14:06:10 CDT; 22h ago

       Docs: https://www.elastic.co/products/beats/filebeat

   Main PID: 1134 (filebeat)

      Tasks: 18 (limit: 43195)

     Memory: 92.6M

     CGroup: /system.slice/filebeat.service

             └─1134 /usr/share/filebeat/bin/filebeat --environment systemd -c /etc/filebeat/filebeat.yml --path.home /usr/share/filebeat --path.config /etc/filebeat --path.data /var/lib/filebeat --path.logs /var/log/filebeat

Apr 07 12:33:57 secon filebeat[1134]: 2021-04-07T12:33:57.787-0500        INFO        [publisher_pipeline_output]        pipeline/output.go:145        Attempting to reconnect to backoff(elasticsearch(https://xxx.xxx.32.33:9200)) with 17>

Apr 07 12:33:57 secon filebeat[1134]: 2021-04-07T12:33:57.787-0500        INFO        [publisher]        pipeline/retry.go:213        retryer: send wait signal to consumer

Apr 07 12:33:57 secon filebeat[1134]: 2021-04-07T12:33:57.787-0500        INFO        [publisher]        pipeline/retry.go:217          done

Apr 07 12:34:02 secon filebeat[1134]: 2021-04-07T12:34:02.093-0500        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"n>

Apr 07 12:34:32 secon filebeat[1134]: 2021-04-07T12:34:32.093-0500        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"n>

Apr 07 12:34:57 secon filebeat[1134]: 2021-04-07T12:34:57.095-0500        ERROR        [publisher_pipeline_output]        pipeline/output.go:154        Failed to connect to backoff(elasticsearch(https://xxx.xxx.32.33:9200)): Get "https:>

Apr 07 12:34:57 secon filebeat[1134]: 2021-04-07T12:34:57.095-0500        INFO        [publisher_pipeline_output]        pipeline/output.go:145        Attempting to reconnect to backoff(elasticsearch(https://xxx.xxx.32.33:9200)) with 17>

Apr 07 12:34:57 secon filebeat[1134]: 2021-04-07T12:34:57.095-0500        INFO        [publisher]        pipeline/retry.go:213        retryer: send wait signal to consumer

Apr 07 12:34:57 secon filebeat[1134]: 2021-04-07T12:34:57.095-0500        INFO        [publisher]        pipeline/retry.go:217          done

Apr 07 12:35:02 secon filebeat[1134]: 2021-04-07T12:35:02.093-0500        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"n>

lines 1-20/20 (END)

filebeat test output

root@secon:/home/bgreen# filebeat test output

elasticsearch: https://xxx.xxx.32.33:9200...

  parse url... OK

  connection...

    parse host... OK

    dns lookup... OK

    addresses: xxx.xxx.32.33

    dial up... OK

  TLS...

    security: server's certificate chain verification is enabled

    handshake... ERROR x509: certificate is valid for 127.0.0.1, not xxx.xxx.32.33

root@secon:/home/bgreen# date

Wed 07 Apr 2021 12:43:31 PM CDT

Thanks again for the help,

Bill

[Bill Green]

Sorry, I forgot to include the indices.

[Juan Emiliano Fontana]

Hi Bill, thanks for the information provided.

You have an all-in-one deployment so take in consideration that the components of Wazuh (Filebeat, Elasticsearh and Kibana) can communicate between themselves using the localhost IP, so the certificate was generated with that IP.

In order to solve this issue please follow the next steps:·      

• Modify the network.host setting  on /etc/elasticsearch/elasticsearch.yml file from xxx.xxx.32.33 to 127.0.0.1:"  <network.host: 127.0.0.1>
  
• Modify the output.elasticsearch.hosts setting on /etc/filebeat/filebeat.yml file  from "xxx.xxx.32.33:9200" to "127.0.0.1:9200":
  output.elasticsearch:
   hosts: ["127.0.0.1:9200"]
• Modify the elasticsearch.hosts setting on/etc/filebeat/kibana.yml file  from "xxx.xxx.32.33:9200" to "127.0.0.1:9200:
  elasticsearch.hosts: https://127.0.0.1:9200

Let me know if this resolves the issue

Regards!

[Bill Green]

Thanks again for the help - that has fixed it in the short term.  I will likely have to rebuild again and see if I can get the actual host IP as part of the cert chain as right now the dashboard is only available to those users who log into the server locally and bring up the browser pointed at the home address.  But, this is a huge help and I can see events now - I was a bit blind as to what was going on so this is a huge relief.

Thank you for the help!

Bill

[Juan Emiliano Fontana]

Glad I can help Bill!

If you need further assistance let us know.

Regards!