No Events at all in Kibana/Wazuh App

[Syed]

Hello Wazuh Team,

First of all hats off to your team on such great product and continuous improvements.

I need some direction for troubleshooting this one. I don't see any events in Kibana as no logs are coming in. I have ELKServer, CentOS, Apache and CloudTrail sending logs. Nothing shows up in Kibana.

Environment:

- In AWS

- ELK Server

- Wazuh Server

- Wazuh 3.2.2

- Nginx for Kibana authentication

No issues logging in. No error messages anywhere on the GUI. Wazuh or Discover shows no events at all.

Troubleshooting:

1. TCPDUMP shows Agents traffic to Manager on port 1514

2. On Manager: in /var/ossec/logs/alerts, alerts.json and alerts.log both files have log data.

3. /var/ossec/logs/ossec.log shows no errors. I can see following AWS activity as well.

             2018/05/20 22:39:04 wazuh-modulesd:aws-cloudtrail: INFO: Fetching logs started

             2018/05/20 22:39:25 wazuh-modulesd:aws-cloudtrail: INFO: Fetching logs finished.

4. API connection test on wazuh gui shows: "Settings. Connection success" and on ELK tcpdump I can see port 55000 connection going to wazuh. curl from ELK also works.

curl -u user:password -k https://WazuhIP:55000

{"error":0,"data":{"msg":"Welcome to Wazuh HIDS API","api_version":"v3.2.2","hostname":"ip-wazuhip.ec2.internal","timestamp":"Sun May 20 2018 23:25:45 GMT+0000 (UTC)"}}

5. When I click on Wazuh on kibana, I don't see any error messages and but I see tons of following errors in nginx error log.

2018/05/20 04:20:15 [error] 21159#21159: *648 connect() failed (111: Connection refused) while connecting to upstream, client: MyDesktopIP, server: , request: "GET /bundles/commons.style.css?v=16627 HTTP/1.1", upstream: "http://[::1]:5601/bundles/commons.style.css?v=16627", host: "ELKIP", referrer: "https://ELKIP/app/kibana"

2018/05/20 04:20:26 [error] 21159#21159: *653 connect() failed (111: Connection refused) while connecting to upstream, client: MyDesktopIP, server: , request: "GET /plugins/kibana/assets/dashboard.svg HTTP/1.1", upstream: "http://[::1]:5601/plugins/kibana/assets/dashboard.svg", host: "ELKIP", referrer: "https://ELKIP/app/wazuh"

2018/05/20 21:22:09 [error] 21159#21159: *716 connect() failed (111: Connection refused) while connecting to upstream, client: MyDesktopIP, server: , request: "GET /bundles/vendors.style.css?v=16627 HTTP/1.1", upstream: "http://[::1]:5601/bundles/vendors.style.css?v=16627", host: "ELKIP", referrer: "https://ELKIP/app/kibana"

2018/05/20 21:22:29 [error] 21159#21159: *729 connect() failed (111: Connection refused) while connecting to upstream, client: MyDesktopIP, server: , request: "GET /bundles/vendors.style.css?v=16627 HTTP/1.1", upstream: "http://[::1]:5601/bundles/vendors.style.css?v=16627", host: "ELKIP", referrer: "https://ELKIP/app/kibana"

2018/05/20 21:23:21 [error] 21159#21159: *789 connect() failed (111: Connection refused) while connecting to upstream, client: MyDesktopIP, server: , request: "GET /ui/favicons/favicon-32x32.png HTTP/1.1", upstream: "http://[::1]:5601/ui/favicons/favicon-32x32.png", host: "ELKIP"

2018/05/20 21:23:32 [error] 21159#21159: *789 connect() failed (111: Connection refused) while connecting to upstream, client: MyDesktopIP, server: , request: "GET /get-list HTTP/1.1", upstream: "http://[::1]:5601/get-list", host: "ELKIP", referrer: "https://ELKIP/app/wazuh"

What else can I check here, Is there something missing in ossec.conf file ?

Syed 

[jua...@wazuh.com]

Hello Syed,

Let me help you with this.

Whenever we face the situation where Kibana is not showing alerts at all, we have to check the data flow to see if the alerts.json content is being sent to Elasticsearch.

We're going to execute some commands to list your current templates and indices on your Elasticsearch server. Make sure you execute these commands as superuser:

curl -XGET localhost:9200/_cat/indices?v
curl -XGET localhost:9200/_cat/templates?v

Please, paste here the output. On the templates command, we should see the wazuh and wazuh-agent templates.

Now the following command must be executed if you have a single-host architecture, that means, you have the Elastic Stack installed on the same machine as the Wazuh server. Therefore, Logstash must be capable of reading the alerts.json file.

lsof /var/ossec/logs/alerts/alerts.json

This output should have an entry with the name java. If not, then you have to execute this command to authorize Logstash to read that file:

usermod -a -G ossec logstash

After this, you have to restart the Logstash service:

systemctl restart logstash

Let me know if these steps help you to solve your problem.

Best regards,

Juanjo

[S.Hasan Rizvi]

Thanks Juan.

In my setup, I have Wazuh on different box. Where do I run the lsof command?

--
You received this message because you are subscribed to a topic in the Google Groups "Wazuh mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/cTJRFOZX6AQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/8067d6ca-b4eb-42b5-8315-d759df805561%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Juanjo Jiménez]

Hello Syed,

If you have Wazuh on a different machine, then you have to run the lsof command on that machine, but that means you need to have Filebeat installed and configured on the same machine as the Wazuh manager, and Filebeat must be the program that can read the alerts.json file.

lsof /var/ossec/logs/alerts/alerts.json

Let me know about the results.

Regards,

Juanjo

To unsubscribe from this group and all its topics, send an email to wazuh+unsubscribe@googlegroups.com.

[Syed]

Thanks Juanjo,

Here is the output you requested. Looks OK to me.

On ELK Server:

On ELK Server:

[cen...@ip-x.x.x.x~]$ curl -XGET localhost:9200/_cat/indices?v

health status index                           uuid                   pri rep docs.count docs.deleted store.size pri.store.size

yellow open   wazuh-monitoring-3.x-2018.05.22 VTj2q6DzQGe1YbPJ35WFIw   5   1        432            0    426.1kb        426.1kb

yellow open   wazuh-monitoring-3.x-2018.05.25 ypPsGisSR_-erjU8om66Ug   5   1        432            0    505.2kb        505.2kb

yellow open   wazuh-monitoring-3.x-2018.05.19 B2g1r5HOQwOWGjksxXX3xQ   5   1        286            0    448.3kb        448.3kb

yellow open   wazuh-monitoring-3.x-2018.05.26 K4k_lKotR_WYGzNfh11EEg   5   1        315            0    593.6kb        593.6kb

yellow open   .kibana                         SV2gBBz5SuyOMwb31ceAgA   5   1          3            0     23.4kb         23.4kb

yellow open   wazuh-monitoring-3.x-2018.05.24 h0u3lq4dRDerx-BGen9wTA   5   1        432            0      582kb          582kb

yellow open   .wazuh                          jb7wEbu9RpeBLMca1FTWyQ   1   1          1            0     10.2kb         10.2kb

green  open   wazuh-alerts-3.x-2018.05.17     eM0Oxb0HSa2ZByQ2V5h0Uw   5   0          1            0     26.7kb         26.7kb

yellow open   wazuh-monitoring-3.x-2018.05.23 P5js3p_tTTuSXWBkKSCnKw   5   1        432            0    522.6kb        522.6kb

yellow open   wazuh-monitoring-3.x-2018.05.21 VnFNA2WZRVWFOJbVwF93Ag   5   1        432            0    550.2kb        550.2kb

yellow open   .wazuh-version                  El8GTBCmQouIYBl6H5H_gA   1   1          1            0      5.1kb          5.1kb

yellow open   wazuh-monitoring-3.x-2018.05.18 62PTpCHmQwOVlTpIC9KVAg   5   1        281            0    366.9kb        366.9kb

yellow open   wazuh-monitoring-3.x-2018.05.20 LQjRoZjWT7ej96UuzYCB_g   5   1        299            0    507.5kb        507.5kb

[cen...@ip-x.x.x.x~]$ curl -XGET localhost:9200/_cat/templates?v

name                          index_patterns           order version

wazuh-kibana                  [.kibana*]               0

kibana_index_template:.kibana [.kibana]                0

logstash                      [logstash-*]             0     60001

wazuh                         [wazuh-alerts-3.x-*]     0

wazuh-agent                   [wazuh-monitoring-3.x-*] 0

On Wazuh Manager:

[ro...@ip-X.X.X.X ~]# lsof /var/ossec/logs/alerts/alerts.json

COMMAND     PID  USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME

filebeat  10983  root    6r   REG  202,1   129589 8447574 /var/ossec/logs/alerts/alerts.json

ossec-ana 18946 ossec   10w   REG  202,1   129589 8447574 /var/ossec/logs/alerts/alerts.json

[Syed]

Also, I find it interesting that I don't see any port 9200 traffic on ELK.

ELK Server:

[ro...@ip-X.X.X.X ~]# tcpdump -nn dst port 9200

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes

[Juanjo Jiménez]

Hello again Syed, and sorry for the late response.

Could you please show me your Logstash and Filebeat configuration files? Here are the commands that you can use to get them and then, send me on a new post on this thread:

cat /etc/logstash/conf.d/01-wazuh.conf
cat /etc/filebeat/filebeat.yml

Don't forget to cover any sensitive information with asterisks (****) such as passwords, if there are any. Thanks for your patience.

Regards,

Juanjo

[Syed]

Here is the information you requested.

Logstash:

[centos@ip-XXXX ~]$ cat /etc/logstash/conf.d/01-wazuh.conf

# Wazuh - Logstash configuration file

## Remote Wazuh Manager - Filebeat input

input {

    beats {

        port => 5000

        codec => "json_lines"

#       ssl => true

#       ssl_certificate => "/etc/logstash/logstash.crt"

#       ssl_key => "/etc/logstash/logstash.key"

    }

}

filter {

    if [data][srcip] {

        mutate {

            add_field => [ "@src_ip", "%{[data][srcip]}" ]

        }

    }

    if [data][aws][sourceIPAddress] {

        mutate {

            add_field => [ "@src_ip", "%{[data][aws][sourceIPAddress]}" ]

        }

    }

}

filter {

    geoip {

        source => "@src_ip"

        target => "GeoLocation"

        fields => ["city_name", "country_name", "region_name", "location"]

    }

    date {

        match => ["timestamp", "ISO8601"]

        target => "@timestamp"

    }

    mutate {

        remove_field => [ "timestamp", "beat", "input_type", "tags", "count", "@version", "log", "offset", "type","@src_ip"]

    }

}

output {

    elasticsearch {

        hosts => ["localhost:9200"]

        index => "wazuh-alerts-3.x-%{+YYYY.MM.dd}"

        document_type => "wazuh"

    }

}

filebeat:

 prospectors:

  - input_type: log

    paths:

     - "/var/ossec/logs/alerts/alerts.json"

    document_type: json

    json.message_key: log

    json.keys_under_root: true

    json.overwrite_keys: true

output:

 logstash:

   # The Logstash hosts

   hosts: ["X.X.X.X:5000"]

   ssl:

      certificate_authorities: ["/etc/filebeat/logstash.crt"]

[Juanjo Jiménez]

Hello again Syed, and sorry for the late response.

Could you please send me your Filebeat and Logstash logs files?

cat /var/log/logstash/logstash-plain.log | grep -i -E "err|warn"
cat /var/log/filebeat/filebeat | grep -i -E "err|warn"

Everything on your configuration files seems correct, so now we're going to check if something is not properly working on Filebeat or Logstash. Thanks for your patience.

Regards,

Juanjo