Filebeat Logs - No Results Match Your Search

704 views
Skip to first unread message

Suat Toksöz

unread,
Oct 24, 2019, 8:30:11 AM10/24/19
to Wazuh mailing list
hi,

filebeat service is working properly, but we can not see any logs from filebeat on kibana.




# Wazuh - Filebeat configuration file
filebeat.modules:
  - module: wazuh
    alerts:
      enabled: true
    archives:
      enabled: false

setup.template.json.enabled: true
setup.template.json.path: '/etc/filebeat/wazuh-template.json'
setup.template.json.name: 'wazuh'
setup.template.overwrite: true
setup.ilm.enabled: true

#output.elasticsearch.hosts: ['http://localhost:9200']
output.elasticsearch.hosts: ['88.88.88.88:9200']
output.elasticsearch.protocol: https
output.elasticsearch.ssl.certificate: "/etc/filebeat/certs/wazuh-manager.crt"
output.elasticsearch.ssl.key: "/etc/filebeat/certs/wazuh-manager.key"
output.elasticsearch.ssl.certificate_authorities: ["/etc/filebeat/certs/ca/ca.crt"]
output.elasticsearch.username: "888888"
output.elasticsearch.password: "888888"

# Filebeat modules configuration
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml





image.png


root@5689659856:~# systemctl status filebeat.service
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
   Loaded: loaded (/lib/systemd/system/filebeat.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2019-10-24 14:24:22 +03; 4min 41s ago
     Docs: https://www.elastic.co/products/beats/filebeat
 Main PID: 3502 (filebeat)
    Tasks: 10 (limit: 4915)
   CGroup: /system.slice/filebeat.service
           └─3502 /usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log

Oct 24 14:24:24 xxxxtest.xxxsuat.xxxxlocal filebeat[3502]: 2019-10-24T14:24:24.051+0300        INFO        pipeline/output.go:105        Connection to backoff(elasticsearch(https://xxxxxx.x.
Oct 24 14:24:52 xxxxtest.xxxsuat.xxxxlocal filebeat[3502]: 2019-10-24T14:24:52.929+0300        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s      
Oct 24 14:25:22 xxxxtest.xxxsuat.xxxxlocal filebeat[3502]: 2019-10-24T14:25:22.928+0300        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s      
Oct 24 14:25:52 xxxxtest.xxxsuat.xxxxlocal filebeat[3502]: 2019-10-24T14:25:52.928+0300        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s      


Best regards,

Suat Toksöz

Wazuh 3.10.2
Elastic Stack 7.3.2

Daniel Moreno

unread,
Oct 24, 2019, 9:23:33 AM10/24/19
to Wazuh mailing list

Hi Suat,


The events shown in Kibana are the alerts triggered by the Wazuh manager, Filebeat sends these alerts from the manager to ElasticSearch. The issue could come from different sources so I’ll try to give to you hints to troubleshoot it.


First of all, it would be great to ensure that Filebeat is able to connect to ElasticSearch successfully by using  filebeat test output


Since you are using SSL and X-Pack features It should return something similar to:


[root@centOS7-2 ~]# filebeat test output
elasticsearch: https://11.0.0.2:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 11.0.0.2
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.2
    dial up... OK
  talk to server... OK
  version: 7.3.2


In case it returns no error the problem may be related to a installation problem. Please ensure the wazuh module and template are correctly installed.


You may check it by looking within /etc/filebeat/ and looking for the file wazuh-template.json, the Wazuh module is usually stored under /usr/share/filebeat/module.

Our documentation shows the steps to download the Wazuh module and template for Filebeat in the Manager installation section.


In case it is not able to connect to ElasticSearch the problem could be related to the configuration. Check out the username and password provided to Filebeat and the path to the certificates.


It is important to ensure Filebeat is reading the alerts file, you may check it by using:


lsof /var/ossec/logs/alerts/alerts.json


The output should be similar to:


[root@centOS7-2 ~]# lsof /var/ossec/logs/alerts/alerts.json
COMMAND    PID USER FD   TYPE DEVICE SIZE/OFF     NODE NAME
filebeat  1228 root    3r REG 253,0  1476488 35520050 /var/ossec/logs/alerts/alerts.json
ossec-ana 6059 ossec   10w REG 253,0 1476488 35520050 /var/ossec/logs/alerts/alerts.json



I hope it helps you.

Regards.

Suat Toksöz

unread,
Oct 24, 2019, 10:01:23 AM10/24/19
to Daniel Moreno, Wazuh mailing list
Hi Daniel,

Re-get the json from the orijinal site:



~# filebeat test output
elasticsearch: https://xx.x.x.x..x.x:9200...

  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: xx.xxx.xx.xxx

    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.2
    dial up... OK
  talk to server... OK
  version: 7.3.2



:~# lsof /var/ossec/logs/alerts/alerts.json

COMMAND    PID  USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
filebeat  3502  root    5r   REG  254,0   115349 396045 /var/ossec/logs/alerts/alerts.json
ossec-ana 4120 ossec   11w   REG  254,0   115349 396045 /var/ossec/logs/alerts/alerts.json


still not showing any logs on the kibana for filebeat*

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/5b220642-4a26-4d88-8b26-cc6a988076f6%40googlegroups.com.


--

Daniel Moreno

unread,
Oct 25, 2019, 4:28:44 AM10/25/19
to Wazuh mailing list

Hello,


I’m sorry those steps didn’t solve your problem.


Let’s check out the manager is generating alerts. You may do it by executing the command: 


tailf /var/ossec/logs/alerts/alerts.json


After executing the command you may force the manager to generate some alerts by performing a sudo su in the same machine for example.

In case you can't see new alerts being generated please check the manager it up and running. You may find errors related to the manager by looking at the /var/ossec/logs/ossec.log file.

In case you can see new alerts its is possible to debug Filebeat by executing:


filebeat -e -d "processors"


Now each time an alert is generated you should be able to see a new event similar to:


2019-10-25T08:13:30.072Z DEBUG [processors] processing/processors.go:183 Publish event: {
 
"@timestamp": "2019-10-25T08:13:30.072Z",
 
"@metadata": {
   
"beat": "filebeat",
   
"type": "_doc",
   
"version": "7.3.2",
   
"pipeline": "filebeat-7.3.2-wazuh-alerts-pipeline"
 
},
 
"fileset": {
   
"name": "alerts"
 
},
 

In case everything worked as expected is time to move to ElasticSearch and look for error there. Let’s look for errors in the log file, please execute:


cat /var/log/elasticsearch/elasticsearch.log | tail -n 100


You will see a bunch of different messages, look for WARN or ERROR messages. In case you find errors here the problem would probably be related to a misconfiguration. Check out the /etc/elasticsearch/elasticsearch.yml file to find any typo.

You may get further information on how to configure Elasticsearch in our documentation.


Don't hesitate to share with us any error you find there and we will try to help you to get it solved.


Regards.

Suat Toksöz

unread,
Oct 25, 2019, 7:17:24 AM10/25/19
to Daniel Moreno, Wazuh mailing list
Hi Daniel,

Thank you for your help. Here is the log outputs for the filebeat:


tailf /var/ossec/logs/alerts/alerts.json
{"timestamp":"2019-10-25T12:07:59.220+0300","rule":{"level":5,"description":"syslog: User authentication failure.","id":"2501","firedtimes":1,"mail":false,"groups":["syslog","access_control","authentication_failed"],"pci_dss":["10.2.4","10.2.5"],"gpg13":["7.8"],"gdpr":["IV_35.7.d","IV_32.2"],"hipaa":["164.312.b"],"nist_800_53":["AU.14","AC.7"]},"agent":{"id":"001","name":"XXXXXXXXPC","ip":"10.211.1.40"},"manager":{"name":"test.xxxxxz.test"},"id":"1571994479.19783","full_log":"Oct 25 12:06:38 XXXXXXXXPC xscreensaver[2244]: pam_sss(xscreensaver:auth): authentication failure; logname= uid=1965401317 euid=1965401317 tty=:0.0 ruser= rhost= user=st000024","predecoder":{"program_name":"xscreensaver","timestamp":"Oct 25 12:06:38","hostname":"XXXXXXXXPC"},"decoder":{},"location":"/var/log/auth.log"}



# filebeat -e -d "processors"
2019-10-25T14:14:55.396+0300 DEBUG [processors] processing/processors.go:183 Publish event: {
  "@timestamp": "2019-10-25T11:14:55.396Z",

  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.3.2",
    "pipeline": "filebeat-7.3.2-wazuh-alerts-pipeline"
  },
  "log": {
    "offset": 57176,
    "file": {
      "path": "/var/ossec/logs/alerts/alerts.json"

    }
  },
  "fileset": {
    "name": "alerts"
  },
  "service": {
    "type": "wazuh"
  },
  "ecs": {
    "version": "1.0.1"
  },
  "agent": {
    "type": "filebeat",
    "ephemeral_id": "e7efb215-67fa-40ee-83af-e41c65964016",
    "hostname": "xxxxx.x.x..x.x.x.l",
    "id": "bb45d883-5186-45a0-87c2-b5c753d43d30",
    "version": "7.3.2"
  },


I dont see any WARN, ERROR on this log.
at /var/log/elasticsearch/elasticsearch.log | tail -n 100
[2019-10-25T00:00:01,200][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [wazuh-server] updating number_of_replicas to [0] for indices [wazuh-monitoring-3.x-2019.10.24]
[2019-10-25T00:15:00,754][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [wazuh-server] updating number_of_replicas to [0] for indices [wazuh-monitoring-3.x-2019.10.24]
[2019-10-25T00:30:01,331][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [wazuh-server] updating number_of_replicas to [0] for indices [wazuh-monitoring-3.x-2019.10.24]
[2019-10-25T00:45:01,009][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [wazuh-server] updating number_of_replicas to [0] for indices [wazuh-monitoring-3.x-2019.10.24]
[2019-10-25T01:00:00,962][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [wazuh-server] updating number_of_replicas to [0] for indices [wazuh-monitoring-3.x-2019.10.24]
[2019-10-25T01:15:01,395][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [wazuh-server] updating number_of_replicas to [0] for indices [wazuh-monitoring-3.x-2019.10.24]
[2019-10-25T01:30:01,110][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [wazuh-server] updating number_of_replicas to [0] for indices [wazuh-monitoring-3.x-2019.10.24]
[2019-10-25T01:45:01,591][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [wazuh-server] updating number_of_replicas to [0] for indices [wazuh-monitoring-3.x-2019.10.24]
[2019-10-25T02:00:01,420][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [wazuh-server] updating number_of_replicas to [0] for indices [wazuh-monitoring-3.x-2019.10.24]
[2019-10-25T02:15:00,889][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [wazuh-server] updating number_of_replicas to [0] for indices [wazuh-monitoring-3.x-2019.10.24]
[2019-10-25T02:30:01,638][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [wazuh-server] updating number_of_replicas to [0] for indices [wazuh-monitoring-3.x-2019.10.24]
[2019-10-25T02:45:01,134][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [wazuh-server] updating number_of_replicas to [0] for indices [wazuh-monitoring-3.x-2019.10.24]
[2019-10-25T03:00:00,967][INFO ][o.e.c.m.MetaDataCreateIndexService] [wazuh-server] [wazuh-monitoring-3.x-2019.10.25] creating index, cause [api], templates [wazuh-agent], shards [2]/[0], mappings [_doc]
[2019-10-25T03:00:01,190][INFO ][o.e.c.m.MetaDataMappingService] [wazuh-server] [wazuh-monitoring-3.x-2019.10.25/wgEyiIgSQCijJH_Z_Pk_pg] update_mapping [_doc]
[2019-10-25T03:13:27,533][INFO ][o.e.c.m.MetaDataCreateIndexService] [wazuh-server] [wazuh-alerts-3.x-2019.10.25] creating index, cause [auto(bulk api)], templates [wazuh], shards [3]/[0], mappings [_doc]
[2019-10-25T03:15:01,420][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [wazuh-server] updating number_of_replicas to [0] for indices [wazuh-monitoring-3.x-2019.10.25]
[2019-10-25T03:30:01,103][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [wazuh-server] updating number_of_replicas to [0] for indices [wazuh-monitoring-3.x-2019.10.25]
[2019-10-25T03:45:01,732][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [wazuh-server] updating number_of_replicas to [0] for indices [wazuh-monitoring-3.x-2019.10.25]
[2019-10-25T04:00:01,448][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [wazuh-server] updating number_of_replicas to [0] for indices [wazuh-monitoring-3.x-2019.10.25]

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

Daniel Moreno

unread,
Oct 25, 2019, 9:22:26 AM10/25/19
to Wazuh mailing list

Please install tcpdump in the machine where ElasticSearch is located. Execute it this way:


tcpdump src WAZUHMANAGERIP -i INTERFACEUSEDBYELASTIC -nn


You may get the interface to use by executing ifconfig, you will get an output similar to:


[root@centos11 ~]# ifconfig
eth0
: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet
10.0.2.15  netmask 255.255.255.0  broadcast 10.0.2.255
        inet6 fe80
::eb6e:598c:c7f1:54c5  prefixlen 64 scopeid 0x20<link>
        ether
08:00:27:c2:05:d3  txqueuelen 1000 (Ethernet)
        RX packets
697  bytes 639884 (624.8 KiB)
        RX errors
0  dropped 0 overruns 0  frame 0
        TX packets
324  bytes 23685 (23.1 KiB)
        TX errors
0  dropped 0 overruns 0  carrier 0 collisions 0

eth1
: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet
11.0.0.11  netmask 255.255.255.0  broadcast 11.0.0.255
        inet6 fe80
::a00:27ff:fe95:695  prefixlen 64 scopeid 0x20<link>
        ether
08:00:27:95:06:95  txqueuelen 1000 (Ethernet)
        RX packets
708  bytes 247745 (241.9 KiB)
        RX errors
0  dropped 0 overruns 0  frame 0
        TX packets
656  bytes 64839 (63.3 KiB)
        TX errors
0  dropped 0 overruns 0  carrier 0 collisions 0


Depending on your OS you might need to use ip a instead of ifconfig.

I’m using 11.0.0.11 as ElasticSearch IP address so I have to use the interface eth1. Once you launch the command you have to generate alerts in the manager, try to execute sudo su there for example.

After that, you should see events reported by the tcpdump command. They should be similar to:


13:10:53.291866 IP 11.0.0.14.50344 > 11.0.0.11.9200: Flags [S], seq 2985244782, win 29200, options [mss 1460,sackOK,TS val 201221 ecr 0,nop,wscale 7], length 0


In case you can’t see the events the problem could be related to a firewall configuration. Please check it out and create rules to allow filebeat to send the events to ElasticSearch.


Regards.

Suat Toksöz

unread,
Oct 25, 2019, 9:34:43 AM10/25/19
to Daniel Moreno, Wazuh mailing list
Hi Daniel,

What I am getting from tcpdump is , from wazuhServer to Client-Machine Flag P :

16:26:26.830729 IP WazuhServer > Client.48958: Flags [P.], seq 13995492:13995704, ack 8569, win 294, options [nop,nop,TS val 25774419 ecr 178099475], length 212
16:26:26.831857 IP 10.0.0.0 > 10.0.0.999.48958: Flags [P.], seq 13995704:13995916, ack 8569, win 294, options [nop,nop,TS val 25774419 ecr 178099475], length 212
16:26:26.832768 IP 10.0.0.0 > 10.0.0.999.48958: Flags [P.], seq 13995916:13996128, ack 8569, win 294, options [nop,nop,TS val 25774419 ecr 178099476], length 212
16:26:26.833082 IP 10.0.0.0 > 10.0.0.999.48958: Flags [P.], seq 13996128:13996340, ack 8605, win 294, options [nop,nop,TS val 25774419 ecr 178099477], length 212
^C
67827 packets captured
67839 packets received by filter
12 packets dropped by kernel

Not: Wazuh server + Elasticsearch on the same server


--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

Daniel Moreno

unread,
Oct 25, 2019, 9:54:55 AM10/25/19
to Wazuh mailing list
Okay, I think I didn't explained it correctly before.

If you are looking for the Wazuh's alerts you have to look at the wazuh-alerts* index pattern instead of filebeat*
As I mentioned, Filebeat only forwards the events created by the Wazuh manager and the indices used to store the alerts are called wazuh-alerts-3.x-DATE
To be able to see the Wazuh alerts please go to Management -> Kibana -> Index Patterns and set the pattern called wazuh-alerts-3.x-* as Default index.

Kibana.png


After that, you may see something similar to this within the Wazuh app:


Screenshot from 2019-10-25 15-54-30.png


I hope it solves the problem.

Regards.


Suat Toksöz

unread,
Oct 25, 2019, 10:13:39 AM10/25/19
to Daniel Moreno, Wazuh mailing list
Hi Daniel,

I am able to see the wazuh alert and wazuh index on kibana, without any problem.

My question is, can I create an index for filebeat and can I see the logs from filebeat.? Also I want to use the filebeat dashboards.



--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

Daniel Moreno

unread,
Oct 28, 2019, 6:08:10 AM10/28/19
to Wazuh mailing list
Hi Suat,

When Filebeat is configured by using our documentation it is set to send the alerts collected by the Wazuh manager to ElasticSearch.
These alerts are stored in an index called wazuh-alerts-3.x-YYYY.MM.DD and we use wazuh-alerts-3.x-* as index pattern.
Filebeat won't send any log but the Wazuh alerts, you need to configure it ( by using a module or directly in the configuration file ) to send different events to Elasticsearch.
By looking at your filebeat.yml file I can see you've set:

# Filebeat modules configuration
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml

These settings will configure the modules to be used, the problem is you still need to enable these modules. For example, to enable the apache module I would add the following to the filebeat.yml file:

# Wazuh - Filebeat configuration file
filebeat.modules:
  - module: wazuh
    alerts:
      enabled: true
    archives:
      enabled: false
  - module: apache

After changing the filebeat.yml file is necessary to restart Filebeat.
Reply all
Reply to author
Forward
0 new messages