Unable to send events from Filebeat to Logstash after the upgrade

[Ezequiel Larrarte]

Hi,

We have some issues with our environmet after the upgrade

Wazuh Manager 3.8.1 -> 3.10.2

ELK 6.5.4 -> 7.3.2

We are unable to receive events generated by Filebeat through Logstash.

We followed the next guide: https://documentation.wazuh.com/3.10/installation-guide/installing-elastic-stack/transform_logstash.html

It works if we configure Filebeat to send events directly to Elasticsearch, but we need Logstash in our setup.

First, we point Filebeat to Elasticsearch and create the pipeline:

filebeat setup --pipelines --modules wazuh

The, we configure Logstash to use the pipeline:

File: /etc/logstash/conf.d/01-wazuh-remote.conf

----------------------------------------------------------------------------

# Wazuh - Logstash configuration file

## Remote Wazuh Manager - Filebeat input

input {

  beats {

      port => 5000

      codec => "json_lines"

  }

}

output {

  elasticsearch {

      hosts => ["elasticsearch-00:9200"]

manage_template => false

      index => "wazuh-alerts-3.x-%{+YYYY.MM.dd}"

pipeline => "%{[@metadata][pipeline]}"

  }

}

----------------------------------------------------------------------------

We 've configured the Logstash output based on this document:

https://www.elastic.co/guide/en/logstash/current/use-ingest-pipelines.html

Extra info (versions):

Filebeat 7.3.2

ELK 7.3.2

Wazuh Manager 3.10.2

Wazuh Agents 3.8.1

Any help would be appreciated ...

Regards!

[Pablo Torres]

Hi Ezequiel,

We are currently working on improving our Logstash configuration guide (https://documentation.wazuh.com/3.10/installation-guide/installing-elastic-stack/transform_logstash.html), indeed latest versions are not working correctly with the steps mentioned in that guide, sorry for any inconvenience it may have caused you, let's try to fix your problem:

1. After following the Logstash guide, were you obtaining an error similar to this?: 

[2019-10-02T11:14:04,135][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"wazuh-alerts-3.x-2019.10.02", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x17c36c5c>], :response=>{"index"=>{"_index"=>"wazuh-alerts-3.x-2019.10.02", "_type"=>"_doc", "_id"=>"VSYtjG0BA5ftPKxbVIrH", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [host] of type [keyword] in document with id 'VSYtjG0BA5ftPKxbVIrH'. Preview of field's value: '{name=master}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:775"}}}}}

We can avoid that error dropping that field in Filebeat, in your filebeat.yml add this processor:

processors:
  - drop_fields:
    fields: ["host"]

2. There is also an error with the Logstash configuration file (we are currently working on improving our Logstash documentation and solve this issue as soon as possible):
The codec plugin is set to json_lines, it should be changed to json.

In short, the result files should be as follows:

 - Filebeat.yml (/etc/filebeat/filebeat.yml)

#Wazuh - Filebeat configuration file

filebeat.modules:

  - module: wazuh

    alerts:

      enabled: true

    archives:

      enabled: false

setup.template.json.enabled: true

setup.template.json.path: '/etc/filebeat/wazuh-template.json'

setup.template.json.name: 'wazuh'

setup.template.overwrite: true

setup.ilm.enabled: false

processors:

  - drop_fields:

    fields: ["host"]

output.logstash:

  hosts: ["REPLACE_WITH_LOGSTASH_IP:5044"]

- Logstash configuration file (/etc/logstash/conf.d/01-wazuh.conf)

# Wazuh - Logstash configuration file
## Remote Wazuh Manager - Filebeat input
input {
    beats {

        port => 5044
        codec => "json"
    }
}


output {
    elasticsearch {
        hosts => ["<REPLACE_WITH_ELASTIC_IP>:9200"]
        index => "wazuh-alerts-3.x-%{+YYYY.MM.dd}"
    }
 }

Then restart both Filebeat and Logstash, wait a few seconds and let me know if alerts are being received by Elasticsearch.

We keep working on this issue, if you are still getting errors, please attach error logs from Elasticsearch and Logstash, so we can give you further assistance:

1. Check Filebeat connection output:

filebeat test output

2. Elasticsearch logs

tail -n 100 /var/log/elasticsearch/<clusterName|elasticsearch).log

3. Logstash logs

tail -n 100 /var/log/logstash/logstash-plain.log

Kind regards,

Pablo Torres