mapper_parsing_exception from logstash

[C. L. Martinez]

Hi all,

 I am receiving the following logstash errors after installing X-Pack in logstash:

[2018-06-14T13:24:40,458][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"wazuh-alerts-3.x-2018.06.14", :_type=>"wazuh", :_routing=>nil}, #<LogStash::Event:0x31b18a15>], :response=>{"index"=>{"_index"=>"wazuh-alerts-3.x-2018.06.14", "_type"=>"wazuh", "_id"=>"kZ54_mMB86eT4RWzM1CD", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [host]", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:114"}}}}}
[2018-06-14T13:24:40,462][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"wazuh-alerts-3.x-2018.06.14", :_type=>"wazuh", :_routing=>nil}, #<LogStash::Event:0x73131863>], :response=>{"index"=>{"_index"=>"wazuh-alerts-3.x-2018.06.14", "_type"=>"wazuh", "_id"=>"kp54_mMB86eT4RWzM1CG", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [host]", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:114"}}}}}
[2018-06-14T13:24:46,303][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"wazuh-alerts-3.x-2018.06.14", :_type=>"wazuh", :_routing=>nil}, #<LogStash::Event:0x6f1b1dab>], :response=>{"index"=>{"_index"=>"wazuh-alerts-3.x-2018.06.14", "_type"=>"wazuh", "_id"=>"lZ54_mMB86eT4RWzSlBY", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [host]", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:114"}}}}}
[2018-06-14T13:24:46,304][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"wazuh-alerts-3.x-2018.06.14", :_type=>"wazuh", :_routing=>nil}, #<LogStash::Event:0x39da1d6b>], :response=>{"index"=>{"_index"=>"wazuh-alerts-3.x-2018.06.14", "_type"=>"wazuh", "_id"=>"lp54_mMB86eT4RWzSlBY", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [host]", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:114"}}}}}

My actual logstash filter and output is:

input {
    beats {
        port => 5000
        codec => "json_lines"
    }
}
filter {
    if [data][srcip] {
        mutate {
            add_field => [ "@src_ip", "%{[data][srcip]}" ]
        }
    }
    if [data][aws][sourceIPAddress] {
        mutate {
            add_field => [ "@src_ip", "%{[data][aws][sourceIPAddress]}" ]
        }
    }
}
filter {
    geoip {
        source => "@src_ip"
        target => "GeoLocation"
        fields => ["city_name", "country_name", "region_name", "location"]
    }
    date {
        match => ["timestamp", "ISO8601"]
        target => "@timestamp"
    }
    mutate {
        remove_field => [ "timestamp", "beat", "input_type", "tags", "count", "@version", "log", "offset", "type","@src_ip"]
    }
}

output {
    elasticsearch {
        hosts => ["127.0.0.1:9200"]
        index => "wazuh-alerts-3.x-%{+YYYY.MM.dd}"
        document_type => "wazuh"
        user => "logger"
        password => "mypass"
    }
}

Any idea?

[C. L. Martinez]

Solved: the problem was with filebeat... It has been updated to 6.3.x release ... Removing this release and install 6.2.4 solves the problem.

Sorry for the noise.

[jesus.g...@wazuh.com]

Hi C.L. Martinez,

The problem is Filebeat 6.3.0 is adding a new field by itself named "host" but we have the field "hostname" so "host" is not needed.

Regarding to Elasticsearch we don't have that field in our template but we don't want that field so the possible solution is to modify

the Logstash configuration (mutate section):

mutate {
     remove_field => [ "timestamp", "beat", "input_type", "tags", "count", "@version", "log", "offset", "type","@src_ip", "host" ]
}

Any case we are not supporting 6.3.0 up to we have tested it properly, we are currently working on it, thanks.

Best regards,

Jesús

[Juanjo Jiménez]

Hello C. L. Martinez,

We finally released a new version of the Wazuh app compatible with Elastic v6.3.0. You can read our upgrading guide to update your Wazuh and Elastic installation. You can also read our changelog to learn about the latest introduced features.

If you have any questions or comments, don't hesitate to open a thread.

Regards,

Juanjo

[C. L. Martinez]

Many thanks Juanjo ... I will test it ...

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/d77c6677-99eb-4f58-9eb0-8e995eb7aa70%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.