Elasticsearch indexing error when pushing Wazuh events through logstash reg,

[venkatesh waran]

Hello Team,

       I have deployed Wazuh and integrated the wazuh manager with the Kibana for visualizations. I have pushed the wazuh alerts to logstash using filebeat. I'm using elasticsearch as a output plugin in logstash. When indexing the events to elasticsearch, I'm running into the following error.

     [2019-01-14T21:16:22,820][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"wazuh-alerts-3.x-2019.01.14", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x21139c71>], :response=>{"index"=>{"_index"=>"wazuh-alerts-3.x-2019.01.14", "_type"=>"doc", "_id"=>"sTcKTWgBob7xJyeb1zwn", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [wazuh-alerts-3.x-2019.01.14] as the final mapping would have more than 1 type: [wazuh, doc]"}}}}

     I have done the Wazuh mapping for the wazuh-alerts-3.x-* in elasticsearch and using 6.5 version. After a walk through the elasticsearch official site, I came to know 6.5 doesn't have the feature multi-type per indices. Please let me know how to resolve this issue. I will be grateful.

Thanks,

Venkatesh.L

      

[Jesus Linares]

Hi Venkatesh,

The multi-type will be invalid en Elastic 7, right now you should see only a warning but not the error: "Could not index event to Elasticsearch".

I think this could be happening for 2 reasons:

• You indexed events in Elasticsearch before applying the Wazuh template.
• You have several templates that apply to the same index pattern.

You can check the template that applied to an index from the UI:

Also, the _cat/template call has useful information: https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-templates.html.

Regards,

Jesus Linares.

[Kobayashi Maru]

Hi Jesus,

I got a similar issue and the output of 'GET /_cat/templates?v&s=name' on my cluster is as follows:

name                        index_patterns                order      version

.data-frame-internal-1      [.data-frame-internal-1]      0          7020099

.data-frame-notifications-1 [.data-frame-notifications-*] 0          7020099

.kibana_task_manager        [.kibana_task_manager]        0          7030099

.logstash-management        [.logstash]                   0          

.management-beats           [.management-beats]           0          70000

.ml-anomalies-              [.ml-anomalies-*]             0          7020099

.ml-config                  [.ml-config]                  0          7020099

.ml-meta                    [.ml-meta]                    0          7020099

.ml-notifications           [.ml-notifications]           0          7020099

.ml-state                   [.ml-state*]                  0          7020099

.monitoring-alerts-7        [.monitoring-alerts-7]        0          7000199

.monitoring-beats           [.monitoring-beats-7-*]       0          7000199

.monitoring-es              [.monitoring-es-7-*]          0          7000199

.monitoring-kibana          [.monitoring-kibana-7-*]      0          7000199

.monitoring-logstash        [.monitoring-logstash-7-*]    0          7000199

.triggered_watches          [.triggered_watches*]         2147483647 

.watch-history-9            [.watcher-history-9*]         2147483647 

.watches                    [.watches*]                   2147483647 

filebeat_template           [filebeat-*]                  0          60001

logstash                    [logstash-*]                  0          60001

wazuh                       [wazuh-alerts-3.x-*]          0          

I got the following error in Logstash, but not sure how to resolve it. I appreciate your help on the 2 points that you mentioned in your prior post. Thanks.

[2019-08-28T16:39:31,216][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"wazuh-alerts-3.x-2019.08.28", :_type=>"wazuh", :routing=>nil}, #<LogStash::Event:0x5c3469f3>], :response=>{"index"=>{"_index"=>"wazuh-alerts-3.x-2019.08.28", "_type"=>"wazuh", "_id"=>"H3Oa2mwBysZ5uUprH_za", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [wazuh-alerts-3.x-2019.08.28] as the final mapping would have more than 1 type: [_doc, wazuh]"}}}}

[Jesus Linares]

Hi Venkatesh.L,

What version of Wazuh and Elastic are you running?. Please, share the Logsatsh configuration.

Elasticsearch did a change regarding the mapping type:

• Indices created in 5.x with multiple mapping types will continue to function as before in Elasticsearch 6.x. 
• Indices created in Elasticsearch 6.0.0 or later may only contain a single mapping type.
• Types will be deprecated in APIs in Elasticsearch 7.0.0
• Types will be completely removed in 8.0.0.
  

It looks like you have 2 mappings: wazuh and doc. Probably, you need to remove the following line in the output of the Logstash configuration:

document_type => "wazuh"

Regards.

On Monday, January 14, 2019 at 4:54:46 PM UTC+1, venkatesh waran wrote: