Kibana cannot create wazuh-monitoring-3.x* index

[Kazim Koybasi]

To upgrade to ELK 6.1 I deleted all elasticsearch data and make a clean reinstall. When I connect Kibana Wazuh dashboard I take No matching indices found: No indices match pattern "wazuh-monitoring-3.x-*" error and kibana log show error below. How can I solve that issue?

Rejecting mapping update to [wazuh-monitoring-3.x-2017.12.21] as the final mapping would have more than 1 type: [wazuh, wazuh-agent]

[Javier Castro]

Hi Kazim,

you can't do an upgrade with those steps.

First, you need to apply the correct templates to the new elasticsearch installation due to big changes, both in the Elastic Stack (from 5.x to 6.x) and Wazuh (from 2.x to 3.x).

I recommend you to carefully follow our installation guide: https://documentation.wazuh.com/current/installation-guide/installing-elastic-stack/index.html

You also have an upgrading guide if that is your case scenario: https://documentation.wazuh.com/current/installation-guide/upgrading/different_major.html#upgrading-different-major

Best regards.

On Fri, Dec 22, 2017 at 6:29 PM, Kazim Koybasi <kazim....@gmail.com> wrote:

To upgrade to ELK 6.1 I deleted all elasticsearch data and make a clean reinstall. When I connect Kibana Wazuh dashboard I take No matching indices found: No indices match pattern "wazuh-monitoring-3.x-*" error and kibana log show error below. How can I solve that issue?

Rejecting mapping update to [wazuh-monitoring-3.x-2017.12.21] as the final mapping would have more than 1 type: [wazuh, wazuh-agent]

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/d744e403-9427-4387-9bed-efde2b42bf19%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kazim Koybasi]

Hi Javier,

Thanks for your quick response.I applied all steps in https://documentation.wazuh.com/current/installation-guide/installing-elastic-stack/index.html. But I can not find a workaround that error.I think it is related with this issues https://discuss.elastic.co/t/unable-to-create-index-with-more-that-1-type-in-6-x/106089 and https://www.elastic.co/guide/en/elasticsearch/reference/6.0/removal-of-types.html but I dont have so much elasticsearch knowledge.Should I remove one of mapping types wazuh or wazuh-agent ?

[Javier Castro]

Hi Kazin, 

the problem is once you've done the upgrade not following the steps it's a bit tricky to fix.

I'll stick with you so hopefully we can do it.

First, we need to see what templates your elasticsearch has stored:

curl localhost:9200/_cat/templates?v

We do that to determine which template was automatically created when you first upgraded so we can delete it.

That way when new alerts will be indexed, only the right template gets applied.

Also, it is helpful to see what indices have already been created:

curl localhost:9200/_cat/indices?v

Best regards.

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/69697302-1bd7-42b2-9479-0a8e8a4d1c38%40googlegroups.com.

[Kazim Koybasi]

Hi Javier,

My outputs for given commands is like that.

curl localhost:9200/_cat/templates?v
name                          index_patterns      order version
logstash                      [logstash-*]        0     60001
kibana_index_template:.kibana [.kibana]           0    
wazuh-agent                   [wazuh-monitoring*] 0    
wazuh                         [wazuh*]            0    
wazuh-kibana                  [.kibana*]          0    

curl localhost:9200/_cat/indices?v
health status index                       uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   .kibana                     w8uhibgvSa-pZeRLH9DNxA   5   1        239            1    233.5kb        233.5kb
yellow open   .wazuh                      DTHPxza-SYeq1l940y1yJg   5   1          1            0     10.4kb         10.4kb
yellow open   .wazuh-version              LA7FvCxbRMGEKR3aFCb7sg   5   1          1            0      6.7kb          6.7kb
green  open   wazuh-alerts-3.x-2017.12.22 7TtpOULeTFeHnaKpwxeuvw   1   0     123602            0    260.2mb        260.2mb
green  open   wazuh-alerts-3.x-2017.12.23 VfRctTE0SzWq8geQ51NmGg   1   0      78145            0     92.7mb         92.7mb

Best regards.

On Friday, 22 December 2017 20:29:13 UTC+3, Kazim Koybasi wrote:

[Javier Castro]

Hi Kazim,

there's the problem: you need to import the new templates to elasticsearch.

I think elasticsearch automatically overrides the templates if you import one with the same name, but, just to be safe, let's delete them before importing again.

1. Stop logstash (we don't want it to be indexing data while we are messing with the templates):

systemctl stop logstash

2. Delete the wrong templates:

curl -XDELETE localhost:9200/_template/wazuh

curl -XDELETE localhost:9200/_template/wazuh-agent

3. Import the good ones, and also a sample alert:

curl https://raw.githubusercontent.com/wazuh/wazuh/3.0/extensions/elasticsearch/wazuh-elastic6-template-alerts.json | curl -XPUT 'http://localhost:9200/_template/wazuh' -H 'Content-Type: application/json' -d @-

curl https://raw.githubusercontent.com/wazuh/wazuh/3.0/extensions/elasticsearch/wazuh-elastic6-template-monitoring.json | curl -XPUT 'http://localhost:9200/_template/wazuh-agent' -H 'Content-Type: application/json' -d @-

curl https://raw.githubusercontent.com/wazuh/wazuh/3.0/extensions/elasticsearch/alert_sample.json | curl -XPUT "http://localhost:9200/wazuh-alerts-3.x-"`date +%Y.%m.%d`"/wazuh/sample" -H 'Content-Type: application/json' -d @-

4. Please check your logstash configuration file before starting it again (/etc/logstash/conf.d/), it should say something like this in its output part (by the end of the file):

output {
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "wazuh-alerts-3.x-%{+YYYY.MM.dd}"
        document_type => "wazuh"
    }
}

5. Restart logstash:

systemctl restart logstash

Regards.

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/8771654e-52e6-4d0c-9854-efeefaa3e3db%40googlegroups.com.

[Javier Castro]

Hi Kazim,

It is safe because even if the process fails, you are not modifying the original data, you are just copying it, with the necessary modifications, into another index.

Keep in mind, though, that we are limited by what logstash and your own system is capable of doing: trying to restore large amounts of data in a low level computer may result in the system hanging, for example.

Best regards.

On Sun, Dec 24, 2017 at 3:06 PM, Kazim Koybasi <kazim....@gmail.com> wrote:

Thanks for your help. It works. I tried this procedure except that stopping logstash lot of time but I saw that is is important to stop logstash before doing any change in template or indices.

I want to ask one more thing is it safe to use https://github.com/wazuh/wazuh/tree/master/extensions/elasticsearch/restore_alerts restore script for restoring wazuh 3 alert ?