Filebeat 6.1.1 mapping conflicts on ElasticSearch 6.1.1

[Miguel Barbero]

Good morning,

currently, we have our own ElastiSearch 6.1.1 and Kibana 6.1.1 services running on their own indepent servers. We came from ES 5.3.2 so we have followed your upgrade procedure documentation. One of the stepts lead us to load a customized template

We've got a third server (OSSEC Manager) which is able to run Docker containers. In this server we've got running just a Docker Wazuh Manager 3.1.0 + Wazuh API 3.1.0 + Filebeat image 6.1.1, which is able to collect logs from several Wazuh agents and forward them to our ElasticSearch directly.

This is our current configuration on Filebeat 6.1.1:

filebeat.prospectors:
- type: log
  paths:
    - /var/ossec/logs/alerts/alerts.json
  json.message_key: log
  json.keys_under_root: true
  json.overwrite_keys: true
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["192.168.34.4:9200", "192.168.34.5:9200", "192.168.34.6:9200"]
  # Customized index name to wazuh-alerts-*
  index: "wazuh-alerts-4.x-%{+yyyy.MM.dd}"
setup.template.enabled: "false"

You'll notice that we are using wazuh-alerts-4.x-%{+yyyy.MM.dd}as index. The reason is that if we put 3.x (which we'd love it) we get a mapping conflict on Filebeat:

2018-02-08T12:58:35Z INFO States Loaded from registrar: 1
2018-02-08T12:58:35Z INFO Loading Prospectors: 1
2018-02-08T12:58:35Z INFO Starting Registrar
2018-02-08T12:58:35Z INFO Starting prospector of type: log; ID: 4769230626552641158
2018-02-08T12:58:35Z INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2018-02-08T12:58:35Z INFO Harvester started for file: /var/ossec/logs/alerts/alerts.json
2018-02-08T12:58:36Z INFO Connected to Elasticsearch version 6.1.1
2018-02-08T12:58:36Z INFO Connected to Elasticsearch version 6.1.1
2018-02-08T12:58:36Z INFO Connected to Elasticsearch version 6.1.1
2018-02-08T12:58:36Z WARN Can not index event (status=400): {"type":"illegal_argument_exception","reason":"Mapper for [message] conflicts with existing mapping in other types:\n[mapper [message] has different [norms] values, cannot change from disable to enabled, mapper [message] is used by multiple types. Set update_all_types to true to update [omit_norms] across all types.]"}

If we use wazuh-alerts-4.x... we got alerts on ElasticSearch but we aren't able to find these logs through Wazuh API (screen captures above)

This wasn't happenning on Filebeat 5.3.x + ES 5.3.x.

How could we set this index on Filebeat 6.1.1 to avoid any conflict with ES 6.1.1 and to get the great experience we got with ES 5.3 + Wazuh 2.0 + Filebeat 5.3 ;-D?

Thanks and kind regards

[jua...@wazuh.com]

Hi Miguel, and sorry for the late response.

Our recent releases of the Wazuh App provide the functionality of changing the current index pattern to be used on the different visualizations of the App. To change the current index pattern, open the app and go to the Settings tab (located on the gear icon from the top right corner of the screen). Then, go to the Pattern tab, and select the desired index-pattern you want to use. Now you should go again to Overview, for example, and see the visualizations properly with the data.

And finally, just one more thing. Can you please tell me what's your currently installed app? To do this, open a terminal as superuser and execute the following command, and paste here the result:

cat /usr/share/kibana/plugins/wazuh/package.json

If you still have more questions or doubts about anything related to Wazuh or the Wazuh App, don't hesitate to post a new comment.

Best regards,

Juanjo

[Miguel Barbero]

Good morning Juanjo,

don't worry at all.

Thanks a lot for your support.

First in all, I answer your question: "Can you please tell me what's your currently installed app?"

:~$ sudo cat kibana-6.1.1-linux-x86_64/plugins/wazuh/package.json
{
  "name": "wazuh",
  "version": "3.1.0",
  "revision": "0375",
  "kibana": {
    "version": "6.1.1"
  },
  "description": "Wazuh App",
  "main": "index.js",
  "keywords": [
    "kibana",
    "wazuh",
    "ossec"
  ],
  "initialPattern" : "wazuh-alerts-3.x-*",
  "author": "Wazuh, Inc",
  "license": "GPL-2.0",
  "repository": {
    "type": "git",
    "url": "https://github.com/wazuh/wazuh-kibana-app.git"
  },
  "bugs": {
    "url": "https://github.com/wazuh/wazuh-kibana-app/issues"
  },
  "homepage": "https://www.wazuh.com/",
  "dependencies": {
    "angular-animate": "1.6.5",
    "angular-aria": "1.6.5",
    "angular-cookies": "1.6.5",
    "angular-material": "1.1.1",
    "angular-md5": "^0.1.10",
    "ansicolors": "^0.3.2",
    "bootstrap": "3.3.6",
    "install": "^0.10.1",
    "lodash": "3.10.1",
    "needle": "^2.0.1",
    "node-cron": "^1.1.2"
  }
}

and secondly, I've just applied your change


​
 but we've got an error:

I've tried to restart Kibana and so on but I'm not sure what I should do. If we recover the default index pattern, the error disappears.

​

Thanks again and kind regards

Saludos cordiales, Miquel

-- 

	Miquel Barbero DevOps Engineer- XML Travelgate Tel: + 34 871 968 181 | Ext: 110 |  mbar...@xmltravelgate.com | www.xmltravelgate.com

--
You received this message because you are subscribed to a topic in the Google Groups "Wazuh mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/U2R-hQpoNqE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+unsubscribe@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/5e2c7e2b-7566-4c6c-b6f5-5e35a8e963e7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[jua...@wazuh.com]

Hello again Miguel,

Ok, so you're using our latest stable app for Wazuh 3.1 and Elastic 6.1.1.

Regarding your Visualize: "field" is a required parameter problem, we can try something and try to fix it.

On the left sidebar of the Kibana interface, locate the Management button (usually at the bottom of the list), and then go to Index Patterns. From there, select the wazuh-alerts-4.x-* index pattern, and find the Refresh button, with two arrows. You can see my attached screenshot.

Click that button to refresh the field list for that index pattern. Keep in mind that this action will reset the favourite statistics (and Kibana will also warn you about this prior to starting the refreshing process).

After that, go again to the app and select the 4.x index pattern. Please, tell me if now everything is working as expected.

Thanks for your patience.

Regards,

Juanjo

[Miguel Barbero]

Good morning Juanjo,

thanks for your support.

I've just followed yours procedure but I'm afraid it's not working.

I attach screen captures with the steps I've done.

I hope I have understood your indications correctly

Thanks for your patiente as well.

Kind regards

​

Saludos cordiales, Miquel

-- 

	Miquel Barbero DevOps Engineer- XML Travelgate Tel: + 34 871 968 181 | Ext: 110 |  mbar...@xmltravelgate.com | www.xmltravelgate.com

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/88070a50-387c-4c83-8e7f-cb026d99e366%40googlegroups.com.

[jua...@wazuh.com]

Hello again Miguel,

I think that we found a possible cause to your problem.

You've configured Logstash to create indices with the wazuh-alerts-4.x format. That's a very legit option and doesn't break anything at first. The problem comes now when you want to use our app. The Wazuh App requires some Elasticsearch templates, which are prepared to be used with the wazuh-alerts-3.x index format. So, if your indices aren't not filtered by the appropriate template, your data will be invalid to the app, since it cannot understand the data received from Logstash and Elasticsearch.

The templates are necessary for the visualizations in order to work properly because they expect specific data fields.

If you want to use the wazuh-alerts-4.x index pattern, modify the template file.

1. Download it at https://raw.githubusercontent.com/wazuh/wazuh/3.1/extensions/elasticsearch/wazuh-elastic6-template-monitoring.json 

2. Save it into a file called template.json

3. On the file, locate a line with this content:

"template": "wazuh-alerts-3.x-*",

And modify it to this:

"template": "wazuh-alerts-4.x-*",

4. Save again the file. Now, execute the following command on the same path as the template.json file:

cat template.json | curl -XPUT 'http://localhost:9200/_template/wazuh' -H 'Content-Type: application/json' -d @-

5. Insert a new sample alert with the appropriate index name:

curl https://raw.githubusercontent.com/wazuh/wazuh/3.1/extensions/elasticsearch/alert_sample.json | curl -XPUT "http://localhost:9200/wazuh-alerts-4.x-"`date +%Y.%m.%d`"/wazuh/sample" -H 'Content-Type: application/json' -d @-

6. Open again the app and select the new index-pattern.

Now, everything should work properly. If you still have problems, don't hesitate to post again so we can continue helping you.

Regards,

Juanjo

[jua...@wazuh.com]

Sorry Miguel, I forgot to mention something in my previous message.

Keep in mind that the old data won't be available to visualize on the app after you change the indexes, but the new alerts will do appear on the app when they're generated.

Regards,
Juanjo

[Miguel Barbero]

Good morning Juanjo and thanks a lot for your support and your appreciated help.

I start to think we are lost in traslation ;-D

Firt at all, we are not using exactly Logstash but Filebeat 6.1.1.

We don't want to use the wazuh-alerts-4.x index pattern at all. We are forced to use it because when we try to use wazuh-alerts-3.x we've got an error that I described in my first mail


​

There seems to be a conflict between the template used by Filebeat and your template (https://raw.githubusercontent.com/wazuh/wazuh/3.1/extensions/elasticsearch/wazuh-elastic6-template-monitoring.json).

I set our Filebeat to not use any template when it wants to write to our ElasticSearch (I think this is a new behaviour with Filebeat 6.1.x. It wasn't in Filebeat 5.3.x).

but, despite all that, we've got the conflict between templates.

So, If we apply your changes, we'll get the same conflict but now with wazuh-alerts-4.x.

Summarizing, we want to use wazuh-alerts-3.x but we are not able to avoid a conflict between the writting template used by Filebeat and the template used by Wazuh App.

This problem is totally new, we have been using Filebeat 5.3.x and Wazuh 2.x so far without any conflict. The problem has arised with Filebeat 6.1.1 and Wazuh 3.x.

I hope I have been more clear this once.

May be, we should open a case support to Filebeat, shouldn't we?

Thanks a lot again and kind regards.

​

​

Saludos cordiales, Miquel

-- 

	Miquel Barbero DevOps Engineer- XML Travelgate Tel: + 34 871 968 181 | Ext: 110 |  mbar...@xmltravelgate.com | www.xmltravelgate.com

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/4b97d445-030e-47c2-b5a2-7964ff09d646%40googlegroups.com.

[Miguel Barbero]

Good morning again,

in other words, I think if we are able to force to Filebeat 6.1.1 to use your Wazuh 3.x template we'll be able to avoid this conflict.

But I don't know how to make this point

Regards

Saludos cordiales, Miquel

-- 

	Miquel Barbero DevOps Engineer- XML Travelgate Tel: + 34 871 968 181 | Ext: 110 |  mbar...@xmltravelgate.com | www.xmltravelgate.com

[jua...@wazuh.com]

Hello again Miguel,

Jose told me the details about your architecture after your conversation with him. Now I see that you're not using Logstash at all.

Wazuh provides a solution in the form of a Kibana plugin, along with other Elastic Stack products such as Elasticsearch and Logstash (and Filebeat for distributed architectures). We use Logstash and configure it to read the contents of the alerts.json file. In addition to that, we configure some filters to adjust some of the alert content to adapt it to Elasticsearch, where we also insert a custom template, defining the different alert fields and its type.

Every part of the Elastic Stack has an important role, so the Wazuh app can work properly and show the alerts data to our users.

Regarding the fact that you had everything working without Logstash on the previous version of Wazuh, 2.x, and the Elastic Stack 5.x, keep in mind that the last major update from Elastic brought a bunch of modifications and breaking changes, so the team had to adapt the Wazuh app for the new version of Elastic.

The team will research about the possibility of avoiding Logstash. For now, I would like to ask you about how you had configured Filebeat without Logstash. Could you please send us the configuration files, so we can take a look at how did you achieve that?

Best regards,

Juanjo