Wazuh indices reindex in elasticsearch for elastic-tuning

1,771 views
Skip to first unread message

SR

unread,
May 22, 2018, 7:01:28 PM5/22/18
to Wazuh mailing list
Hi wazuh Team,

I found that I have not set the sharding before setting up the wazuh infrastructure ..Now I am trying to follow below steps -


After reindexing wazuh-alerts- and wazuh-monitoring health looks good to green but still other indices are yellow. What I should do to make other indices to green.


# curl http://localhost:9200/_cat/indices

yellow open .wazuh                          52jUNhPcRmu7Gllov1wvHg 5 1     1 0  10.4kb  10.4kb

green  open wazuh-alerts-3.x-2018.05.22     dlSFaWysSLOccpU8C0ERNw 1 0 10437 0   2.2mb   2.2mb

yellow open .wazuh-version                  hARpeYzJRnavRf_Mv1H4dA 1 1     1 0   5.1kb   5.1kb

green  open wazuh-monitoring-3.x-2018.05.22 YrECiOihQx6aURuoJxiVuw 1 0    94 0  50.6kb  50.6kb

yellow open .kibana                         bZLFpfp3TdiYpVsYXi88WA 5 1   266 1 244.3kb 244.3kb



curl "http://localhost:9200/_template/wazuh?pretty&filter_path=wazuh.settings"

{

  "wazuh" : {

    "settings" : {

      "index" : {

        "number_of_shards" : "1",

        "number_of_replicas" : "0",

        "refresh_interval" : "5s"

      }

    }

  }

}


Thanks,

SR

jua...@wazuh.com

unread,
May 23, 2018, 3:41:04 AM5/23/18
to Wazuh mailing list
Hello SR, let me help you with this.

The steps provided in our documentation helped you to tune your wazuh-alerts and wazuh-monitoring indices. Now we have to tune the .wazuh and .wazuh-version indices. Since they were already created before the tuning process, they're using a different shards and replicas configuration. I'll guide you through the process to modify this.

Keep in mind that we're going to delete the .wazuh and .wazuh-version indices, so you'll have to re-insert your API entries on the Wazuh app.

1. Stop Kibana:
systemctl stop kibana

2. Modify the Wazuh app config file to adjust the number of shards and replicas for the .wazuh and .wazuh-version indices. The file is located at /usr/share/kibana/plugins/wazuh/config.yml, and it has a section like this:
# Configure .wazuh and .wazuh-version indices shards and replicas.
#wazuh.shards          : 1
#wazuh.replicas        : 1
#wazuh-version.shards  : 1
#wazuh-version.replicas: 1

Uncomment the lines and adjust the number of shards and replicas for each index. Let's suppose you want to have 1 shard and 0 replicas for each one:
# Configure .wazuh and .wazuh-version indices shards and replicas.
wazuh.shards          : 1
wazuh.replicas        : 0
wazuh-version.shards  : 1
wazuh-version.replicas: 0

Now save the file.

3. Delete the existing .wazuh and .wazuh-version indices. Don't worry, it's safe to delete them, but you'll have to re-insert your Wazuh API entries on the Settings tab:
curl -XDELETE localhost:9200/.wazuh
curl -XDELETE localhost:9200/.wazuh-version

4. Remove the Kibana optimization bundles. This will help to refresh your current Wazuh app installation:
rm -rf /usr/share/kibana/optimize/bundles

5. Restart the Kibana service:
systemctl restart kibana

Now, wait some minutes until the optimization process has finished. After opening the app for the first time and inserting the API entry, you can check again your indices health with this command:
curl -XGET localhost:9200/_cat/indices?v

Paste here the output. The health for each index should be now in green. Let me know about the results of this steps, and if you still have problems or questions, we'll keep helping you in everything we can.

Regards,
Juanjo

SR

unread,
May 23, 2018, 8:40:36 AM5/23/18
to Wazuh mailing list
Hi Juanjo,

Thank you for the detailed steps.. Below are my results and still those indices are yellow.  .kibana indices I deleted in this process and it turned to be green now but wazuh indices are still yellow.

1. Stop Kibana:
systemctl stop kibana

# systemctl stop kibana


2. Modify the Wazuh app config file to adjust the number of shards and replicas for the .wazuh and .wazuh-version indices. The file is located at /usr/share/kibana/plugins/wazuh/config.yml, and it has a section like this:
# Configure .wazuh and .wazuh-version indices shards and replicas.
#wazuh.shards          : 1
#wazuh.replicas        : 1
#wazuh-version.shards  : 1
#wazuh-version.replicas: 1

Uncomment the lines and adjust the number of shards and replicas for each index. Let's suppose you want to have 1 shard and 0 replicas for each one:
# Configure .wazuh and .wazuh-version indices shards and replicas.
wazuh.shards          : 1
wazuh.replicas        : 0
wazuh-version.shards  : 1
wazuh-version.replicas: 0

vi /usr/share/kibana/plugins/wazuh/config.yml





Now save the file.

3. Delete the existing .wazuh and .wazuh-version indices. Don't worry, it's safe to delete them, but you'll have to re-insert your Wazuh API entries on the Settings tab:
curl -XDELETE localhost:9200/.wazuh
curl -XDELETE localhost:9200/.wazuh-version

[root@wazuh-xxx vol1]# curl -XDELETE localhost:9200/.wazuh

{"acknowledged":true}[root@wazuh-xxx vol1]# curl -XDELETE localhost:9200/.wazuh-version

{"acknowledged":true}[root@wazuhcurl -XGET localhost:9200/_cat/indices?v


# curl -XGET localhost:9200/_cat/indices?v

health status index                           uuid                   pri rep docs.count docs.deleted store.size pri.store.size

green  open   wazuh-alerts-3.x-2018.05.22     dlSFaWysSLOccpU8C0ERNw   1   0      11109            0      2.4mb          2.4mb

green  open   wazuh-alerts-3.x-2018.05.23     k_Y0LzKzQaSkfGKOKrKsmQ   1   0       1346            0    422.9kb        422.9kb

green  open   .kibana                         ATyMQTr5RTGI3DaR_Tfcxw   1   0        266          156      236kb          236kb

green  open   wazuh-monitoring-3.x-2018.05.22 YrECiOihQx6aURuoJxiVuw   1   0        106            0    145.4kb        145.4kb

green  open   wazuh-monitoring-3.x-2018.05.23 j0UgIjiGQLeFoOFlLLL9xQ   1   0         52            0    134.9kb        134.9kb


4. Remove the Kibana optimization bundles. This will help to refresh your current Wazuh app installation:
rm -rf /usr/share/kibana/optimize/bundles

rm -rf /usr/share/kibana/optimize/bundle




5. Restart the Kibana service:
systemctl restart kibana

# systemctl restart kibana


Now, wait some minutes until the optimization process has finished. After opening the app for the first time and inserting the API entry, you can check again your indices health with this command:
curl -XGET localhost:9200/_cat/indices?v


# curl -XGET localhost:9200/_cat/indices?v

health status index                           uuid                   pri rep docs.count docs.deleted store.size pri.store.size

yellow open   .wazuh-version                  62QoWvTDR_CiRZZTe52itQ   1   1          1            0        5kb            5kb

green  open   wazuh-alerts-3.x-2018.05.22     dlSFaWysSLOccpU8C0ERNw   1   0      11109            0      2.4mb          2.4mb

green  open   wazuh-alerts-3.x-2018.05.23     k_Y0LzKzQaSkfGKOKrKsmQ   1   0       1346            0    422.9kb        422.9kb

yellow open   .wazuh                          A2IfkcTQSlG8g8Wx4iqJAw   1   1          0            0       230b           230b

green  open   .kibana                         ATyMQTr5RTGI3DaR_Tfcxw   1   0        266          156    236.2kb        236.2kb

green  open   wazuh-monitoring-3.x-2018.05.22 YrECiOihQx6aURuoJxiVuw   1   0        106            0    145.4kb        145.4kb

green  open   wazuh-monitoring-3.x-2018.05.23 j0UgIjiGQLeFoOFlLLL9xQ   1   0         52            0      135kb          135kb



Please let me know, what I am missing here

Thanks,
SR

jua...@wazuh.com

unread,
May 23, 2018, 10:49:49 AM5/23/18
to Wazuh mailing list
Hello again SR,

Could you please paste here your current app revision?
cat /usr/share/kibana/plugins/wazuh/package.json

We addressed an issue with the shards and replicas settings on a previous version of the Wazuh app, and there's a possibility you're not using the minimum revision with that bugfix (I've just tested this on my development environment and I was able to adjust the replicas to 0).

In addition to this, I would like to tell you that a yellow index status doesn't mean anything harmful or critical per se to your Elasticsearch cluster. Those indices won't increase their size severely and the yellow status only means that since you're using a single-node cluster, Elasticsearch can't reallocate the replicas to another node. The good thing is that your wazuh-alerts and wazuh-monitoring indices are properly configured and in green status.

In any case, we're here to help you with anything you need.

Regards,
Juanjo

SR

unread,
May 23, 2018, 11:44:44 AM5/23/18
to Wazuh mailing list
Hi Juanjo,

Here is the details on package.json. I am trying to setup monitoring on cluster health. I need to have cluster green :) .


# cat /usr/share/kibana/plugins/wazuh/package.json

{

  "name": "wazuh",

  "version": "3.2.1",

  "revision": "0387",

  "kibana": {

    "version": "6.2.3"

  },

  "description": "Wazuh App",

  "main": "index.js",

  "keywords": [

    "kibana",

    "wazuh",

    "ossec"

  ],

  "author": "Wazuh, Inc",

  "license": "GPL-2.0",

  "repository": {

    "type": "git",

    "url": "https://github.com/wazuh/wazuh-kibana-app.git"

  },

  "bugs": {

    "url": "https://github.com/wazuh/wazuh-kibana-app/issues"

  },

  "homepage": "https://www.wazuh.com/",

  "dependencies": {

    "angular-animate": "1.6.5",

    "angular-aria": "1.6.5",

    "angular-cookies": "1.6.5",

    "angular-material": "1.1.1",

    "angular-md5": "^0.1.10",

    "ansicolors": "^0.3.2",

    "install": "^0.10.1",

    "js-yaml": "3.10.0",

    "lodash": "3.10.1",

    "needle": "^2.0.1",

    "node-cron": "^1.1.2",

    "winston": "3.0.0-rc1"

  }

}


Thanks,

SR

Juanjo Jiménez

unread,
May 23, 2018, 12:28:52 PM5/23/18
to Wazuh mailing list
Hello again SR,

You have an old Wazuh app version. The package you're currently using has a bug in the configuration file and you can't set the shards and replicas of the .wazuh and .wazuh-version indices to 0. You have to update your Elastic version to at least 6.2.4 in order to use the Wazuh app package with the fix. I also recommend you to update your Wazuh manager to the latest available version, 3.2.2, so you can get the latest improvements and bugfixes.

You can read our upgrading guide in order to perform the process.

If you finally get to update your Elastic version, you can simply follow my previous steps in order to adjust the shards and replicas of those indices.

Let me know about the results, or if you have doubts about the upgrading process, I'll be glad to help you.

Regards,
Juanjo

SR

unread,
May 24, 2018, 9:25:07 PM5/24/18
to Wazuh mailing list
Hi Juanjo,


I tried to upgrade to 3.2.2 and 6.2.4 but still no luck  indices are still in yellow.

 cat /usr/share/kibana/plugins/wazuh/package.json

{

    "name": "wazuh",

    "version": "3.2.2",

    "revision": "0390",

    "kibana": {

        "version": "6.2.4"

    },

    "description": "Wazuh app",

        "rison": "^0.1.1",

        "winston": "3.0.0-rc1"

    }

}


Thanks,
SR

SR

unread,
May 29, 2018, 11:58:35 AM5/29/18
to Wazuh mailing list
Hi Juanjo,

In this steps of elastic tuning, is reindex the indices is mandatory even though I am in 6.2.3?


I am getting this error -

{"date":"2018-05-29T15:50:36.414Z","level":"error","location":"initialize.js init 6","message":"[index_not_found_exception] no such index, with { resource.type=\"index_expression\" & resource.id=\".wazuh-version\" & index_uuid=\"_na_\" & index=\".wazuh-version\" }"}


Thanks ,
SR 

SR

unread,
Jun 1, 2018, 5:59:11 PM6/1/18
to Wazuh mailing list
Hi Juanjo,

Now I updated wazuh to 3.2.2 and Elkstack to 6.2.4. On top that trying to do these elastic tuning. 

Is I need to do reindexing ? Is this mandatory pls confirm.


I followed your above steps and the steps mentioned in below link with out reindex


So far I see everything working fine and indices are in green.

green open wazuh-alerts-3.x-2018.06.01     -1BJwkL8RWCWXmqVKziGLg 1 0   26 0  80.3kb  80.3kb

green open wazuh-monitoring-3.x-2018.05.29 vxOmgsQkQDO_am9_xyiQtA 5 0   18 0 234.9kb 234.9kb

green open wazuh-monitoring-3.x-2018.06.01 x-vf1ipHTGSfSSEY-BvB1A 1 0   88 0   147kb   147kb

green open .wazuh                          MzEnbzGNSXWEGbG-k-jGjg 5 0    1 0  10.6kb  10.6kb

green open wazuh-alerts-3.x-2018.05.31     o3ooag18T7SYdCH2HlqWQg 5 0  162 0   585kb   585kb

green open wazuh-alerts-3.x-2018.05.29     0Zs-wZV2RR6Jb2pVYkRtzQ 5 0 1689 1 979.9kb 979.9kb

green open wazuh-alerts-3.x-2018.05.30     soeBr7r5RTqxNkl8YB8rHA 5 0 5418 0   1.7mb   1.7mb

green open .wazuh-version                  71Br8yFNT9CsZK18A9bczw 1 0    1 0   5.1kb   5.1kb

green open wazuh-monitoring-3.x-2018.05.31 7JrRsDBfT7GG65xPwB6Rhg 5 0   94 0 340.9kb 340.9kb

green open wazuh-monitoring-3.x-2018.05.30 SCeBk4V5Rm-UhET3D_tQdQ 5 0  148 0 329.7kb 329.7kb

green open .kibana                         CB2dCSGyQuilIjoPsXbvBQ 5 0  267 1 247.6kb 247.6kb



Could you please confirm is the reindex mandatory just to avoid future issues.


Thanks,

SR

SR

unread,
Jun 4, 2018, 12:11:08 PM6/4/18
to Wazuh mailing list
Hi Juanjo,

Any update on my query ? Please let me know. Really appreciate your help.

Thanks,
SR

Juanjo Jiménez

unread,
Jun 4, 2018, 6:51:33 PM6/4/18
to Wazuh mailing list
Hello again SR, and sorry for the late response.

It's not necessary to reindex if you didn't change the indices configuration after upgrading to the latest Wazuh and Elastic Stack version. Now that you have all your indices at green health, everything should be working fine. Keep in mind that if you modify your cluster, shard or replicas settings in the future, a reindex might be mandatory in order to make everything work properly.

If you need more help or you have any questions, please let me know so we can help you.

Regards,
Juanjo

SR

unread,
Jun 5, 2018, 1:16:29 PM6/5/18
to Wazuh mailing list
Hi Junajo,


I followed these steps to summarize ..


1. Wazuh 3.2.1 running from past 2 months
2. Upgraded wazuh to 3.2.2 and elk stack to 6.2.4
3. Applied shard and replicas settings for all different indices, and  /usr/share/kibana/plugins/wazuh/config.yml setting changes
Example -

curl -XPUT 'localhost:9200/wazuh-monitoring-*/_settings?pretty' -H 'Content-Type: application/json' -d'

{

  "settings" : {

    "number_of_replicas" : 0

  }

}

'

4. After checking the indices all were green

If you see here I have not reindexed. So I should reindex in step3? If not will cause issues in future?Please let me know.


Thanks,
SR

Juanjo Jiménez

unread,
Jun 8, 2018, 4:49:30 AM6/8/18
to Wazuh mailing list
Hello again SR, and sorry for the late response.

You only have to re-index the .wazuh-alerts and .wazuh-monitoring indices the first time. If I'm not mistaken, since you already did the reindex process before upgrading the Elastic Stack to the latest version, it's not required to do that twice.

After upgrading the Elastic Stack and the Wazuh app, you could properly update the .wazuh and .wazuh-version shards and replicas settings, so they could have a green health status.

In brief:
  1. After installing the Elastic Stack, follow the reindex process with this documentation guide.
  2. Apply shards and replicas settings on the Wazuh app config file.
  3. If you upgrade to a new version of Elastic Stack v6.x, it's not necessary to re-index again the indices if you already did that.
  4. An upgrade on the Elastic Stack or Wazuh app DOES require re-configure again the shards and replicas settings for .wazuh and .wazuh-version due to the configuration file being deleted when updating the app.
Let me know if everything is clear for you now, and if you need more help, don't hesitate to ask again.

Thanks for your patience.

Regards,
Juanjo

SR

unread,
Aug 18, 2018, 3:35:34 PM8/18/18
to Wazuh mailing list
Hi Juanjo Jiménez,

I was able to setup shards, replica and make the elastic indices health to green. But I see 

wazuh-monitoring-3.x-2018.0x.0x indices health getting into yellow state.Even though I have setup replica to 0.




# curl "http://localhost:9200/_template/wazuh?pretty&filter_path=wazuh.settings"

{

  "wazuh" : {

    "settings" : {

      "index" : {

        "number_of_shards" : "1",

        "number_of_replicas" : "0",

        "refresh_interval" : "5s"

      }

    }

  }

}



But when I run this curl command it will set to green but new indices will be create in yellow. Is there any permanent fix for this.


curl -XPUT 'localhost:9200/wazuh-monitoring-*/_settings?pretty' -H 'Content-Type: application/json' -d'

  {

    "settings" : {

      "number_of_replicas" : 0

    }

  }

  '


Thanks,
SR

Juanjo Jiménez

unread,
Aug 20, 2018, 2:41:01 AM8/20/18
to SR, Wazuh mailing list

Hello SR,

The wazuh-monitoring indices use a different index template from the wazuh-alerts ones. You need to modify that template in order to change the number of shards and replicas.

To do this, you can open the Dev Tools tab on the left sidebar of the Kibana interface. This tool allows you to send requests to Elasticsearch and easily modify some settings.

Put the following content on the left pane of the tab:

PUT _template/wazuh-agent
{
  "order": 0,
  "template": "wazuh-monitoring-3.x-*",
  "settings": {
    "index": {
      "refresh_interval": "5s",
      "number_of_shards": "1",
      "number_of_replicas": "0"
    }
  },
  "mappings": {
    "wazuh-agent": {
      "properties": {
        "@timestamp": {
          "type": "date",
          "format": "dateOptionalTime"
        },
        "status": {
          "type": "keyword"
        },
        "ip": {
          "type": "keyword"
        },
        "host": {
          "type": "keyword"
        },
        "name": {
          "type": "keyword"
        },
        "id": {
          "type": "keyword"
        },
        "cluster": {
          "properties": {
            "name": {
              "type": "keyword"
            }
          }
        }
      }
    }
  }
}

And click on the green Play button on the top of the request block.

You should get a response like this (ignore the deprecation warning):

{"acknowledged": true}

Now, all the newly created wazuh-monitoring-3.x-* indices will have this new shards and replicas settings.

Let me know if this helps.

Regards,
Juanjo


--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/b844890c-d445-42e5-b083-247e232db345%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

SR

unread,
Aug 21, 2018, 1:57:36 PM8/21/18
to Wazuh mailing list
Hi Juanjo,

I made above changes and so far it's looking good. We will monitor for few days.

Thanks,
SR

Juanjo Jiménez

unread,
Aug 21, 2018, 2:29:09 PM8/21/18
to SR, Wazuh mailing list
Hi SR,

That's great! Let us know if you have more questions or problems.

Regards,
Juanjo

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/d0f4c524-9fbb-456c-8fc0-57cbeb076750%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages