Wazuh Backup - Snap

[Suat Toksöz]

Hi,

I want to have my elasticsearch data available for 3 years, first year must be available as fast as possible, the 2 year could be cold (I can reach rarely). What is the structure should I have for this?

How can I estimate my daily total data , that has been inserted to the elasticsearch?

Is there any zip options on the wazuh snap system?

Best regards,

Suat Toksoz

[Mayte Ariza]

Hi Suat,

That is a really hard question to answer because there are many variables concerning this matter: the number of documents, their average size, the Elasticsearch heap size, the shard size, the number of shards… 

It depends on your workload and your Elasticsearch server resources.

Are you using daily indices in Elasticsearch? If so you just need to get your Elasticsearch indices and check the primary storage size (pri.store.size) in wazuh-alerts indices to estimate your daily data:

curl 'https://localhost:9200/_cat/indices?v&s=index'

Regarding the wazuh snap system, could you explain it with more details? What do you mean by that? Do you want to take snapshots of your wazuh-alerts* indices or save your logs files ?

I truly recommend you to take a look at the followings resources in order to design your cluster using Elasticsearch best practices:

https://www.youtube.com/watch?v=ilP7tG6tabI&feature=youtu.be

https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster

https://www.elastic.co/blog/managing-time-based-indices-efficiently

Best regards,

Mayte Ariza

[Suat Toksöz]

Hi Mayte,

Here is the result of indices, I am using daily indices.

health status index                              uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .apm-agent-configuration           gti7ijhITau5TsmCw839cg   1   1          0            0       566b           283b
green  open   .kibana_1                          dBNHEKFCRs6FuYoWGLx81Q   1   1          6            2     89.4kb         44.7kb
green  open   .kibana_2                          Bb8k4oThSfKEmHZWgZr-pw   1   1        236           23      497kb        256.2kb
green  open   .kibana_task_manager_1             Y5GENRxwQVSGFUw5-l4x1g   1   1          2            0     76.3kb         38.1kb
green  open   .monitoring-es-7-2020.03.10        nCq-svABSYqeocI4OYjH-A   1   1     397978            0    450.3mb        225.1mb
green  open   .monitoring-es-7-2020.03.12        80jU3rY-QGOBJcexVvmltQ   1   1     447693       319432    549.1mb        274.2mb
green  open   .monitoring-es-7-2020.03.13        xxucsV16SW6jlmWkyJjkgA   1   1     604641       581256    743.5mb        379.4mb
green  open   .monitoring-es-7-2020.03.14        c4aNRZUmTKGz9uqOyC8uEQ   1   1     622830        40512    683.8mb        341.9mb
green  open   .monitoring-es-7-2020.03.15        c0J4KDuIQ-yxA4NhRirU7w   1   1     641659       101400      713mb        356.5mb
green  open   .monitoring-es-7-2020.03.16        cY8ISRlGQZGpuNoyobEUJA   1   1     304657       401024    763.9mb        381.7mb
green  open   .monitoring-kibana-7-2020.03.10    ZIGxkyWMQxCU75qO-6O_jg   1   1      12554            0      5.4mb          2.7mb
green  open   .monitoring-kibana-7-2020.03.12    jQRlNkXHQjGBgp9jICB0dA   1   1      12481            0      6.1mb            3mb
green  open   .monitoring-kibana-7-2020.03.13    Fi0HJat2RfChLMHihcfzaw   1   1      17279            0      7.6mb          3.8mb
green  open   .monitoring-kibana-7-2020.03.14    6QpTTaVpRMKIchdsTi9DMQ   1   1      17278            0      7.4mb          3.6mb
green  open   .monitoring-kibana-7-2020.03.15    w2f7qsXaSt2thDdF2we7Gg   1   1      17280            0      7.5mb          3.7mb
green  open   .monitoring-kibana-7-2020.03.16    W14oUi2gT3e7jdjnHaj1qw   1   1       6944            0      3.4mb          1.7mb
green  open   .security-7                        vcjrLHHXRlGC2yG741j-MA   1   1         50            5    169.2kb         92.4kb
green  open   elastalert_status                  GYp0jA5qRZ6Wlu16iJ5-ZQ   1   1       2518            0      5.2mb          2.5mb
green  open   elastalert_status_error            jNGFGWt4SJ-Fm52DmA2aXg   1   1      16145            0      1.9gb        997.4mb
green  open   elastalert_status_past             oOW8iLPLQyeDokoRsY1nKw   1   1          0            0       566b           283b
green  open   elastalert_status_silence          QYUweebsTpe0tzOII5HVwQ   1   1         38            0     21.7kb         10.8kb
green  open   elastalert_status_status           TIDNeq3XSieqE_VLYlw5tQ   1   1        132            0      122kb           65kb
green  open   fluentd-20200302                   MvECa0xFRrihoZ2qjBEYfQ   1   1       2233            0   1003.4kb        502.9kb
green  open   wazuh-alerts-3.x-2020.03.13        NE-ZtlP8S4WkWFOIzemSHw   3   1    9122978            0      7.5gb          3.7gb
green  open   wazuh-alerts-3.x-2020.03.14        Phta-5AHT6uwtuAxzSYt8g   3   1    5764010            0      4.6gb          2.3gb
green  open   wazuh-alerts-3.x-2020.03.15        P0XyIhXsRm2dOOlnoGNM7g   3   1    5088139            0        4gb            2gb
green  open   wazuh-alerts-3.x-2020.03.16        39AEuV97QgqogxgiSjRfpA   3   1    3353385            0      3.5gb          1.5gb
green  open   wazuh-monitoring-3.x-2020.03.13    Zlp5z7YySLqAGWiTlMXpfA   2   0      45248            0     10.5mb         10.5mb
green  open   wazuh-monitoring-3.x-2020.03.14    huvvn7H3QKqylbr4wSGOMA   2   0      45312            0     10.6mb         10.6mb
green  open   wazuh-monitoring-3.x-2020.03.15    J4q3ba2ATpKT2jZpAwk0tg   2   0      45312            0     10.4mb         10.4mb
green  open   wazuh-monitoring-3.x-2020.03.16    0IRZW5IfQRGyAVGryBDm3A   2   0      23206            0      6.1mb          6.1mb
green  open   winlogbeat-7.5.1-2020.03.05-000001 XGyLL_uaSKaOSHR2QPH4Qg   1   1    1491700            0      1.7gb        870.8mb

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/a83e1c49-d874-4acc-806f-b64ed4f028d0%40googlegroups.com.

--

Best regards,

Suat Toksoz

[Mayte Ariza]

Hi Suat,

I would not take into account the .monitoring indices since they are usually small and get deleted after seven days. 

The wazuh-monitoring* indices are also small so I would not worry about them when estimating the daily data. However, once there are no more write operations on past indices, it would be useful to reindex them in a new index (e.g. wazuh-monitoring-3.x-2020) to keep the number of shards as low as possible.

Based on the wazuh-alerts* indices from the last three days, you are indexing an average of 2.7gb per day on Elasticsearch. Considering your desired retention period, that would be almost 1TB on your hot Elasticsearch nodes and 2TB in your cold ones.

Of course, the workload may change over time, so it would need to be adjusted from time to time.

I hope it helps.

 

Best regards,

Mayte Ariza