Logstash or Elasticsearch keeps crashing

[Erik Vetters]

Hi,

since some days ... nearly every day logsthash or elasticsearch keeps crashing. And events do not show up in that period in the

wazuh kibana app.

If I restart the service elasticsearch and logsthash everything is ok again, expect that events when the logsthash/elastic crashed.

The only thing I have found is this in the logsthas log (many times in that timeframe)

[2018-07-11T23:54:43,883][INFO ][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>1}
[2018-07-11T23:55:47,886][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})

Any hints on this issue ?

Many Greetings

Erik

[jesus.g...@wazuh.com]

Hi Erik, 

Most common problem when you see that message is Elasticsearch falls into read-only mode because it hasn't got enough disk space.

 

Check where is your Elasticsearch storing data:

# cat /etc/elasticsearch/elasticsearch.yml | grep "path.data"

// Example output:path.data: /var/lib/elasticsearch

In example we are going to check /var/lib/elasticsearch:

# cd  /var/lib/elasticsearch
# du -sh

// Example output
30M     .

Check your file system free space:

# df -h

// Example output
S.ficheros     Tamaño Usados  Disp Uso% Montado en
/dev/sda2        100G   9,7G   91G  10% /
devtmpfs         1,9G      0  1,9G   0% /dev
tmpfs            1,9G      0  1,9G   0% /dev/shm
tmpfs            1,9G   8,8M  1,9G   1% /run
tmpfs            1,9G      0  1,9G   0% /sys/fs/cgroup
/dev/sda1        5,0G   173M  4,9G   4% /boot
/dev/sda5         90G    39M   90G   1% /home
tmpfs            380M    12K  380M   1% /run/user/42
tmpfs            380M      0  380M   0% /run/user/0

Depending on the results from the above commands we can continue in different way or solving it.

Regards,

Jesús

[Erik Vetters]

Ok ... Then I have that issue, that is correct ... Disk space is currently limited for some time until I can migrate paths to disk another image. 

I have posted in another thread how I can delete events in a specific timeframe (e.g from - till )

https://groups.google.com/d/msg/wazuh/M5AfXmX1TTI/XkoMCipcBAAJ

I need some help here ... I'm currently not yet that fit in elasticsearch

Many Greetings and help very appreciated

Erik

[jesus.g...@wazuh.com]

Ok Erik, then this problem is closed and now you know what was happening, let's

continue in https://groups.google.com/d/msg/wazuh/M5AfXmX1TTI/XkoMCipcBAAJ

Regards,

Jesús