elasticsearch won't start

[Jason Youngquist]

Hi all.

I'm running Security Onion version 16.04.6 LTS patched as of 5/18/19.

I am having a problem where elasticsearch won't start.

I believe it might because I'm low on disk space, but I'm not  100% sure.  Sometimes elasticsearch will start, but then only for a handful of minutes and then it will die on me again.  It's like playing wack-a-mole.

Here's a copy of my sostat-redacted

and 

elasticsearch.log

kibana.log

Is disk space the culprit, or do you think it is something else?  Appreciate any thoughts.

If you need anything else from me, please let me know.

Thanks.

Jason Youngquist

[Juan Carlos Rodríguez]

Hi Jason,

You have many logs entries like this:

high disk watermark [90%]

The watermark in Elasticsearch is the disk usage limit and it prevents from indexing more documents or allocate shards in this node. This causes unexpected errors in Elasticsearch and must be fixed as soon as possible.

Having the cluster with problems to allocate shards and storing documents causes more errors to all the ecosystem such as the elastalert warning messages about missing shards.

Kibana itself is failing too due to the same reason.

At this point, my suggestion is to review which plugins are generating a huge amount of events like elastalert that you may want to disable or at least take care of what it’s doing.

If all your deployment is fine and you still need those events, you need to increase the disk space (you can also increase the watermark for Elasticsearch https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html), and finally, you probably will need add more nodes for better performance.

Once your Elasticsearch is fixed, rest of the components are expected to be working (including Kibana).

Let us know your thoughts and how we can try to help you.

Regards,
Juan Carlos