ElasticSearch suddenly stopped working

[Caleb Anderson]

A few months ago I reinstalled our Atom server from scratch, and up until today it has been working fine. 

I'm the only one who has access to the server, so I know noone has been messing with anything, but all of a sudden the elasticsearch service won't start. 

Its throwing the following error:

Oct 21 09:48:58 atom systemd[1]: Starting Elasticsearch...

Oct 21 09:48:58 atom systemd[1]: elasticsearch.service: Control process exited, code=exited status=200

Oct 21 09:48:58 atom systemd[1]: Failed to start Elasticsearch.

Oct 21 09:48:58 atom systemd[1]: elasticsearch.service: Unit entered failed state.

Oct 21 09:48:58 atom systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

I installed the version recommended by ATOM in the documentation.

From what I can see, error 200 is a permissions or missing file error - I've had a look at the .yml for elasticsearch, but can't really see anything that looks out of the ordinary there. 

Where should I start troubleshooting this? I've tried googling but they just say the error is one of those 2 things, and not where to look for reconfiguring.

[Dan Gillean]

Hi Caleb, 

Most often when we see these kind of issues with Elasticsearch, it ends up being a memory issue. I have seen users with 4GB of RAM on their servers report issues such as this (like this one). 

Specifically, it's often about the JVM heap size.  I believe the default Elasticsearch Heap size on Ubuntu is 2GB, which means you should have at least 4GB of RAM available - I don't know if the default Heap size is the same for CentOS or other operating systems. See the following for tips on adjusting heap size if needed: 

• https://www.elastic.co/guide/en/elasticsearch/reference/5.6/heap-size.html  
• Related info on mlockall, which can affect memory usage:  https://www.elastic.co/guide/en/elasticsearch/reference/5.6/setup-configuration-memory.html#mlockall

It can be hard to track down memory crash reports for ES as I don't think they are written to the Elasticsearch log.  You may need to check the syslog for out of memory error messages.

If you've upgraded recently, make sure you are using the correct version of Java. You can update Java to Version 8 with the following: 

• sudo add-apt-repository ppa:openjdk-r/ppa
• sudo apt-get update
• sudoapt install openjdk-8-jre-headless software-properties-common

When I run java -version in my Vagrant box, here is what I get in return: 

openjdk version "1.8.0_212"
OpenJDK Runtime Environment (build 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03)
OpenJDK 64-Bit Server VM (build 25.212-b03, mixed mode)

Note that Java version 1.8.0_131 or later is recommended for ES 5.6, as per this documentation. 

If you do want to check permissions, the only real important permission I can think of can be set in the root AtoM directory - run the following (it won't do any harm if the permissions are already correct):

• sudo chown -R www-data:www-data /usr/share/nginx/atom
  

For reference, some general tips on managing Elasticsearch are summarized in this thread: 

• https://groups.google.com/d/msg/ica-atom-users/Hdy0bqzoucg/w7ll2z4ADgAJ

I hope this helps! 

Regards, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

      

This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorised disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error-free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. King's Christian College is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the College.

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/d8f2d38e-aeee-4a4e-ba5f-9cd7a084c168%40googlegroups.com.