Lightsail 2.6.3 Install. Elasticsearch Failing

[EdderShredder]

Hello guys.

I have an issue with a new install of 2.6.3 on Lightsail.

The elasticsearch service will not start.

There are no logs in \var\log\elasticsearch.

I followed the installation instructions to the letter.

And followed the instruction to drop and import our database.

● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2021-04-08 10:52:16 UTC; 3min 29s ago
     Docs: http://www.elastic.co
  Process: 28634 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet -Edefault.path.logs=${LOG_DIR} -Edefault.path.data=${DATA_DIR} -Edefault.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
  Process: 28626 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
 Main PID: 28634 (code=exited, status=1/FAILURE)

Apr 08 10:52:15 ip-172-26-9-85 elasticsearch[28634]: 2021-04-08 10:52:15,917 main ERROR Null object returned for RollingFile in Appenders.
Apr 08 10:52:15 ip-172-26-9-85 elasticsearch[28634]: 2021-04-08 10:52:15,917 main ERROR Null object returned for RollingFile in Appenders.
Apr 08 10:52:15 ip-172-26-9-85 elasticsearch[28634]: 2021-04-08 10:52:15,917 main ERROR Null object returned for RollingFile in Appenders.
Apr 08 10:52:15 ip-172-26-9-85 elasticsearch[28634]: 2021-04-08 10:52:15,918 main ERROR Null object returned for RollingFile in Appenders.
Apr 08 10:52:15 ip-172-26-9-85 elasticsearch[28634]: 2021-04-08 10:52:15,918 main ERROR Unable to locate appender "rolling" for logger config "root"
Apr 08 10:52:15 ip-172-26-9-85 elasticsearch[28634]: 2021-04-08 10:52:15,919 main ERROR Unable to locate appender "index_indexing_slowlog_rolling" for logger config "index.indexing.slowlog.index"
Apr 08 10:52:15 ip-172-26-9-85 elasticsearch[28634]: 2021-04-08 10:52:15,919 main ERROR Unable to locate appender "index_search_slowlog_rolling" for logger config "index.search.slowlog"
Apr 08 10:52:15 ip-172-26-9-85 elasticsearch[28634]: 2021-04-08 10:52:15,919 main ERROR Unable to locate appender "deprecation_rolling" for logger config "org.elasticsearch.deprecation"
Apr 08 10:52:16 ip-172-26-9-85 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Apr 08 10:52:16 ip-172-26-9-85 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

[Dan Gillean]

Hi Ed, 

Unfortunately I don't personally know enough about Lightsail to help, but it could be that either there are some ports that are blocked for which ES needs access, or else some of the required dependencies are either missing or using different versions. For example, what version of Java do you have installed on the server? What version of PHP? Can you confirm that all required PHP extensions were properly installed? How much memory does the server have (we recommend 7GB if possible).

I suspect it may be a memory issue (this seems to be a common reason for ES failing), but I did find one post online with a similar issue, where the user needed to configure the Log Path and Data path. Not sure if that's helpful, but see: 

• https://stackoverflow.com/questions/50126396/null-object-returned-for-rollingfile-in-appenders
  

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/44038936-6e50-4986-8acd-5ca14573639an%40googlegroups.com.

[EdderShredder]

After much stress I think the issue was with the permissions for var/log/elasticsearch

This was set to not be writeable by the group. I also had to delete the files within as they were assigned root!

I then updated every package on the server and ended up having to remove the paths in elasticsearch.yml.

Using the default paths worked.

As with our previous live server. The elasticsearch log is growing exponentially.

A snippet of this log is as follows.

https://pastebin.com/QkSxtszf

[Jim Adamson]

Regarding your growing ES logs, attached is our customised log4j2.properties file (typically lives in /etc/elasticsearch). I've delineated our customisations with # Start local customisation and # End local customisation. It should help reduce disk space used (at the cost of fewer logging lines). I've also attached an unaltered log4j2.properties file for comparison.

Thanks, Jim

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/6249e209-38a0-426c-a3b3-1c6ddc2398e3n%40googlegroups.com.

--

Jim Adamson

Systems Administrator/Developer
Facilities Management Systems
IT Services

LFA/237 | Harry Fairhurst building | University of York | Heslington | York | YO10 5DD

+44 (0)1904 323859

Email disclaimer: https://www.york.ac.uk/docs/disclaimer/email.htm

[EdderShredder]

Ok I seem to have the unalterted log which is set to debug.

I will change this, delete the elasticsearch log and let it generate a new one.

Thanks Jim