Lightsail 2.6.3 Install. Elasticsearch Failing

148 views
Skip to first unread message

EdderShredder

unread,
Apr 8, 2021, 6:56:08 AM4/8/21
to AtoM Users
Hello guys.

I have an issue with a new install of 2.6.3 on Lightsail.
The elasticsearch service will not start.
There are no logs in \var\log\elasticsearch.

I followed the installation instructions to the letter.
And followed the instruction to drop and import our database.

● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2021-04-08 10:52:16 UTC; 3min 29s ago
     Docs: http://www.elastic.co
  Process: 28634 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet -Edefault.path.logs=${LOG_DIR} -Edefault.path.data=${DATA_DIR} -Edefault.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
  Process: 28626 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
 Main PID: 28634 (code=exited, status=1/FAILURE)

Apr 08 10:52:15 ip-172-26-9-85 elasticsearch[28634]: 2021-04-08 10:52:15,917 main ERROR Null object returned for RollingFile in Appenders.
Apr 08 10:52:15 ip-172-26-9-85 elasticsearch[28634]: 2021-04-08 10:52:15,917 main ERROR Null object returned for RollingFile in Appenders.
Apr 08 10:52:15 ip-172-26-9-85 elasticsearch[28634]: 2021-04-08 10:52:15,917 main ERROR Null object returned for RollingFile in Appenders.
Apr 08 10:52:15 ip-172-26-9-85 elasticsearch[28634]: 2021-04-08 10:52:15,918 main ERROR Null object returned for RollingFile in Appenders.
Apr 08 10:52:15 ip-172-26-9-85 elasticsearch[28634]: 2021-04-08 10:52:15,918 main ERROR Unable to locate appender "rolling" for logger config "root"
Apr 08 10:52:15 ip-172-26-9-85 elasticsearch[28634]: 2021-04-08 10:52:15,919 main ERROR Unable to locate appender "index_indexing_slowlog_rolling" for logger config "index.indexing.slowlog.index"
Apr 08 10:52:15 ip-172-26-9-85 elasticsearch[28634]: 2021-04-08 10:52:15,919 main ERROR Unable to locate appender "index_search_slowlog_rolling" for logger config "index.search.slowlog"
Apr 08 10:52:15 ip-172-26-9-85 elasticsearch[28634]: 2021-04-08 10:52:15,919 main ERROR Unable to locate appender "deprecation_rolling" for logger config "org.elasticsearch.deprecation"
Apr 08 10:52:16 ip-172-26-9-85 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Apr 08 10:52:16 ip-172-26-9-85 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

Dan Gillean

unread,
Apr 8, 2021, 11:08:50 AM4/8/21
to ICA-AtoM Users
Hi Ed, 

Unfortunately I don't personally know enough about Lightsail to help, but it could be that either there are some ports that are blocked for which ES needs access, or else some of the required dependencies are either missing or using different versions. For example, what version of Java do you have installed on the server? What version of PHP? Can you confirm that all required PHP extensions were properly installed? How much memory does the server have (we recommend 7GB if possible).

I suspect it may be a memory issue (this seems to be a common reason for ES failing), but I did find one post online with a similar issue, where the user needed to configure the Log Path and Data path. Not sure if that's helpful, but see: 
Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056
@accesstomemory
he / him


--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/44038936-6e50-4986-8acd-5ca14573639an%40googlegroups.com.
Message has been deleted

EdderShredder

unread,
Apr 11, 2021, 7:56:20 AM4/11/21
to AtoM Users
After much stress I think the issue was with the permissions for var/log/elasticsearch
This was set to not be writeable by the group. I also had to delete the files within as they were assigned root!
I then updated every package on the server and ended up having to remove the paths in elasticsearch.yml.
Using the default paths worked.

As with our previous live server. The elasticsearch log is growing exponentially.
A snippet of this log is as follows.

Jim Adamson

unread,
Apr 14, 2021, 11:17:12 AM4/14/21
to ica-ato...@googlegroups.com
Regarding your growing ES logs, attached is our customised log4j2.properties file (typically lives in /etc/elasticsearch). I've delineated our customisations with # Start local customisation and # End local customisation. It should help reduce disk space used (at the cost of fewer logging lines). I've also attached an unaltered log4j2.properties file for comparison.

Thanks, Jim



--
Jim Adamson
Systems Administrator/Developer
Facilities Management Systems
IT Services
LFA/237 | Harry Fairhurst building | University of York | Heslington | York | YO10 5DD

log4j2.properties
log4j2.properties.distrib
Message has been deleted

EdderShredder

unread,
Apr 16, 2021, 10:42:57 AM4/16/21
to AtoM Users
Ok I seem to have the unalterted log which is set to debug.

I will change this, delete the elasticsearch log and let it generate a new one.
Thanks Jim
Reply all
Reply to author
Forward
0 new messages