You can delete all the indices - the search:populate task will generate the index for your live site, so there's no need to preserve it. In fact, sometimes to help troubleshoot we suggest users try manually deleting the index, like so:
- curl -XDELETE 'localhost:9200/atom'
I would suggest you delete all the indices you find. See also my general suggestions below.
c) Before I start looking at how to add a node, is there anything obvious I should try?
Some general suggestions:
First, make sure you have the correct version of ES installed for your AtoM version, and if you've previously had AtoM installed on this server, make sure there aren't multiple versions of ES and Java, as this may be causing conflicts. With AtoM 2.5, this should typically be Elasticsearch 5.6.
If you've upgraded recently, make sure you are using the correct version of Java. You can update Java to Version 8 with the following:
- sudo add-apt-repository ppa:openjdk-r/ppa
- sudo apt-get update
- sudoapt install openjdk-8-jre-headless software-properties-common
Like I said, you may want to confirm you don't have multiple conflicting versions of Java as well. When I run java -version in my Ubuntu 18.04 Vagrant box, here is what I get in return:
openjdk version "1.8.0_212"
OpenJDK Runtime Environment (build 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03)
OpenJDK 64-Bit Server VM (build 25.212-b03, mixed mode)
Note that Java version 1.8.0_131 or later is recommended for ES 5.6, as per this documentation
I don't think this is a permissions issue, but if you have made changes based on the above, we should ensure our permissions are properly set. If you do want to check permissions, the only real important filesystem permission setting can be configured in the root AtoM installation directory - run the following (it won't do any harm if the permissions are already correct):
- sudo chown -R www-data:www-data /usr/share/nginx/atom
In terms of memory: 8GB for your installation should definitely be fine. There is a small chance that ES itself doesn't have enough memory for the JVM heap_size, so if, once you've removed the additional indices, checked the permissions, etc. you are still encountering issues, you could try adjusting the heap size value. The size of the heap is declared in /etc/default/elasticsearch - - specifically the ES_HEAP_SIZE for your install. Once the value is changed you will have to restart Elasticsearch. Remember, this shouldn't be set higher than half of the total available RAM on the server. If you do want to try to adjust this, I've found the following links to help guide you with tuning ES:
There is also this StackOverflow thread on adjusting heap size:
Note that when I asked one of our developers about the best way to make JVM heap size changes, they told me the following:
"You can try setting the JVM heap size, but we have never had any luck with actually reducing the memory requirements of ES. Increasing the memory available to ES to at least 2GB or more has always been the better solution in our experience."
I personally think this should be minimum 4GB or more total memory, since I believe that the default JVM heap size in ES is 2GB.
Some other forum threads that might be handy:
If you need to remove and reinstall ES (for example, if you have the wrong version, or multiple versions running:
In this thread, a user shared how they adjusted the heap size:
Finally, if none of the above has helped resolve the issue, the next thing you could try is to see if the ES logs have any more information than what you've found in your stack trace. In our recommended Ubuntu installation instructions, the Elasticsearch (ES) log is normally located in /var/log/elasticsearch/elasticsearch.log and you could check there to see if there's more information available. Try doing some web searches with the error message to see if you can find further suggestions online - and remember to make sure they are for the correct version of ES!
In terms of cluster health - yellow should not be an issue. AtoM installations will generally show a yellow (warning) status for shard health. For now, I wouldn't worry too much about the yellow health status - I think it's primarily because we're only using one node, and ES is warning you that this could involve data loss if the node goes down. According to the ES docs, "yellow means that the primary shard is allocated but replicas are not." Since ES is not our primary data store and it's easy to repopulate, I don't think this is critical. See:
In terms of running multiple nodes: it's certainly possible. You can have a whole cluster of nodes for AtoM - however, our team doesn't have a lot of experience with these configurations, and it can get quite complex with shard replication, the various master nodes, etc. Be sure to consult the ES 5.6 documentation if you try this - though I would first focus on seeing if you can resolve the existing issues before diving into this.
Let us know how it goes!