Elasticsearch error / missing connection
110 views
Skip to first unread message
Helen
Apr 29, 2020, 9:36:22 AM4/29/20
to AtoM Users
We've been having problems with our AtoM installation giving us an error when trying to search or navigate - "Elasticsearch error: Elastica\Exception\ClientException"
I'm not able to rebuild the index - it tells me there's no enabled connection - and restarting the Elasticsearch service doesn't help.
It's virtual server, and managed by a hosting company - they installed Elasticsearch for us but don't have any more knowledge than I do :/
It shouldn't be a memory issue as the server has 8GB; there's only one Elasticsearch node, so it's possible that I need to add a node, but given what I'm reading on the forum, I think I'm right in thinking this shouldn't be necessary?
The cluster health is showing as yellow, with all the shards unassigned - however, I've realised that, because I'd run a couple of test installations before upgrading to the 2.x version from a much older installation, there are three indices instead of just one for the live site. I want to try removing the unused indices before I start trying to add a node, so:
a) Can I confirm that the index named in config/search.yml will be the index in use?
b) If I accidentally delete the live site's index, how easy would it be to reinstate?
b) Before I start looking at how to add a node, is there anything obvious I should try?
Any help is appreciated!
I'm not able to rebuild the index - it tells me there's no enabled connection - and restarting the Elasticsearch service doesn't help.
It's virtual server, and managed by a hosting company - they installed Elasticsearch for us but don't have any more knowledge than I do :/
It shouldn't be a memory issue as the server has 8GB; there's only one Elasticsearch node, so it's possible that I need to add a node, but given what I'm reading on the forum, I think I'm right in thinking this shouldn't be necessary?
The cluster health is showing as yellow, with all the shards unassigned - however, I've realised that, because I'd run a couple of test installations before upgrading to the 2.x version from a much older installation, there are three indices instead of just one for the live site. I want to try removing the unused indices before I start trying to add a node, so:
a) Can I confirm that the index named in config/search.yml will be the index in use?
b) If I accidentally delete the live site's index, how easy would it be to reinstate?
b) Before I start looking at how to add a node, is there anything obvious I should try?
Any help is appreciated!
Dan Gillean
Apr 29, 2020, 10:41:59 AM4/29/20
to ICA-AtoM Users
Hi Helen,
I do suspect that the multiple indices are contributing to the issues you are encountering. Regarding your questions:
a) Can I confirm that the index named in config/search.yml will be the index in use?
Yes, that's correct. If that configuration file is missing, AtoM will fall back to using the the default configuration file in the arElasticsearchPlugin. In your case, it doesn't sound like the config file is missing, but just wanted to mention this.
b) If I accidentally delete the live site's index, how easy would it be to reinstate?
You can delete all the indices - the search:populate task will generate the index for your live site, so there's no need to preserve it. In fact, sometimes to help troubleshoot we suggest users try manually deleting the index, like so:
- curl -XDELETE 'localhost:9200/atom'
I would suggest you delete all the indices you find. See also my general suggestions below.
c) Before I start looking at how to add a node, is there anything obvious I should try?
Some general suggestions:
First, make sure you have the correct version of ES installed for your AtoM version, and if you've previously had AtoM installed on this server, make sure there aren't multiple versions of ES and Java, as this may be causing conflicts. With AtoM 2.5, this should typically be Elasticsearch 5.6.
If you've upgraded recently, make sure you are using the correct version of Java. You can update Java to Version 8 with the following:
- sudo add-apt-repository ppa:openjdk-r/ppa
- sudo apt-get update
- sudoapt install openjdk-8-jre-headless software-properties-common
Like I said, you may want to confirm you don't have multiple conflicting versions of Java as well. When I run java -version in my Ubuntu 18.04 Vagrant box, here is what I get in return:
OpenJDK Runtime Environment (build 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03)
OpenJDK 64-Bit Server VM (build 25.212-b03, mixed mode)
Note that Java version 1.8.0_131 or later is recommended for ES 5.6, as per this documentation.
I don't think this is a permissions issue, but if you have made changes based on the above, we should ensure our permissions are properly set. If you do want to check permissions, the only real important filesystem permission setting can be configured in the root AtoM installation directory - run the following (it won't do any harm if the permissions are already correct):
- sudo chown -R www-data:www-data /usr/share/nginx/atom
In terms of memory: 8GB for your installation should definitely be fine. There is a small chance that ES itself doesn't have enough memory for the JVM heap_size, so if, once you've removed the additional indices, checked the permissions, etc. you are still encountering issues, you could try adjusting the heap size value. The size of the heap is declared in /etc/default/elasticsearch - - specifically the ES_HEAP_SIZE for your install. Once the value is changed you will have to restart Elasticsearch. Remember, this shouldn't be set higher than half of the total available RAM on the server. If you do want to try to adjust this, I've found the following links to help guide you with tuning ES:
- https://www.elastic.co/guide/en/elasticsearch/reference/5.6/vm-max-map-count.html
- https://www.elastic.co/guide/en/elasticsearch/reference/5.6/tune-for-indexing-speed.html
- https://www.elastic.co/guide/en/elasticsearch/reference/5.6/setup-configuration-memory.html
- The primary article on adjusting heap size in ES: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/heap-size.html
- Related info on mlockall, which can affect memory usage: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/setup-configuration-memory.html#mlockall
There is also this StackOverflow thread on adjusting heap size:
Note that when I asked one of our developers about the best way to make JVM heap size changes, they told me the following:
"You can try setting the JVM heap size, but we have never had any luck with actually reducing the memory requirements of ES. Increasing the memory available to ES to at least 2GB or more has always been the better solution in our experience."
I personally think this should be minimum 4GB or more total memory, since I believe that the default JVM heap size in ES is 2GB.
Some other forum threads that might be handy:
If you need to remove and reinstall ES (for example, if you have the wrong version, or multiple versions running:
In this thread, a user shared how they adjusted the heap size:
Finally, if none of the above has helped resolve the issue, the next thing you could try is to see if the ES logs have any more information than what you've found in your stack trace. In our recommended Ubuntu installation instructions, the Elasticsearch (ES) log is normally located in /var/log/elasticsearch/elasticsearch.log and you could check there to see if there's more information available. Try doing some web searches with the error message to see if you can find further suggestions online - and remember to make sure they are for the correct version of ES!
In terms of running multiple nodes: it's certainly possible. You can have a whole cluster of nodes for AtoM - however, our team doesn't have a lot of experience with these configurations, and it can get quite complex with shard replication, the various master nodes, etc. Be sure to consult the ES 5.6 documentation if you try this - though I would first focus on seeing if you can resolve the existing issues before diving into this.
Let us know how it goes!
he / him
--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/e9111751-3531-4ab6-bc2f-11dc6997c9a6%40googlegroups.com.
Helen
Apr 30, 2020, 10:56:13 AM4/30/20
to ica-ato...@googlegroups.com
Thanks Dan, this is all really helpful.
Unfortunately removing the indices didn't help (although on the plus side the cluster health is now green :/ ) - when I try to run search:populate it still says there's no enabled connection.
We're using AtoM 2.4, with ES 1.4.5 and Java 1.8.0_252 - elsewhere on the site we have some legacy code that requires PHP 5.x, and if I can't sort ElasticSearch as it currently stands, I'll obviously have to work out how to upgrade the server without disrupting that too much.
However, checking with the web hosts, although the ES_HEAP_SIZE is 2GB, the JVM heap size is set to 256M - 1GB - I'm guessing this should match the ES_HEAP_SIZE, as per https://www.elastic.co/guide/en/elasticsearch/reference/5.6/heap-size.html#heap-size ?
Unfortunately removing the indices didn't help (although on the plus side the cluster health is now green :/ ) - when I try to run search:populate it still says there's no enabled connection.
We're using AtoM 2.4, with ES 1.4.5 and Java 1.8.0_252 - elsewhere on the site we have some legacy code that requires PHP 5.x, and if I can't sort ElasticSearch as it currently stands, I'll obviously have to work out how to upgrade the server without disrupting that too much.
However, checking with the web hosts, although the ES_HEAP_SIZE is 2GB, the JVM heap size is set to 256M - 1GB - I'm guessing this should match the ES_HEAP_SIZE, as per https://www.elastic.co/guide/en/elasticsearch/reference/5.6/heap-size.html#heap-size ?
Dan Gillean
May 1, 2020, 11:23:25 AM5/1/20
to ICA-AtoM Users
Hi Helen,
From my (limited) understanding of the heap size documentation, I think that your interpretation is correct. I'd try raising this to at least 1GB min / 2GB max, and see if that helps. You can also set both min and max values to 2GB (or higher, if you have enough total system memory) if you want.
Also, I'm not sure if it might matter, but we tested and developed AtoM 2.4 against Elasticsearch 1.7, so you could potentially consider upgrading ES to see if that helps.
Alternatively we should have a new 2.5.4 release available next week, so waiting for that, and then upgrading to the latest version is also an option. If you don't want to wait, all the fixes are currently in the stable/2.5.x branch (we are just finalizing the tarball packaging etc), so you could conceivably install now, following our Option 2 instructions in the documentation and installing from our GitHub code repository instead of waiting for a tarball.
Keep in mind if you do upgrade, that a number of dependencies have changed. We've upgraded Elasticsearch to v5.6, moved PHP to v7, dropped support for Ubuntu 14.04 and PHP5, and added documentation for Ubuntu 18.04 (with PHP 7.2). You'll want to ensure that you've upgraded all relevant dependencies and don't have duplicate versions on your server, which could cause conflicts.
Cheers,
he / him
On Thu, Apr 30, 2020 at 10:56 AM Helen <ad...@womenslibrary.org.uk> wrote:
Thanks Dan, this is all really helpful.
Unfortunately removing the indices didn't help (although on the plus side the cluster health is now green :/ ) - when I try to run search:populate it still says there's no enabled connection.
We're using AtoM 2.4, with ES 1.4.5 and Java 1.8.0_252 - elsewhere on the site we have some legacy code that requires PHP 5.x, and if I can't sort ElasticSearch as it currently stands, I'll obviously have
However, checking with the web hosts, although the ES_HEAP_SIZE is 2GB, the JVM heap size is set to 256M - 1GB - I'm guessing this should match the ES_HEAP_SIZE, as per https://www.elastic.co/guide/en/elasticsearch/reference/5.6/heap-size.html#heap-size ?
--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/410a9e75-8239-4dad-b76f-44322d84c1af%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages