2.4.1 installation issue (configure search)

[pierre]

Greetings folks,

-slackware14.2 x86_64

-jdk8u192

-elasticsearch1.7.6

-php5.6.38

-mysql mariadb10.0.37

-nogearman <planning to install after>

-httpd2.4.37

-nofirewall

launching the installer from http://localhost work fine until the search configuration with ES?
either with the memory message or the attached stack trace in the file configure_search.pdf.

1-communication with sql database seems to work just fine

2-extending the size(s) in /etc/php.ini leads to no solution

3-what is wrong with elasticsearch ?

Thanks,
Pierre

[Dan Gillean]

Hi Pierre, 

I will start with some general suggestions. 

First, I'm not familiar with Slackware, so the first thing you should probably do is ensure that all the required PHP extensions, additional libraries, and other tools that AtoM needs are available as packages in Slack. If using Ubuntu is an option available to you, I do recommend it: we do all our development and testing using Ubuntu, and this is what we maintain documentation for. We are able to offer more troubleshooting assistance when we know the environment being used. 

Other O/S platforms *should* be able to run AtoM, but it's always a good idea to double-check that you can get everything you need! In some cases, the exact version of the extension, library, or tool is required - for example, in our Ubuntu 16.04 instructions, PHP 7.0 is required - we've discovered issues with later versions of PHP (such as 7.2) and while we are working on fixes for the 2.5 release, these are not yet corrected for AtoM 2.4, so using a higher version of PHP would cause issues. 

The second suggestion is to make sure you're installing the correct version of AtoM. Our current stable release (2.4.1) uses ES 1.7.6, but in the upcoming 2.5 release, we have upgraded Elasticsearch to v5.5 - in 2.5, trying to use ES 1.7 would cause many errors, just as using ES 5 in AtoM 2.4.1 would! 

My next recommendation is to make sure you have enough system resources - particularly memory - for the installation. Our requirements page includes some notes on what we use when deploying to hosted instances (such as via OVH or DigitalOcean, two external hosting providers we have used in the past for AtoM). There, we list the following recommendations: 

• Processor: 2 vCPUs @ 2.3GHz
  
• Memory: 7GB
• Disk space (processing): 50GB at a minimum for AtoM’s core stack plus more storage would be required for supporting any substantial number of digital objects.

You can likely get away with less memory once AtoM is installed, but ES is a memory hog during installation, so I would try to give it as much memory as you possibly can during installation -  you can always scale it back down after if needed. 

The next thing you could try is to see if the ES logs have any more information than what you've found in your stack trace. In our recommended Ubuntu installation instructions, the Elasticsearch (ES) log is normally located in /var/log/elasticsearch/elasticsearch.log and you could check there to see if there's more information available. Try doing some web searches with the error message to see if you can find further suggestions online -  and remember to make sure they are for the correct version of ES! 

You could also verify that ES is running, verify the cluster health, and check which version you have installed using cURL (run sudo apt-get install curl to install it if it's not already installed).  Run from the server where ES is installed and using the default port, you could try: 

• curl -XGET http://localhost:9200
• curl -XGET 'http://localhost:9200/_cluster/health?pretty=true' 

See also: https://www.elastic.co/guide/en/elasticsearch/reference/1.7/cluster-health.html

This might help you diagnose the issue. 

Other useful ES commands  in Ubuntu 16.04 - I'm not sure what process manager is used in Slackware so these commands may be different in your environment, but if systemd manages ES as a service in your environment, then you can use the following commands to control the service: 

• sudo systemctl stop elasticsearch
  
• sudo systemctl enable elasticsearch
  
• sudo systemctl start elasticsearch
  
• sudo systemctl status elasticsearch
  

If you ever needed to manually delete the search index from AtoM, use: 

• curl -XDELETE 'localhost:9200/atom'
  

However, if you haven't yet gotten to the step where you can run the web installer and name your ES index, this command likely won't work as of yet.

For permissions, AtoM expects everything to be owned by the www-data user. You can try resetting permissions with the following: 

• sudo chown -R www-data:www-data /usr/share/nginx/atom

Finally, I'm not a developer or a system administrator so I'm not sure, but it's possible that you might actually need your webserver in place to be able to properly configure ES via the web installer? You might want to go back a step and set up Nginx and any other missing dependencies before getting to the web installer step. 

Hopefully something here will be useful to you - please let us know how it goes! 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To post to this group, send email to ica-ato...@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/16216f90-986f-4a5c-831b-cfeadafef2c1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pierre]

Hi Dan,

thanks for the long detailed answer. Interesting that elasticsearch is a memory hog : ) , RAM is only 4Go on this production machine I work on.

yes most of the dependencies are installed by default on slackware and ownerships are www-data:www-data as per the atom.conf file in the /etc/php-fpm.d/ directory.

chown -R www-data:www-data /var/www

My aim is to stick with slackware but give me more time to try the install process on unbuntu 14 and see if i can pass the search config step.

I moved elasticsearch from 1.7.3 to 1.7.5 with the below curl results, not sure why version is still at 1.4.2

{
  "status" : 200,
  "name" : "Vashti",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "1.4.2",
    "build_hash" : "927caff6f05403e936c20bf4529f144f0c89fd8c",
    "build_timestamp" : "2014-12-16T14:11:12Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.2"
  },
  "tagline" : "You Know, for Search"
}
{
  "cluster_name" : "elasticsearch",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0
}

[Dan Gillean]

Hi Pierre, 

The fact that your output shows not only the incorrect version, but also a build timestamp from 2014 might suggest that there's another ES installation on this server? Is that possible? If so, you might want to investigate that angle further - you might be getting conflicts for this reason. 

Was there anything useful in the ES logs? Did you try stopping and restarting the ES service? 

ES is really just a memory hog during installation; after that it seems quite performant. If it's at all possible to increase the RAM temporarily that could still help. Keep us posted on what you find and what else you try! 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To post to this group, send email to ica-ato...@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/4a5857fa-4d91-405c-a0ca-029e04b1828d%40googlegroups.com.

[pierre]

Hi Dan,

2.4.1 installation (configure search) is solved, root cause on my production machine was the lack of memory setting in the jvm options, i had to create this file per elasticsearch documentation here:
https://www.elastic.co/guide/en/elasticsearch/reference/6.5/settings.html

Thanks,

/etc/elasticsearch/jvm.options

and add the following, java :-|

#
-Xmx2g


Also besides the above de-installing elasticsearch, deleting the 2 directories below and re-installing elasticsearch 1.7.5 gave sane outputs on cluster version and health, although status is yellow still.

/var/log/elasticsearch

and

/var/lib/elasticsearch

back to version sanity

{
  "status" : 200,
  "name" : "Lament",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "1.7.5",
    "build_hash" : "00f95f4ffca6de89d68b7ccaf80d148f1f70e4d4",
    "build_timestamp" : "2016-02-02T09:55:30Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.4"

  },
  "tagline" : "You Know, for Search"
}
{
  "cluster_name" : "elasticsearch",

  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 4,
  "active_shards" : 4,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 4,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0
}

[Dan Gillean]

Interesting! 

Thank you for updating us and sharing your solution, Pierre, I'm sure this will come in useful in future ES troubleshooting threads. 

Let us know if you run into any further issues, but otherwise, for now, I wouldn't worry too much about the yellow health status - I think it's primarily because we're only using one node, and ES is warning you that this could involve data loss if the node goes down. According to the docs, "yellow means that the primary shard is allocated but replicas are not." Since ES is not our primary data store and it's easy to repopulate, I don't think this is critical.  See: 

• https://discuss.elastic.co/t/elasticsearch-health-status-is-yellow-how-to-up-green/81765
  

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

On Tue, Dec 11, 2018 at 3:16 PM pierre <bell...@gmail.com> wrote:

Hi Dan,

2.4.1 installation (configure search) is solved, root cause on my production machine was the lack of memory setting in the jvm options, i had to create this file per elasticsearch documentation.

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To post to this group, send email to ica-ato...@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/8237bbf1-0a2b-498f-ae64-2d90c53ac347%40googlegroups.com.