extremely slow search response from elasticsearch

[elizabet...@mcgill.ca]

Hello folks,

We in the process of migrating from atom 2.4 to 2.6 (RHEL 7.9, mysql 8, elasticsearch 5.6.16) we are migrating to a dev, qa and prod environment.  Migration to dev and qa went as expected.  However, in our prod environment we find that elasticsearch queries are so slow that nginx times out before a response can be returned.  I timed the response to 'php symfony search:status' and found it took over 2 minutes for elasticsearch to reply. The information returned is identical to that returned on our dev and qa servers.  The same thing occurs when we try querying cluster health: we get the same information back on all three servers, but on prod it takes over 2 minutes to get a reply from elasticsearch.

Has anyone experienced anything similar?

[Dan Gillean]

Hi Elizabeth, 

I've not seen this before personally. Are the dev, QA, and prod instances all on the same server? Do they share index names? What about other elements such as memory, or PHP pools, etc?

I would suggest you try some kind of resource monitoring so you can determine if the bottleneck is memory related (my first guess), or something else. We provide some basic suggestions on using htop in the Troubleshooting documentation here: 

• https://www.accesstomemory.org/docs/latest/admin-manual/maintenance/troubleshooting/#monitoring-active-processes-with-htop
  

That might give us more information about the nature of the bottleneck. Let us know what you find. 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/4437bc0c-0d3e-478e-b1ef-22f850fa5b40n%40googlegroups.com.

[Karl Goetz]

Hi Elizabeth

Does your upgrade involve changing servers or is it done in place?

thanks

kk

---

From: ica-ato...@googlegroups.com <ica-ato...@googlegroups.com> on behalf of elizabet...@mcgill.ca <elizabet...@mcgill.ca>
Sent: Wednesday, 2 June 2021 6:41 AM
To: AtoM Users <ica-ato...@googlegroups.com>
Subject: [atom-users] extremely slow search response from elasticsearch

 

Hello folks,

We in the process of migrating from atom 2.4 to 2.6 (RHEL 7.9, mysql 8, elasticsearch 5.6.16) we are migrating to a dev, qa and prod environment.  Migration to dev and qa went as expected.  However, in our prod environment we find that elasticsearch queries are so slow that nginx times out before a response can be returned.  I timed the response to 'php symfony search:status' and found it took over 2 minutes for elasticsearch to reply. The information returned is identical to that returned on our dev and qa servers.  The same thing occurs when we try querying cluster health: we get the same information back on all three servers, but on prod it takes over 2 minutes to get a reply from elasticsearch.

Has anyone experienced anything similar?

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/4437bc0c-0d3e-478e-b1ef-22f850fa5b40n%40googlegroups.com.

This email is confidential, and is for the intended recipient only. Access, disclosure, copying, distribution, or reliance on any of it by anyone outside the intended recipient organisation is prohibited and may be a criminal offence. Please delete if obtained in error and email confirmation to the sender. The views expressed in this email are not necessarily the views of the University of Tasmania, unless clearly intended otherwise.

[elizabet...@mcgill.ca]

Hi Dan,

Apologies for the delay in replying.  Our dev, QA and prod instances are each on different servers. The three servers are deployed with puppet and an ansible playbook is used to deploy packages, so they should be identical. But php smyfony search:status revealed that host is set to '127.0.0.1' on our dev and qa servers, but is set to 'localhost' on our prod server.  If I use 127.0.0.1 on the command line when querying elasticsearch on our prod server, I get answers immediately.  So I'm feeling hopeful that fixing network.host on elasticsearch.yml will fix the issue.

[Dan Gillean]

Good sleuthing, Elizabeth!

So the defaults we include are stored in the following file: 

• https://github.com/artefactual/atom/blob/qa/2.x/plugins/arElasticSearchPlugin/config/search.yml#L15
  

However, during installation this file is copied elsewhere - and depending on how you install, this can either be in config/search.yml, or in apps/qubit/config/search.yml. If you have both files, I'd suggest making the change in both, as I'm not sure which will override the other if both are present. 

Remember to restart elasticsearch after making any changes here, as well as possibly clearing the application cache and restarting PHP-FPM. 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/1995dcda-c7c8-4cdb-b9b9-1a939065d265n%40googlegroups.com.

[elizabet...@mcgill.ca]

Thanks indeed Dan,

So it turns out that there is a config/search.yml file on our prod server which indeed sets host to 'localhost'.  Changing host to '127.0.0.1' and restarting elasticsearch fixed the problem.

The odd thing is, there is no search.yml file in either of the two locations config/ or apps/qubit/config/ on our dev and qa servers.  I'd be curious to know what circumstances might result in a configuration with no search.yml in the atom code tree.  But in any case, thanks so much for helping us resolve this!

best regards,

[elizabet...@mcgill.ca]

correction: search.yml is indeed in the atom code tree on our dev and qa servers -  in it's original location:  plugins/arElasticSearch/config and has '127.0.0.1' as the host.