Rebuild search index and clear cache every week

[John Coquitlam]

Hi everyone,
We have been encountering the bad gateway on our site the past few weeks.
At first it was because ElasticSearch's logs taking up a lot of space.
After fixing Elasticsearch's logging, we would rebuild search index and clear cache.
Things would work afterwards.

Now we have to do rebuild search index and clear cache every week.
Clearing cache now takes around 2 hours.
It appears to be stuck on qubit_prod-cli.lck during the majority of the time.
Every time things work fine afterwards.

Anyone ever experience this?
Why is a rebuild search index needed every week?
However, it takes longer time now to clear cache?

I have attached screen captures of when clearing cache appears stuck.

Thanks,
John

[John Coquitlam]

Forgot to mention we are on ATOM 2.8 on Ubuntu 20.04 and Nginx and the standard installation following the instructions.

Thank you!

[Jim Adamson]

Incidentally we saw the same issue (502 Bad Gateway) around the time you posted your message, a few weeks ago (Ubuntu 20.04, AtoM 2.7.3). I don't know if things have now settled for you but for us, it was linked to low disk space. Oddly, even though I freed what I thought was enough space, leaving ~10% of the root filesystem free, this seemed not enough to make a lasting difference. So if this is continuing for you, you might try increasing the overall capacity of the disk, if you can.

I hope that helps.

Thanks, Jim

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/ica-atom-users/fe2cee83-7f58-4879-8916-06ce0517ee7cn%40googlegroups.com.

--

Jim Adamson

Systems Administrator/Developer
Facilities Management Systems
IT Services

LFA/023 | Harry Fairhurst building | University of York | Heslington | York | YO10 5DD

+44 (0)1904 323859

Email disclaimer: https://www.york.ac.uk/about/legal-statements/email-disclaimer/

[John Coquitlam]

Hi Jim,

Thanks for your reply. In our case, when we log in, the system indicates 

Usage of: 51.5% of 97.87GB

Memory usage: 53%

Now every morning, I do a sudo systemctl restart php7.4-fpm

and that keeps the system running.

If I forget a day, we will get the bad gateway error again.

Not really a solution.

Any advice?

Thanks,

John 

[Jim Adamson]

Hi John,

I don't have any specific advice for addressing this problem, but since Ubuntu 20.04 goes end-of-support at the end of this month then I imagine you'll be looking to upgrade anyway? AtoM 2.9 and Ubuntu 24.04 perhaps may solve the problem even if you don't work out specifically what caused it?

Thanks, Jim

To view this discussion visit https://groups.google.com/d/msgid/ica-atom-users/dc15908d-9b37-4070-91af-44b8dcce5503n%40googlegroups.com.

[John Coquitlam]

Thanks Jim.

We were going to wait until 2.9.1 but with this error we might have to go for 2.9

John

[John Coquitlam]

Hi everone,
I have looked further into this.
In /var/log/nginx/error.log, I found a lot of fpm failed lines

2025/05/20 10:25:39 [error] 64525#64525: *924907 connect() to unix:/run/php7.4-fpm.atom.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 191.31.169.49, server: _, request: "GET /index.php/informationobject/browse?media=print&places=1848&sf_culture=es&sort=lastUpdated&sortDir=asc&subjects=10888&topLod=0 HTTP/1.1", upstream: "fastcgi://unix:/run/php7.4-fpm.atom.sock:", host: "xxx"
2025/05/20 10:26:22 [error] 64525#64525: *927209 FastCGI sent in stderr: "PHP message: Empty module and/or action after parsing the URL "/ir/" (/)" while reading response header from upstream, client: 45.156.30.133, server: _, request: "GET /ir/ HTTP/1.1", upstream: "fastcgi://unix:/run/php7.4-fpm.atom.sock:", host: "xxx"

In /var/log/php7.4-fpm.log, it is recommending raising pm.max_children

[18-May-2025 00:39:50] WARNING: [pool atom] seems busy (you may need to increase
 pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are
2 idle, and 18 total children
[18-May-2025 00:39:59] WARNING: [pool atom] server reached pm.max_children setti
ng (30), consider raising it

Currently, I have just followed the instructions for setting up atom.conf
Any suggestions for the pm.max_children value?

pm = dynamic
pm.max_children = 30
pm.start_servers = 10
pm.min_spare_servers = 10
pm.max_spare_servers = 10
pm.max_requests = 200

Thanks,
John

[Jim Adamson]

Hi John,

I can't really comment on a recommended pm.max_children value, but I do know that we saw similar log lines, and lots of PHP-FPM processes when running top, when we were experiencing something similar to what you reported.

When you upgrade your OS and AtoM, if the problem continues, you might consider enabling the AtoM Javascript Challenge that was introduced in 2.9.1. If the problem is caused by bots overwhelming your instance, then hopefully this new feature will help.

Thanks, Jim

To view this discussion visit https://groups.google.com/d/msgid/ica-atom-users/ab253a31-c0a0-446b-b231-fb12c1d468f0n%40googlegroups.com.