2.4, when deleting a description got connection timed out and 404 error message

[Cindy]

Hi,

When we tried to delete a description(44 series,150 files) from UI, we got: Connection timed out in the error log and 404 error message. Then the site was running very slowly and eventually ended up with a “The webpage cannot be found” message. 

We noticed when deleting process was started, the number of php-fpm processes increased drastically and MySQL server was 100% cpu usage. 

We've tried adjusting the php settings, restarting nginx, php-fpm and other services, it didn't work. We had to restart MySQL server and finally the site is back to normal.

Now we've asked administrator users not to do any deletions from UI until we get a solution on this.

I'm not too sure what has happened to cause this, it looks like the deletion processes are resource intensive, it affects all the other processes.

Any ideas and suggestions?

Thanks,

Cindy

[Dan Gillean]

Hi Cindy, 

AtoM can sometimes time out when trying to delete large descriptions, but I don't think I've seen what you describe happen before. It does sound like you have already tried what my first recommendations would be: restart services, check the webserver error logs, increase the PHP execution limits, etc. 

One theory one of our devs had was that perhaps the query log (or other MySQL logs) is/are enabled, and this is slowing everything down and consuming additional resources?

We'll need to access the MySQL command-prompt to check. To access the MySQL command prompt from the terminal, we need to know our database name, as well as the username and password we used during installation and configuration of the database. You can access it with these credentials like so: 

• mysql -u yourusername -pYOURPASSWORD atom

This should give you access to the MYSQL command prompt. In the example command above, I've used yourusername, YOURPASSWORD, and atom (as the database name) as placeholders - please swap in the correct ones used during the installation process. If you don't recall what they are, you can always try looking in 2 places to see the credentials you used - either in config/config.php, or in /root/my.cnf. 

You can press CTRL (or Command on a Mac) and C at any time to exit the MySQL command prompt, by the way. 

The following should show us if the general log is enabled: 

• SHOW GLOBAL VARIABLES LIKE 'general_log';
  

You can also disable the general log with: 

• SET GLOBAL general_log = 'OFF';

There are other logs as well - in fact, the slow query log might be useful to see if it is in fact a database operation that is causing issues. I have never tried these, but here are some links: 

• https://dev.mysql.com/doc/refman/5.6/en/server-logs.html (and related sub-chapters)
• On Percona, see: https://www.percona.com/doc/percona-toolkit/2.2/pt-query-digest.html 

Finally, it sounds like you are likely already aware, but there is an option to delete descriptions from the command-line, which can be useful for large hierarchies. See: 

• https://www.accesstomemory.org/docs/latest/admin-manual/maintenance/cli-tools/#delete-a-description  
  

Good luck! Let us know what you find! 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To post to this group, send email to ica-ato...@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/3498d0ab-db35-4902-a031-e676d226834e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Cindy]

Hi Dan,

I imported the database data to the local MySQL database on our test sever and tried deleting the same description from the test site UI. Got same errors in log:

2019/04/25 13:52:01 [error] 16158#16158: *489 upstream timed out (110: Connection timed out) while reading response header from upstream, client: ip-address, server: _, request: "POST /index.php/frank-c-peters-fonds/informationobject/delete HTTP/2.0", upstream: "fastcgi://unix:/var/run/php-fpm.atom.sock", host: "our_atom_site.ca", referrer: "https://our_atom_site.ca/index.php/frank-c-peters-fonds/informationobject/delete"

2019/04/25 13:52:01 [error] 16158#16158: *489 open() "/usr/share/nginx/atom/50x.html" failed (2: No such file or directory), client:  ip-address, server: _, request: "POST /index.php/frank-c-peters-fonds/informationobject/delete HTTP/2.0", upstream: "fastcgi://unix:/var/run/php-fpm.atom.sock", host: "our_atom_site.ca", referrer: "https://our_atom_site.ca/index.php/frank-c-peters-fonds/informationobject/delete"

2019/04/25 13:54:57 [error] 16158#16158: *489 upstream timed out (110: Connection timed out) while reading response header from upstream, client:  ip-address, server: _, request: "GET / HTTP/2.0", upstream: "fastcgi://unix:/var/run/php-fpm.atom.sock", host: "our_atom_site.ca"

2019/04/25 13:54:57 [error] 16158#16158: *489 open() "/usr/share/nginx/atom/50x.html" failed (2: No such file or directory), client:  ip-address, server: _, request: "GET / HTTP/2.0", upstream: "fastcgi://unix:/var/run/php-fpm.atom.sock", host: "our_atom_site.ca"

2019/04/25 14:01:28 [error] 16159#16159: *510 upstream timed out (110: Connection timed out) while reading response header from upstream, client:  ip-address, server: _, request: "GET / HTTP/2.0", upstream: "fastcgi://unix:/var/run/php-fpm.atom.sock", host: "our_atom_site.ca"

2019/04/25 14:01:28 [error] 16159#16159: *510 open() "/usr/share/nginx/atom/50x.html" failed (2: No such file or directory), client:  ip-address, server: _, request: "GET / HTTP/2.0", upstream: "fastcgi://unix:/var/run/php-fpm.atom.sock", host: "our_atom_site.ca"

• general_log was OFF when deletion was running.

mysql> SHOW GLOBAL VARIABLES LIKE 'general_log';

+---------------+-------+

| Variable_name | Value |

+---------------+-------+

| general_log   | OFF   |

+---------------+-------+

•  In my nginx.conf, I have:

        error_page 404 /404.html;

            location = /40x.html {

        }

        error_page 500 502 503 504 /50x.html;

            location = /50x.html {

        }

But 50x.html file was not found in the folder. Then I copied the /nginx /html/50x.html to the folder, "/usr/share/nginx/atom/50x.html" No such file or directory' error not showing anymore, but still got connection timed out error and the nginx 500 error page. 

I have following settings in php.ini:

post_max_size = 72M
upload_max_filesize = 64M

memory_limit = 512M

max_execution_limit = 30

Thanks a lot,

Cindy

To unsubscribe from this group and stop receiving emails from it, send an email to ica-ato...@googlegroups.com.

[José Raddaoui]

Hi Cindy,

If you followed the installation instruction in the documentation (it looks like you're using PHP-FPM from the error logs) you should edit the AtoM pool configuration to increase the "max_execution_time" and to make other PHP configuration changes for AtoM.

https://www.accesstomemory.org/fr/docs/2.4/admin-manual/installation/linux/ubuntu-xenial/#php

Best regards.

[Cindy]

Hi José,

Thank you for your suggestion. 

php_admin_value[max_execution_time]=120 was already set.

I changed max_execution_limit=120 in /etc/opt/remi/php70/php.ini as well.

Still got Connection timed out error.

Cindy

[José Raddaoui]

Hi Cindy,

Maybe it's the Nginx configuration which is causing the timeout or "request_terminate_timeout " is enabled in you PHP-FPM config. This may help:

https://easyengine.io/tutorials/php/increase-script-execution-time

Best regards.

[Cindy]

Thanks for the suggestion, I've managed to delete that Fonds (has 194 descendants) on our staging UI without 504 timeout error.

Here are the changes I made.

• /etc/opt/remi/php70/php.ini: max_execution_limit = 1200
• /etc/opt/remi/php70/php-fpm.d/atom.conf: php_admin_value[max_execution_time] = 1200
• /etc/nginx/default.d/atom.conf: fastcgi_read_timeout 1200 in location ~ ^/(index|qubit_dev)\.php(/|$) block

I tried increasing timeout to 600s, it worked for another Fonds (has 106 descendants) but was not enough for the first Fonds until I increased it to 1200s.

The deletion process took about 12 mins to run and mysql was about 90% usage when the process was running.

Although increasing fastcgi_read_timeout to 1200 worked on staging site(with local MySQL database), I'm still not sure if it will fix the issue on production site( with remote MySQL database).

I was wondering if you can confirm what is the likely cause the timeouts? What would your recommendation be to eliminate the timeouts for deleting a Fonds/Collection?

Thanks in advance!

Cindy