Alison Diamond
Dan Gillean
Hi Alison,
Any time you encounter a 500 error, the first thing we recommend is to check the webserver logs for more information on the nature of the error.
If you have followed our recommended installation instructions, you can check the most recent error log entries with:
sudo tail -f /var/log/nginx/error.log
See:
Feel free to share any related error message you find here if you're not sure how to proceed, and we will try to offer suggestions on next steps. However, I strongly suspect that the issue is that the job scheduler just needs a restart.
There are several tasks in AtoM that can take a long time to execute - longer than the default 1 minute timeout limit that most browsers impose to prevent requests from exhausting all system resources. To avoid the browser timeouts from terminating long-running processes midway through (and potentially causing data corruption), AtoM uses Gearman as a job scheduler to perform these tasks asynchronously in the background. This way, the user can keep using AtoM while the job runs, and the length of the job is not bound to 1 minute or less.
Examples of things that AtoM currently uses the job scheduler for:
Imports (both CSV and XML) from the user interface
Exports from the clipboard
Moving a description from one parent to another
Generating finding aids and reports
Handling DIP uploads from Archivematica
And more - including the fact that in some cases, the job scheduler will handle an update the search index when new descendant records are added to an archival hierarchy.
The details for installing the job scheduler (Gearman) can be found here:
If you have already installed and configured Gearman, then it's possible that the job scheduler simply needs a restart. You will know this is the case if you look in the error logs and see a message like, "No Gearman worker available that can handle the job."
To restart the atom-worker, see:
sudo systemctl restart atom-worker
Some notes on this:
You can check the status of the atom-worker at any time with: sudo systemctl status atom-worker.This is useful to make sure the restart was successful.
When the worker dies, AtoM will attempt to automatically restart it - however, to prevent the system from being caught in an endless loop, we added a restart limit - 3 retries every 24 hours. You can reset this fail limit with the following command:
sudo systemctl reset-failed atom-worker
Then run the restart command again if the previous attempts didn't work.
If none of these suggestions help to resolve the issue, and/or this is not what you find in the error logs, then please let us know what you discover, as well as more information about your installation (Operating system, PHP version, etc) and we can provide further suggestions.
If it's not the job scheduler, there are also a a few other common maintenance tasks you could try running that often resolve issues. All of the following commands should be run from the root AtoM installation directory - if you have followed our recommended installation instructions, then this is typically:
/usr/share/nginx/atom
If the commands won't run and you've confirmed you are in the right directory, try running them as the www-data user - this is the user to whom we assign all filesystem rights when installing and configuring AtoM. To do so, simply precede the commands below with sudo -u www-data.
Rebuild the nested set:
Useful for dealing with issues in hierarchies, such as archival descriptions or controlled vocabularies
php symfony propel:build-nested-set
See: https://www.accesstomemory.org/docs/latest/admin-manual/maintenance/cli-tools/
Generate slugs
Useful if a bad import or a timeout has left a record without a slug (the unique part of the URL to a record). By default the task will only generate slugs where they are missing for main entities.
php symfony propel:generate-slugs
See: https://www.accesstomemory.org/docs/latest/admin-manual/maintenance/cli-tools/#generate-slugs
Clear the application cache
Useful for making sure you're not seeing outdated cached content (which is normally helpful for speeding up page load times for end users).
php symfony cc
See: https://www.accesstomemory.org/docs/latest/admin-manual/maintenance/clear-cache/
Restart PHP-FPM
FPM has its own cache, so useful for doing in conjunction with the cache:clear command above
This command depends on the version of PHP you have installed (which you can always check using php -version)
PHP 7.0: sudo systemctl restart php7.0-fpm
PHP 7.2: sudo systemctl restart php7.2-fpm
Also, if you have memcached installed, it is good to clear that cache as well. This command should not have any ill effects if you don't have memcached installed, so you can always try it out:
sudo systemctl restart memcached
Repopulate the search index
Useful for ensuring that the search index is up to date, and that there aren't records in the database that don't show up in search or browse pages.
Note: of all these common maintenance tasks, this is the only one that can take a long time to run (depending on how many records you have), and WILL temporarily remove content from your user interface as the index is rebuilt, so this one is best run outside of business hours if possible. However, as a sort of measure, reindexing the data in our public demo site usually only takes a couple minutes.
php symfony search:populate
See: https://www.accesstomemory.org/docs/latest/admin-manual/maintenance/populate-search-index/
Note: If you have a lot of records, this task can take a long time to run, and will make records temporarily unavailable in the user interface as they are reindexed. Best to run this after business hours. However, you can also use some of the task options to reduce the time it takes by limiting how it runs - for example, using the --update option will not make existing records unavailable as the task progresses (it will progressively replace the index rather than deleting it first). The --exclude-types option can be used to target just certain entity types, rather than all of them, and the --slug option can be used just to target specific records, taking the slug as input. See the documentation for further details and options.
Let us know how it goes!
Cheers,
--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/e3bc2cd2-dd8e-45ec-8e1d-903ee40f19f4n%40googlegroups.com.
Alison Diamond
On 26 May 2021, at 18:32, Dan Gillean <d...@artefactual.com> wrote:
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CAC1FhZJ269DKVYNtBv-iW0Aw9rHA3k4BrDkvQkJ%2BHb%3D-ot%3D2rw%40mail.gmail.com.
Dan Gillean
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/7697B247-1637-4F51-AD5B-260B4DEC50BA%40gmail.com.
Ruaraidh Wishart
Dan Gillean
- sudo tail -f /var/log/apache2/error.log
- sudo chown -R www-data:www-data /usr/share/nginx/atom
- 2.6.4 - v184
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/3d7e7ca8-9b45-4c59-afe4-d5dae8c30320n%40googlegroups.com.