Authority Record edits timing out

[thearch...@gmail.com]

Greetings all,

I recently came up against a strange (and previously unseen) behaviour that I'd really like to correct. While trying to edit an authority record to add dates and adjust the biographical sketch, AtoM seemed to hang up on saving the changes with a 504 Gateway Time-Out error. I have encountered 504 errors on larger edits in the past, but they always recover with the changes intact after they've completed their run - but this time, the authority record remains without changes saved. The end-to-end string from the error log appears as follows:

2018/03/07 11:44:05 [warn] 15665#15665: *20 an upstream response is buffered to a temporary file /var/cache/nginx/fastcgi_temp/4/00/0000000004 while reading upstream, client: 192.168.35.43, server: _, request: "GET /index.php/repository/add?linkExisting=true HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.atom.sock:", host: "lib-atoml01-dev.library.queensu.ca", referrer: "http://lib-atoml01-dev.library.queensu.ca/index.php/willis-d-g-mike/edit"

2018/03/07 11:45:18 [error] 15665#15665: *22 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.35.43, server: _, request: "POST /index.php/willis-d-g-mike/edit HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.atom.sock", host: "lib-atoml01-dev.library.queensu.ca", referrer: "http://lib-atoml01-dev.library.queensu.ca/index.php/willis-d-g-mike/edit"

2018/03/07 11:53:40 [error] 15665#15665: *46 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.35.43, server: _, request: "GET /index.php/willis-d-g-mike/edit HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.atom.sock", host: "lib-atoml01-dev.library.queensu.ca", referrer: "http://lib-atoml01-dev.library.queensu.ca/index.php/willis-d-g-mike"

I've made similar edits as recently as last week without any trouble, so I'm not too sure what has happened in the meantime to cause this. We've tried adjusting the php parameters and restarting all services, but to no avail.

Thanks in advance for any advice!

Jeremy

[Dan Gillean]

Hi Jeremy, 

A couple questions, based on things I have previously seen lead to timeouts on authority records: 

1. Are these authority records linked to a lot of descriptions?
   
2. More specifically, have you linked creators at all levels of description, instead of just linking at the parent level and allowing inheritance to handle the lower levels?
   
3. Do you have a lot of users/groups with custom permissions in your site?
   

Questions 1 and 2 have to do with timeouts due to the number of affected relations. Creator inheritance in AtoM is not just a way to implement the ICA's general principles of description (not repeating information unnecessarily at lower levels;  describing at the highest relevant level, etc), it's also a strategy to help mitigate this outcome. 

If you have linked at all levels and want an easy way to resolve this (and thereby hopefully reduce the occurrence of timeout issues), then you are quite possibly in luck: AtoM 2.5 has a new command-line task coming that will check your descriptions for unnecessarily repetitive linking, and remove links where inheritance would produce the same results. See: 

• https://projects.artefactual.com/issues/11910
  

We started this task development with a script that can be run independently, so you could use this in your current AtoM instance. You can find the script here: 

• https://gist.github.com/sbreker/9870b86c9a980658dbbc9b7e9a3b1287
  

Basic instructions on use: 

• Download the script - make sure you save it with a .php extension (not a .txt one!)
• Place the script in your root AtoM directory (e.g. /usr/share/nginx/atom)
• Use the tools:run command line task to execute the script - AKA: 

•  php symfony tools:run /usr/share/nginx/atom/actor_update.php

If you have a lot of relations but they are not due to 2, then you may need to experiment more with changes to the PHP execution limits. We've updated our documentation on this recently - see: 

• https://www.accesstomemory.org/docs/latest/admin-manual/installation/execution-limits/
  

Regarding 3, we have seen cases (particularly in portal sites such as the provincial/territorial sites in Canada, with many users accessing AtoM using custom group/user permissions to restrict their access to particular repositories, or specific descriptions etc) where adding a number of custom permissions can cause timeouts for all authenticated user groups. I've seen cases where authenticated users get timeout errors just trying to browse authority records - something that public users can do without issue! 

This is because when you add custom permissions, every node on the page has to be checked against potentially multiple permissions configurations, including inheritance from other groups, etc, before the node is loaded. So if you have a lot of descriptions linked to an authority, then each link has to be checked against each user/group to determine if it can be loaded or not first, which can take a while. I've previously written more about this in the forum, here: 

• https://groups.google.com/d/msg/ica-atom-users/ot8ufFi-ZWY/0t2fPuBNAwAJ
  

The permissions module in AtoM has not seen significant development or maintenance work since its creation in version 1.2, and is currently one of the biggest bottlenecks in AtoM. We would love to overhaul it, as always, it's more difficult to find community support for maintenance work than adding new features. 

I hope some of this helps! Let us know :)

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-users+unsubscribe@googlegroups.com.
To post to this group, send email to ica-ato...@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/c2ca63ca-6495-4112-9659-e7f82e60cea7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dan Gillean]

Glad I could help, Jeremy! Let us know how it goes, and if this helps resolve some of the timeout issues you've been experiencing. 

Regards, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

On Fri, Mar 9, 2018 at 11:25 AM, <thearch...@gmail.com> wrote:

Thank you so much for your response, Dan! In answer to your questions:

1. Are these authority records linked to a lot of descriptions? Yes they are
2. More specifically, have you linked creators at all levels of description, instead of just linking at the parent level and allowing inheritance to handle the lower levels? Yes we did - we actually hadn't noticed this in our import script, and I suspected this could have been the source of our troubles, so I thank you for confirming this!
   
3. Do you have a lot of users/groups with custom permissions in your site? No

We have downloaded the script, and are running it as I write. This fix is exactly what we needed!!

Cheers,

Jeremy

To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.

To post to this group, send email to ica-ato...@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/c2ca63ca-6495-4112-9659-e7f82e60cea7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-users+unsubscribe@googlegroups.com.

To post to this group, send email to ica-atom-users@googlegroups.com.

Visit this group at https://groups.google.com/group/ica-atom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/d1d8621d-61c3-488c-8c0f-157fcd0a49b3%40googlegroups.com.

[thearch...@gmail.com]

Hi Dan,

The script has worked like a charm ... but not without a hitch! After a handful of false starts (subsequent power outages shut down our active terminal and stopped the script - we learned our lesson and used "nohup" to decouple it from the terminal), we were able to start the script working. We have had it going for 24 hours, and managed to get through 12,500 descriptions, which, unfortunately, will take us 20 days to complete the task for our entire database. We've noticed that it is starting to slow down, too.

So my questions are:

1) Can we edit while the script is running? (I'm not keen on doing this, but thought I'd ask on behalf of staff anxious to enter descriptions)

2) Can we pause the script and pick it up from the most recently edited description to run on weekends?

3) Is there something we can do to prevent the script from slowing down? (e.g. clear the cache or something?)

Thanks in advance for any advice!

Jeremy

[thearch...@gmail.com]

Should have added in my previous post that we adjusted the php memory_limit to 2 GB (in case you were wondering!)

Cheers,

Jeremy

[Dan Gillean]

Hi Jeremy, 

We've had a bunch of issues with Google deciding to bounce messages from Artefactual staff today, so I'm reposting Steve's response, just in case. Apologies for the duplication, but we wanted to make sure a copy made it to the web forum for future reference. 

From Steve: 

Regarding your questions about the script:

1) Can we edit while the script is running? (I'm not keen on doing this, but thought I'd ask on behalf of staff anxious to enter descriptions)

a) The script is modifying the database - specifically it is modifying creators and creation events.  I would say that entering new Descriptions will not affect the script.  I would also state that modifying existing Descriptions should be fine.  I would avoid modifying creators and creation events on Descriptions while the script is running as there is a remote chance that an editor could make a change on the same record that is currently being modified - I would state thought that this probably has a low probability of occurring.

2) Can we pause the script and pick it up from the most recently edited description to run on weekends?

a) There is currently no mechanism to 'pause' the script.  However you should be able to just kill the script whenever you want to stop it - All work done to that point will be committed to the DB.  The next time you restart the script it will re-query the database to look for affected creation events on descriptions, and it should continue essentially where it left off.  This is probably the easiest way to manage this.

b) A second option available to you would be to choose one top level description to deal with for a given run. This can be set in the script by uncommenting and modifying the variable '$descrSlug' and setting it to the slug of a top level Archival Description. This will restrict the changes to that description hierarchy and the script will finish when this specific description tree has been fully examined.  This technique would require more management on your part, modifying the variable before each run, and recording what has already been completed.  In addition, for a large description hierarchy, there is still no guarantee that it would finish exactly within one weekend.

3) Is there something we can do to prevent the script from slowing down? (e.g. clear the cache or something?)

a) One possibility is that perhaps you are running into a large description hierarchy that has a large amount of linked creators - since any changes the script will make to creators linked to a specific description are dependent on what creators are linked to descriptions above this record in the hierarchy, the script needs to do a large amount of tree traversal to arrive at a decision as to what will be done to the creators on the current description.  The complexity of the description hierarchies can cause the script to take longer.

b) A second possibility is that your server is running out of memory, causing memory swapping. If this is the case, I would recommend limiting the script to specific description hierarchies - See Answer 2B above.  Limiting the scope should reduce the script's memory consumption.

Hope this helps!  Let us know if you have further questions.

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

On Fri, Apr 6, 2018 at 12:36 PM, <sbr...@artefactual.com> wrote:

Hi Jeremy

Regarding your questions about the script:

1) Can we edit while the script is running? (I'm not keen on doing this, but thought I'd ask on behalf of staff anxious to enter descriptions)

a) The script is modifying the database - specifically it is modifying creators and creation events.  I would say that entering new Descriptions will not affect the script.  I would also state that modifying existing Descriptions should be fine.  I would avoid modifying creators and creation events on Descriptions while the script is running as there is a remote chance that an editor could make a change on the same record that is currently being modified - I would state thought that this probably has a low probability of occurring.

2) Can we pause the script and pick it up from the most recently edited description to run on weekends?

a) There is currently no mechanism to 'pause' the script.  However you should be able to just kill the script whenever you want to stop it - All work done to that point will be committed to the DB.  The next time you restart the script it will re-query the database to look for affected creation events on descriptions, and it should continue essentially where it left off.  This is probably the easiest way to manage this.

b) A second option available to you would be to choose one top level description to deal with for a given run. This can be set in the script by uncommenting and modifying the variable '$descrSlug' and setting it to the slug of a top level Archival Description. This will restrict the changes to that description hierarchy and the script will finish when this specific description tree has been fully examined.  This technique would require more management on your part, modifying the variable before each run, and recording what has already been completed.  In addition, for a large description hierarchy, there is still no guarantee that it would finish exactly within one weekend.

3) Is there something we can do to prevent the script from slowing down? (e.g. clear the cache or something?)

a) One possibility is that perhaps you are running into a large description hierarchy that has a large amount of linked creators - since any changes the script will make to creators linked to a specific description are dependent on what creators are linked to descriptions above this record in the hierarchy, the script needs to do a large amount of tree traversal to arrive at a decision as to what will be done to the creators on the current description.  The complexity of the description hierarchies can cause the script to take longer.

b) A second possibility is that your server is running out of memory, causing memory swapping. If this is the case, I would recommend limiting the script to specific description hierarchies - See Answer 2B above.  Limiting the scope should reduce the script's memory consumption.

Hope this helps!  Let us know if you have further questions.

Steve

To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-users+unsubscribe@googlegroups.com.
To post to this group, send email to ica-atom-users@googlegroups.com.

Visit this group at https://groups.google.com/group/ica-atom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/f9a8a48c-f093-4bcf-aa10-713e7fba04df%40googlegroups.com.

[thearch...@gmail.com]

Hi Dan and Steve,

Thanks so much for this information. We will simply plan accordingly, and eventually get through all of the descriptions using the tips you've provided.

Cheers,

Jeremy