How could attempting to access one particular top level record result in a 500 error?
67 views
Skip to first unread message
Patrick Goetz
Jan 22, 2020, 4:01:38 PM1/22/20
to AtoM Users
We're still experimenting with Atom 2.5.3 and haven't put it in production yet. Yesterday our digital archivist uploaded a dozen or so records from a CSV file. There was a bad character in the file, resulting in an error messag
e:
...
Row 83: Matching description found, updating using delete and replace; row (id: 606, culture: en, legacyId: DP-01)...; Malformed UTF-8 characters, possibly incorrectly encoded
[info] [2020-01-21 09:26:29] Job 681 "arFileImportJob": Job finished.
but the records appeared to otherwise load and are visible from the web interface. The problem is one particular top level record can now mp longer be accessed or deleted (and not the one containing the offending non-UTF-8 character, which was in a sub-record of a different top level entry). All the other top level records can be accessed and deleted, but when I try to acess this particular record I get a 500 Internal Server Error.
This problem began after the CSV file was loaded. That particular record was already existant and useable but was deleted and reloaded from CSV. I've looked carefull at the CSV file, and the entry for the problematic record exactly matches the syntax for another top level record which behaves normally.
Checking the nginx log files I see:
2020/01/22 20:14:10 [error] 207#207: *433 FastCGI sent in stderr: "PHP message: Parent Resource id '1' does not exist" while reading response header from upstream, client: 127.0.0.1, server: _, request: "GET /index.php/general-convention-of-the-episcopal-church HTTP/1.1", upstream: "fastcgi://unix:/run/php7.2-fpm.atom.sock:", host: "erap-atx:8084", referrer: "http://catalog.episcopalarchives.org/index.php/informationobject/browse
Is there any way to recover from this without wiping out the entire database and starting over, say by editing the mysql database directly? If so what tables need to be modified in order to get rid of the offending record?
Dan Gillean
Jan 23, 2020, 6:15:08 PM1/23/20
to ICA-AtoM Users
Hi Patrick,
It sounds like the import got corrupted mid process. I have some initial recommendations to try - after that I will follow up with my team and see if they have further ideas if these don't work:
First, I strongly recommend you make a backup of your data first, just in case!
It's possible that it's just the nested set that is corrupted - you could try rebuilding it with the following command, run from AtoM's root installation directory:
I would also suggest clearing the application cache and repopulating the search index after:
- php symfony cc
- php symfony search:populate
If that doesn't work, you could perhaps try using the command-line to delete the resource:
- php symfony tools:delete-description slug
- Where slug in the above is the slug or permalink (the unique part of the URL) of the target description
- See: https://www.accesstomemory.org/docs/latest/admin-manual/maintenance/cli-tools/#delete-a-description
Finally, just to make sure there aren't any other hidden problems, I would recommend the following, even if one of the above solutions does work - on our Troubleshooting page we have a SQL query that helps identify common forms of data corruption in AtoM - see:
Let us know how it goes!
he / him
--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/cf417fd2-1d93-4a84-8ce1-4834d1134b66%40googlegroups.com.
Patrick Goetz
Jan 29, 2020, 12:04:57 PM1/29/20
to AtoM Users
Dan, thanks so much for your detailed response! I question the wisdom of using nested-sets in mysql rather than just going with a graph DB (e.g. Dgraph?) for tree-like hierarchies, but maybe none of the open source graph DBs were or are ready for prime time? (I have limited experience with this technology.) Looking at how nested sets work it does feel like using a hammer to install screws. But in any case, this command:
php symfony propel:build-nested-setcompletely resolved the issue were having, so I looked no further. Knowing how to use raw SQL to detect missing slugs could be useful in the future, so I'm adding the entirety of your response to my atom catalog notes.
Thanks again!
Dan Gillean
Feb 5, 2020, 1:22:18 PM2/5/20
to ICA-AtoM Users
Hi Patrick,
Glad to hear that rebuilding the nested set solved your issues!
Regarding AtoM, databases, and nested sets:
First, some of AtoM's choice to use a relational database comes from the project history - AtoM's 1.0 beta release was in 2008, when robust open source graph databases were in their infancy - perhaps even embryonic at that point. Given that our business model makes it challenging to find funding for large scale maintenance work (and fully switching from a relational database to a graph would be a large project), this is something we have not pursued, as it would certainly require community support for us to take on.
That said, we have certainly been investigating options such as this when considering AtoM's long-term future. It's important to note that there are many things that relational databases are still better at - and many large scale projects are still using a relational database as the primary data store, while serializing to a sidecar graph db to support graph-like queries and other things more suited to that data structure. It's not inconceivable that AtoM may take this approach in the future to be able to provide linked data support. I think we'd need to see a strong case for the overall advantages to consider using only a graph database.
It's also important to note that I'm an archivist, not a developer - so any errors in the above assertions are mine!
Finally, I'll add that we ARE moving away from relying on a nested set model wherever it is advantageous to do so in the upcoming 2.6 release. We have upgraded AtoM to MySQL v8, which now supports Common Table Expressions (CTE) as an alternative method. We have a lot of detailed analysis about the performance improvements this is providing AtoM with in the following ticket:
... and we also have several other open tickets (such as #13240 and #13241 for example) we hope to implement before the 2.6 release.
Cheers,
he / him
--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/295f4003-05fb-41ab-9692-11ffa9906fc1%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages