Skip to first unread message
Matt McKenzie
Aug 28, 2023, 3:56:47 PM8/28/23
to AtoM Users
I was testing out a new CSV import and it failed, but it got as far as creating an Archive Description that appears in the list of top level descriptions. Originally it produced a 500 error if you tried to edit the description. I delete all the information_object records (and slugs) I could find that were created from that import, BUT the descriptions still appear in the Browse Archive Descriptions page.
What am I missing? The Browse page must get the information from somewhere!
Matt
Dan Gillean
Aug 29, 2023, 9:01:53 AM8/29/23
to ica-ato...@googlegroups.com
Hi Matt,
When you say you deleted all information_object records, what do you mean - did you manually delete them from the information_object table in MySQL?
If so, there may be additional issues. A single description is actually spread across multiple tables in AtoM's database, starting with the object table (where all main entities have an entry, and where the unique object ID of each record in AtoM is assigned and then used as a key on all other tables), and extending across a number of others, including possibly information_object and information_object_i18n, note and note_i18n, property and property_i18n, keymap, slug, and more.
If you have made manual deletions only in the information_object table, then you may have left a number of partial records throughout your database. If you have a backup you can load, I'd strongly suggest that - it will probably save you a lot of headaches! Otherwise, if that is what you did but a backup is not an option... I will have to consult our team and see if they have further suggestions for you.
If that's' not what you mean, then it's possible that, given the error, the search index or the application cache was not properly updated, which would explain why you are seeing records that "aren't there" anymore. I'll suggest a few things you can try, most of which are command-line tasks. The tasks are listed below, with a bit of context for you. You will need access to the command-line for this - if you don't personally have this access, then pass this information on to your IT person or hosting provider, etc.
First, we actually have a command-line task in the latest version of AtoM that can help undo a bad import. See:
Next, I would suggest that you run a few of AtoM's common maintenance tasks, which typically resolve hiccups like these. We will even run a few extra tasks for good measure that won't hurt anything if there's no issue, but may resolve problems we haven't identified yet if they exist:
- Generate slugs - in case any of your deletions was interrupted and a description was left partially still in the database but without a slug, this will make it whole again so you can properly delete (if that's what you want)
- Build nested set - the nested set helps manage hierarchical relationships in AtoM's descriptions, and can sometimes break when an import goes wrong, leading to weird display issues. This will quickly rebuild it
- Clear Symfony's cache - this is the application cache, where copies of recently viewed records are stored so they can be loaded quickly without needing to fetch them from the database. Occasionally this holds out-of-date information, so flushing it ensures we are seeing the latest state of your AtoM installation
- Restart PHP-FPM - this is another type of cache that the application uses - restarting it also flushes the cache
- Rebuild search index - the search index is how records are displayed on all search and browse pages. Again, sometimes if a large operation goes wrong, then the search index can get out of sync with the database, so rebuilding it should fix that. Of all the tasks, this one may take the longest to run, and will mean some downtime for public users as all search/browse results disappear and then are added back in as the indexing progresses. If your site is a public-facing production site, then you may want to run this task outside of normal business hours, or at least at low-traffic times.
Finally, remember that your web browser also has its own cache, meaning it could be serving you a local copy of your AtoM site that is out of date! When you re-test after running these tasks, I suggest either clearing your web browser cache first (something you can easily look up online if you don't know how to do), or else test first in an incognito / private browser window, where the cache is typically disabled by default.
Let us know if this helps!
Cheers,
he / him
--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/f9f6e5f5-746f-41bc-87e4-d4670c0c4eb0n%40googlegroups.com.
Matt McKenzie
Aug 29, 2023, 12:48:41 PM8/29/23
to AtoM Users
Thank you for your reply. Yes. Indeed, deleting objects manually! As the entire server is an experiment, I naturally ignored every piece of advice about backup, although as you point out, much time would have been saved had I made one before messing about.
I may then have followed the resetting guidance out of sequence as nothing seemed to get rid of the orphaned Archive Descriptions. Anyway having now just re-run the populate index feature, all is well and the rogue descriptions have disappeared.
Of course, this still done raise the question of why a failed import should create uneditable and undeletable descriptions (using the User web interface) with the enigmatic server 500 error if one clicked on them.
Matt
Dan Gillean
Aug 29, 2023, 1:17:55 PM8/29/23
to ica-ato...@googlegroups.com
Hi Matt,
Glad to hear all is well for now - and that this is a server for experimentation, and not something to be used in production. I would strongly advise you to flush the database (you can use the tools:purge command to remove all entries) before you ever decide to start adding data you wish to keep - while things work now, knowing that there are partial (i.e. corrupted) entries in the db means that you could encounter other unexpected issues down the line, that might be harder to resolve.
In terms of why an import can lead to 500 errors.... it really depends and it's hard to say without knowing more about both your environment and the records you were attempting to import. However, in most cases it's often because 1) some process times (i.e. reaches one of the execution limits established in your PHP pool, or similar) or 2) the import process exhausts all available resources (like memory most often, or CPU, disk space, etc) such as when the CSV is too large or the system is under-resourced (or both), or 3) there are actual errors in the CSV itself. In all of these cases, you can end up with partially created rows in the database - i.e. information objects that have data in some of those tables I mentioned in my last post, but not others. This data corruption then returns a 500 error when another piece of code makes a request that can't be performed (like loading the view page of a record that's only partially imported). Of course, sometimes it's bugs in the application, deployment issues, CSV encoding issues, or any other number of causes - but it happens. Release 2.7 and later now include a CSV validation option that tries to catch data-related issues in advance - see:
As for why you can still see records when things are deleted from the database? That's not unexpected at all if you know a bit about how search indexes work, especially if you made manual deletions rather than using AtoM-supported processes to perform those deletions. I am no developer myself, but here's a quick summary of what I have learned from my colleagues over the years working with AtoM:
Essentially, it would be extremely inefficient for AtoM to access the database to return search and browse results - reading nearly every information object in the database just to prepare some stub results on demand. This would also make operations like faceting and filtering much harder. Instead, the Elasticsearch index basically creates a large JSON blob of all the data needed for search and browse, and Elasticsearch has its own built-in methods for rapidly handling filtering, faceting, and other boolean operations. As such, the search index is almost like a second flat data store, optimized for fast reads. The MySQL database remains the source of truth for the application however - which is why we can so easily run a reindex, where the previous JSON blob is thrown away, and a new one generated from the database. When that's not needed however, basically every search or browse page you see in AtoM is populated from the search index (i.e. no need to access the DB at all for that), while the individual view and edit pages of specific descriptions use PHP to interact with the database, fetching and returning records. Another way of flipping this around - you can delete your whole search index and see 0 records in search/browse - but if you know the URL to a specific record, you can still access it, since this uses the ORM and MySQL database and not the search index.
In any case, I'm glad that this was a test server, and that everything's working as expected for now! Thanks for updating us.
Cheers,
he / him
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/740c3a9a-ae48-43da-850e-24fb324dbb34n%40googlegroups.com.
Matt McKenzie
Aug 30, 2023, 6:39:03 AM8/30/23
to AtoM Users
Thank you.
Yes. I was attempting to replicate the CSV import function that Archives Hub used to have before it was recently put off line and mapping the field names to the slightly different field names required here. The problem was that my entry of "English" in the language field was rejected for a reason I have yet to discover. As it also rejected "eng", I then paused, having now created the two uneditable orphan Archive Descriptions... So tomorrow I may well look at the CSV validator and see what it thinks of my data.
I have had more success with my own software that imports an Archives Hub CSV formatted file and exports an EAD file from it, which then can be successfully loaded into AtoM... Having looked at this for a while now, it's interesting to see the way that AtoM manages to deal with the hierarchical nature of archive data and map it to a relational database engine.
Anyway, we move on and up.
Matt
Dan Gillean
Aug 30, 2023, 8:18:38 AM8/30/23
to ica-ato...@googlegroups.com
Hi Matt,
AtoM's underlying PHP framework, Symfony, uses an older internationalization (i18n) library, so for language and culture fields in general in the CSV, AtoM expects the 2-letter ISO 639-1 language codes, like "en". We have a list of supported languages here:
In addition to the CSV Validation documentation link I sent, we also have fairly extensive documentation on how the CSV import works in AtoM - see:
You might also want to take a look at the sample data in the CSV templates we provide - we populate 2 rows (one example fonds, and a child item) with the names of the relevant content standard fields, so that you can use them as crosswalks to your chosen standard. You can import these examples to see exactly how they render in AtoM if you'd like. The CSV templates are kept in AtoM itself (at lib/task/import/example), or on our wiki, here:
Finally, given what you're trying to do, the general advice contained in our slide deck about data migration may also be helpful, as it includes some things to consider when converting to or from another source:
Hope these resources help!
he / him
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/dd3d9f84-26e5-4bd0-a087-df2a1b1c758an%40googlegroups.com.
Mohamed vall
Aug 30, 2023, 12:50:43 PM8/30/23
to ica-ato...@googlegroups.com
Hello everyone
Why I have this problem I can’t install in my Ubuntu the vagrant ?
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CAC1FhZ%2BySiwY7tTYwDRV_xTeVoRc8JdA6-1SoRPOHXGz4sWGLg%40mail.gmail.com.
IT
Dan Gillean
Aug 30, 2023, 2:12:37 PM8/30/23
to ica-ato...@googlegroups.com
Hi Mohamed,
I've never seen this issue before, but searching for it online, I have found this thread:
It's possible you are getting this error due to other things installed in your environment? Try running the command as suggested in that link to see if it helps:
- vagrant up --provider virtualbox
Hope it helps - if not, I recommend searching the error online as I think this has more to do with your environment than something specific about the AtoM Vagrant box that we can help you with. Good luck!
Cheers,
he / him
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CAA0%3D6uckyLPe_e2UtzFPb%2BEPHZAY1hL8S4Rjpdt6%3D1cn8LnqpQ%40mail.gmail.com.
Mohamed vall
Sep 7, 2023, 2:50:13 PM9/7/23
to ica-ato...@googlegroups.com
Hi everyone
How can I find or use Dublin Core in my Atom vagrant ?
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CAC1FhZJv%2B7nqRV0xC2G%3Dcq4P_dt1R%3D17vT%2BZr%2BPT3eFmWtko0w%40mail.gmail.com.
IT
محمود محمد
Sep 7, 2023, 3:05:34 PM9/7/23
to ica-ato...@googlegroups.com
Step 1 go to setting in admin
Step 2 go to default template
Step 3 choice the standard that you want to use it [ isad g2 - Dublin core ]
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CAA0%3D6ue8b1-g6Adv590Vj4Rj0E-ZVZBMDhaanyC1LQj8UHqKDA%40mail.gmail.com.
Dan Gillean
Sep 7, 2023, 3:39:42 PM9/7/23
to ica-ato...@googlegroups.com
Hi Mohamed,
Mahmoud is correct! You can find the way to change the default template in our documentation, here:
Note that you can also change an individual description (or the current description and its descendants) from the edit page of an archival description, without changing the global default. So for example, if you want to continue using ISAD(G) as your default, but you have one collection that you want to display using Dublin Core. See for example:
Finally it's worth mentioning as well that we have done what we can to crosswalk all the standards-based templates behind the scene. Meaning: if you have an ISAD(G) description, then you can flip it to Dublin Core, and wherever there is an equivalent, it will display in the new template. For example, Scope and Content in ISAD(G) will display as Description in Dublin Core when flipped.
You can even preview your data in another template without having to enter edit mode if you want - see slide 9 and 10 in this slide deck for more information:
Cheers,
he / him
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CAB5aXsmzwNCeN6RhRxkdBbiO4rjnDAhiYs-3jPqqwHcU0b0bLw%40mail.gmail.com.
Mohamed vall
Sep 8, 2023, 6:20:51 PM9/8/23
to ica-ato...@googlegroups.com
Hello dears ,
Why every time I install composer I can’t do this ?
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CAC1FhZ%2B8XWFdL35Y3%2B1P1Y1F-j_o_wfkd87Covqo85qFvq_QWw%40mail.gmail.com.
IT
Dan Gillean
Sep 11, 2023, 12:10:10 PM9/11/23
to ica-ato...@googlegroups.com
Hi Mohamed,
Did you install AtoM following Option 1, using the tarball download from our website?
If yes, then you are running an unnecessary step. The steps described under the Option 2 heading are an alternative method of installation. They are not required if you have followed Option 1. Specifically, Composer is not used at all with Option 1 because we include the dependencies that Composer installs directly in the downloadable tarball.
If you did follow Option 1, then you can skip all those steps under Option 2, and jump down to the next main section, Create the Database.
Cheers,
he / him
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CAA0%3D6ucPrV%3Dd88Zu%3DXMVBMjqzrYQypkqbrkDVhEZzOFioVH92Q%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages