Question about "update matches in place" CSV import + documentation

[Creighton Barrett]

Hi there,

We're trying out the CSV import options in v2.4.1 and are trying to determine the best approach to updating some existing descriptions. We want to export some descriptions using the clipboard, edit the descriptions, change the parents of *some* but not all item-level descriptions, and then import the descriptions as updates. We want the parent descriptions to stay in the same location, but move some child descriptions so they appear under a new parent. In other words:

• File #1
• File #1, Item #1
• File #1, Item #2
• File #2
• File #2, Item #1
• File #2, Item #2

Becomes:

• File #1
• File #2
• File #2, Item #1
• File #2, Item #2
• File #2, Item #3 (was File #1, Item #1)
• File #2, Item #4 (was File #1, Item #2)

I've been looking at the docs for updating existing descriptions as CSV import: https://www.accesstomemory.org/en/docs/2.4/user-manual/import-export/csv-import/#update-existing-descriptions-via-csv-import

And I see that the docs reference an option to "skip unmatched records."

But when I access the CSV import page in our AtoM site, I see an option to "skip matched records." It looks like the feature has changed in v2.4.1 but I am not sure what to make of the various warnings and descriptions in the documentation. Should I assume that everything is 100% reversed? What happens now if you try to use the "Skip matched records" option during a "Delete and replace" import?

I think we want to use the "update matches ignoring blank fields in CSV" option described here: https://www.accesstomemory.org/en/docs/2.4/user-manual/import-export/csv-import/#update-matches-in-place

But when I tested a small job I found that the import created new records at the end of the finding aid and that the originals were not updated. So I'm trying to better understand how to prepare the CSV data and select the right options to achieve what we want to do.  

Also, is it still true that we cannot update eventDates, eventStartDates, and eventEndDates information via the update matches CSV import? We're using the RAD CSV template.   

Any thoughts or info on the documentation would be greatly appreciated!

Thank you,

Creighton

[Dan Gillean]

Hi Creighton, 

First, some responses to your questions: 

But when I access the CSV import page in our AtoM site, I see an option to "skip matched records." It looks like the feature has changed in v2.4.1 but I am not sure what to make of the various warnings and descriptions in the documentation. Should I assume that everything is 100% reversed? What happens now if you try to use the "Skip matched records" option during a "Delete and replace" import?

The options on the import page are context sensitive. It shows the "skip matched records" option when the field above, Update behaviors, is set to import new records. This gives users the option to have new record imports still perform the matching criteria check, but in reverse - if there are exact matches found, you can skip them from importing using this option, and the use the console output to identify the affected rows and determine if this was an accidental duplicate, or a nearly-identical record that you want imported and treated as new. 

If you change the Update behaviors drop-down to "Update matches" or "Delete matches" then the options below also change - now the option present should be "Skip unmatched records, with 2 limiters to help focus the matching (by repository or by top-level description). 

But when I tested a small job I found that the import created new records at the end of the finding aid and that the originals were not updated. So I'm trying to better understand how to prepare the CSV data and select the right options to achieve what we want to do.  

First, given the above question, I would first suggest you make sure you picked the right import option! ;)

However, here's where things tricky. 

The primary issue here is that the CSV update import functionality was not designed with roundtripping in a single system in mind, and because of this there are some facts that make this more difficult. I've previously tried to describe this at greater length in the following post: 

• https://groups.google.com/d/msg/ica-atom-users/OWFFysVAdnk/SLUJ663PAAAJ
  
• See also: https://groups.google.com/d/msg/ica-atom-users/dQUNEdZKYBM/Eyh2PZpKAgAJ

Essentially, the legacyID value in your export is in fact the object ID in AtoM's database - not the legacyID value that might be stored in the keymap table if those records were previously imported. If you created those records via the user interface, there won't be a legacyID value or source_name to match against in the keymap table anyway. Consequently, the import falls back to trying to match exactly on title, identifier, and repository name. If you've edited any of these fields in your CSV, then it's unlikely you are going to get a match. This is at least why the "Skip unmatched" can be helpful - though if and when you do finally attempt this in a production system, I still strongly recommend making backups before proceeding. 

As noted in the first thread, Steve has captured some ideas for making roundtrip updates in a single system work better, in ticket #12281. That one is in the Wishlist because the work involved is a bit more than Artefactual can take on as a bug fix or minor release enhancement, so we'll need to find a community sponsor to be able to implement the ideas captured there. 

I haven't tried this myself, but if you are handy with SQL, I suppose you could actually go into the keymap table for the related parent records and add the information object ID as the legacyID value, and the name of your CSV as the source_name value? That should lead to better matching. Could be a lot of work though if you're trying to update many different collections. 

Also, is it still true that we cannot update eventDates, eventStartDates, and eventEndDates information via the update matches CSV import? We're using the RAD CSV template.   

There may be some help here, in 2.4.1: 

• https://projects.artefactual.com/issues/12102

 See issue 1, case 3 for the desired behaviors: 

If an event row already exists with the same description, authority records, and the CSV import has different dates then overwrite the existing event dates

I didn't personally test this issue in great detail, but I'm hopeful this will give you what you need. 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To post to this group, send email to ica-ato...@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CAHueW_VWeC0vd3WiUt_Es0L8SUoV%3DHB_qa7-cTEq9e1xsX1fpg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Creighton Barrett]

Hi Dan,

Thanks so much for helping me to understand how this feature works! The job report shows that I selected the wrong update type (oops!) and I now see that the skip option changes based on the selected update type. Sorry for making you explain what now seems obvious!.

I also appreciate the background on CSV import and links to the tickets. We (obviously) haven't used the CSV import features much, so this is really helpful as we figure out the best approach to doing some new things. I see these topics come up on the list, but it is difficult to round everything up like you always manage to do. Thank you!

Yes, it looks like the fix for ticket #12012 will do the trick. But it's the roundtripping that we'll need to think about. I will review ticket #12281 and the other forum threads in more detail. I see the Update matches in place docs also mentions that RAD and DACS notes cannot be updated with this method. I'm guessing that is still accurate? And I assume that the proposed fix in ticket #12281 would update information objects and related events and notes? In other words, it would update *all* fields included in the CSV template?

Thank you,

Creighton

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CAC1FhZKqz%3DOotxXK4LTp1v5YOMqaS-J%2B088V7odQh-BgpVcWCg%40mail.gmail.com.

[Dan Gillean]

Hi Creighton, 

Some quick responses: 

 I see the Update matches in place docs also mentions that RAD and DACS notes cannot be updated with this method. I'm guessing that is still accurate? 

That's correct - as of now, this has not been changed. 

And I assume that the proposed fix in ticket #12281 would update information objects and related events and notes? In other words, it would update *all* fields included in the CSV template?  

I'm not sure if that is built into the proposal or not - Steve is also a bit swamped and working on a project with a tight deadline, so it might be a bit before I have a chance to snag him for questions. However, if it's not factored in now, and someone chooses to sponsor this enhancement and wants to prioritize the ability to update all fields, we can certainly factor that into the analysis when preparing an estimate. 

Cheers,  

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CAHueW_VCg2up%2Bo9-kdHWYWKFJgd7mpD6_HtoUg6G%3DiXi03k-FA%40mail.gmail.com.

[Creighton Barrett]

Awesome, thanks Dan! This is helpful, thanks for the quick response. 

Cheers,

Creighton

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CAC1FhZ%2BUZH-qRM%3DT7Aqp2OFi9jfF5Bq6hC9Ax6xN1FmX3nx_RA%40mail.gmail.com.