Importing records

[Mark Smith]

Here in the Shetland Archives we have decided to replace our CALM catalogue with AtoM.  We have been testing version 2.3 for a few months and are going to install 2.4 for production.  For the last month or so I have been producing CSV files of the various collections we hold and importing them into our test version.  The process of formatting the files is kind of slow going but the imports are working well so far (the modifications to imports via the job scheduler in 2.4 will be very useful indeed).

The question I have, though, is about the ISADG import template I've been using.  I downloaded the template from the 2.3 documentation, but does anyone know if the 2.4 template differs in any way?  If I didn't have to move the data from the 2.3 template I've been working with, that would make me pretty happy.

I've also played around a little with a tool called Open Refine that Dan from Artefactual mentioned on another thread.  I've not really got to grips with it yet.  Does anybody have any experience of that tool?

[Dan Gillean]

Hi Mark, 

You'll be happy to learn that there are no changes in the ISAD(G) CSV import template between 2.3 and 2.4. The only changes in our templates between 2.3 and 2.4 are the following: 

Authority records

• Add 2 new columns for Actor occupation data - actorOccupations and actorOccupationNotes. These columns both accept piped multiple values, and correlate with one another. 
• Related documentation on Actor occupations: https://www.accesstomemory.org/docs/2.4/user-manual/add-edit-content/authority-records/#add-access-points-to-your-authority-record

Repository records

• Add new CSV template for import/export of repository data

I have yet to add these to the wiki - my apologies. Part of the delay is that we've discovered that a number of fields are not roundtripping in the new repository CSV import/export. We're currently working on a fix for this that will be included in the next version, and backported to our stable/2.4.x branch when completed. We won't be creating a new 2.4 tarball, but anyone upgrading or installing AtoM for the first time can follow option 2 in our installation instructions (install from our git repository), and can always later easily pull in any other patches and fixes we merge back into the stable branch without a full upgrade.  

In the meantime, users can always find our CSV templates included in the AtoM code (or via our GitHub page), at lib/task/import/example. You can also use the clipboard to export a repository record to see a sample of the repository CSV as it currently is, and/or add occupation access points to an authority record and export it, or manually add the 2 new columns to your template for import. 

I'll try to get the new  samples on our wiki soon. 

Regarding OpenRefine, it is a GREAT tool for normalizing your data prior to a migration, and we use it internally at Artefactual whenever we can. The facets alone are a powerful way to look for names and terms with variant spellings etc that you can easily resolve and merge with a couple clicks. You can split data into different columns or concatenate data into one column, and much much more. 

There are many great free resources out there worth checking out! Start on the OpenRefine homepage - you'll find a whole free book as well as some videos linked there: 

• http://openrefine.org/
  

The OpenRefine GitHub wiki also has a huge list of resources, including a curated list of external resources such as tutorials and the like, here: 

• https://github.com/OpenRefine/OpenRefine/wiki
  
• https://github.com/OpenRefine/OpenRefine/wiki/External-Resources
  

There's also a public user forum, here: 

• https://groups.google.com/forum/#!forum/openrefine

If you do a web search you will find even more resources. Here's a simple introductory tutorial I found online, from a Canadian archivist (though it doesn't use archival examples at all): 

• https://zapier.com/blog/openrefine-guide/
  

If you are looking to perform a specific action (like concatenating 2 columns together), you can generally just do a web search for what you want, and usually find a pre-crafted example of how to construct the query. Good luck! 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-users+unsubscribe@googlegroups.com.
To post to this group, send email to ica-atom-users@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/57fadcf5-7b1c-4122-9692-905c7854f451%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mark Smith]

Thanks Dan.  That makes it very clear.  And the resources on Open Refine look really useful.  So far, the collections I've been working with are only a few hundred records, so cleaning up the data manually isn't that difficult.  I reckon Open Refine will be just the thing when I come to our larger collections.

Thanks again,

Mark

To post to this group, send email to ica-ato...@googlegroups.com.

Visit this group at https://groups.google.com/group/ica-atom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/57fadcf5-7b1c-4122-9692-905c7854f451%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "AtoM Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ica-atom-users/5ZEy8php0tc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ica-atom-users+unsubscribe@googlegroups.com.

To post to this group, send email to ica-atom-users@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CAC1FhZJXkSHTcQmhpOuA%2BE5PF%2B2S3KvhCxiOcwkoJZ5vo1joLQ%40mail.gmail.com.