Generating of ID from archival inventory & individual file numbers

[Linda Skābene]

Greetings from an employee of the Latvian State Historical Archive! 

I've recently found out about AtoM and I think it could greatly supplement the work of my department, however I have one (or possibly several) technical questions.

USE CASE

We have our own system akin to AtoM that is built around the ISDF. However, said sistem is Old and Frail, to say the least, so we try to get our series, sub-series and files all done and sorted before we import anything into it. This is a viable strategy when the given fonds is either on the smaller size or the variation of files in it is not very large. But, when faced with a fonds that contains 150 000+ files, this strategy simply won't do.

So, I was hoping that a private install of AtoM would be robust enough that I could use it to import all my CSV's and sort out this giant fond "in private", without touching our rickety old system or attempting to use Excel or anything else. I've been reading up on AtoM's documentation and I've got a grasp of where my data would go in the CSV, except one variable.

THE ISSUE

Now, I have no idea how this is done in other archives outside of my general area, but our archive system is built upon the system used in the Soviet Union, where fonds, when inventorized by hand, were split across multiple archival inventory lists which cannot be considered modern series (this is where my department ocmes in to sort them according to modern standards).

The issue arrises with the ID fields. I think it is best explained through an example.

Say, we have a fonds - "F6999 Vidzeme, Kurzeme and Estonia Manor Documents".
The total amount of files belonging to this fonds is 10995. They are split across 43 inventory lists, and each list starts with 1. So you can have two different items, with their full ID's being 6999_4_1 and 6999_5_1.

Now, our old system handles this by asking you which inventory list you are importing for and then it reads the CSV and attaches the inventory list number to the file numbers automatically. 

Hopefully that all made sense.

How should I fill in AtoM's CSV in this case?

Thank you in advance,
L. Skābene

[Alberto Pereira]

Hi,

this is interesting! I think it depends if you want to keep the original intermediary levels (the "inventory list") and original identifiers. 

In AtoM, there is an important distinction that could help in this case. Identifiers are the reference codes of each description contained in their own levels (so 1,2,3,4 can be repeated in different levels), and Full reference codes, which are the archival references of descriptions. So, for instance, you can have a Fond A, with two child collections C and D, and each with files identified by 1,2,3 etc. Atom generates the full reference codes by joining the identifiers of all levels with a specific separator (- by default). So you can have description A/C/1 and A/D/1, etc.

In your case, you can keep the Inventory list as an intermediary level.

The csv could look something like this:

legacyId     | parentId  | identifier | title                                                                            | levelOfDescription
6999          |                 | F6999    | Vidzeme, Kurzeme and Estonia Manor Documents | Fonds
6999_4      | 6999        | 4            | Inventory list 4                                                           | Inventory list
6999_4_1  | 6999_4    | 1            | (file title)                                                                     | File
6999_5      | 6999        | 5            | Inventory list 5                                                           | Inventory list

6999_5_1  | 6999_5    | 1            | (file title)                                                                     | File

Which would create, for instance, a description with the full refrence code of F6999-4-1 and another with F6999-5-1   

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/ica-atom-users/6bb4563d-d70c-4d52-9c75-f98ba053cbf3n%40googlegroups.com.

[Johan Pieterse]

Hi  

Alberto's approach is right, and it's the standard way to handle this in AtoM. The distinction that solves it is between identifier and full reference code. An identifier only has to be unique among its siblings, so 1 can repeat freely across different inventory lists. AtoM builds the full reference code by walking up the hierarchy and joining the identifiers, so 6999_4_1 and 6999_5_1 become F6999-4-1 and F6999-5-1 and stop colliding. Keeping each inventory list as an intermediate level is exactly right, and it preserves the original arrangement rather than flattening it — which matters if anyone later needs to trace a file back to the handwritten list it came from.

Two practical points before you import 150,000 rows.
Your levelOfDescription values must already exist in the Levels of Description taxonomy. "Inventory list" isn't one of AtoM's defaults, so either add the term first through Manage > Taxonomies, or map the lists onto Series. Rows naming a level that doesn't exist will fail, and finding that out at row 90,000 is painful.

At your scale, import with indexing switched off and build the search index afterwards:
 php symfony csv:import --index=false --source-name="F6999" /path/to/file.csv
 php symfony search:populate

Importing with indexing on is the usual reason large AtoM imports crawl or fall over. Import the fonds row first, then the 43 inventory lists, then the files, so every parentId already exists when it's referenced. Splitting the files into one CSV per inventory list is worth doing, it keeps each run manageable and means a failure costs you one list rather than the whole fonds.

A private working instance for exactly this kind of sorting is a sound use of AtoM, and 150,000 descriptions is well within what it handles.

Dr. Johan Pieterse

[Alberto Pereira]

Hi Johan,

Thanks for the thorough explanation. Just 2 small additional pieces of information:

- I'm not an archivist, so there may be a more formal way of addressing this context;

- Regarding

'Your levelOfDescription values must already exist in the Levels of Description taxonomy. "Inventory list" isn't one of AtoM's defaults, so either add the term first through Manage > Taxonomies, or map the lists onto Series. Rows naming a level that doesn't exist will fail, and finding that out at row 90,000 is painful."

I don't think the term needs to exist. By now, I must've created dozens of levels of description that were malformed (typos, etc) in the original csv import file. I'll make a few tests and check the code anyway, because I may be mistaken.

To view this discussion visit https://groups.google.com/d/msgid/ica-atom-users/4564ba54-80f2-4ef9-a921-3fd2047b0550n%40googlegroups.com.

[pieters...@gmail.com]

Hi

 

I know it auto creates them. But always feel it is saver to first create it.

 

But yes you are right

 

 

Groete / Regards

Johan Pieterse

082 337-1406

To view this discussion visit https://groups.google.com/d/msgid/ica-atom-users/CAM--OVu%2B9%3DiiqfU07%3DTpg9ikhJO%3Di7oREGAzgA17%3DWXcOykz1Q%40mail.gmail.com.

[Linda Skābene]

Hello and thank you both,

I think I've understood, but just to double-check the order of actions. First, I should make the levelofdescription (or not? I'll let you talk it out :) ) for my inventory lists and then import my files, so that they get their right reference codes, and after that I can start building my series and subseries and sorting things into their correct place?

[Johan Pieterse]

Hi

It auto creates. I like to first add them to the system to be sure.

Your choice. 

Johan Pieterse
082 337-1406

To view this discussion visit https://groups.google.com/d/msgid/ica-atom-users/ab921776-333d-406e-89cb-a523fba2e432n%40googlegroups.com.