CSV imports with new ID

[alberto capellán]

Hi,i'm working with atom 2.2

I'm using .csv files to import new data to my database with the user interface.

My main trouble is the fact that each time I use the same file with new registers the oldest get a duplicate in the database.

I'm not sure if it is intended or is just a problem with templates.  This is an example with my first two register.

idcolegial  qubitParentSlug  nameAccessPoints generalNote alternativeIdentifier2  creators  placeAccessPoints  creationDates  identifier title 637 libros-de-informacion-de-limpieza-de-sangre-3 AMORES, JUAN DE Número de viajes: 1. -- . --Fecha de licenciamiento: 1695 Colegio de San Telmo Sevilla,Sevilla 1688-1695 Libro 217, fols. 420-424 Información sobre JUAN DE AMORES 800 libros-de-informacion-de-limpieza-de-sangre-3 VERA, JOSE DE Número de viajes: 0. -- Bautizado en Omnium Sanctorum. --Edad: 12. --Últ Colegio de San Telmo Sevilla, Sevilla 1688-1721 Libro 217, fols. 298-306 Información sobre JOSE DE VERA

[Dan Gillean]

Hi there Alberto,

I'm not sure I'm fully understanding your use case, but perhaps I can explain some of how the CSV import fields work and it might clarify the situation. Then you can let me know if I have misunderstood, or if I have helped to answer you questions in any way.

Some questions:

Looking at this sample you have pasted, it seems you are trying to import 2 records as children of an existing record in AtoM (with a slug of libros-de-informacion-de-limpieza-de-sangre-3). Is this correct?

I'm guessing that the idcolegial column is the legacyID column - is this correct?

Do you mean that the parent description (e.g. libros-de-informacion-de-limpieza-de-sangre-3) is somehow duplicated when you import? Or that with each import, the other records you import are duplicated, but you only want to update them?

Anything further you can tell us will help. Thanks!

Now some clarifications:

First of all, the CSV import is not intended to be used to update existing records at this time - by default it will always create new records on import. In the next version of AtoM, there will be a more robust update option, that will allow you to update existing descriptions. Some of the basics are included for the command line in 2.3, but they are not documented at this point, as we are rewriting them and enhancing them for use via the user interface import in the 2.4 release. If you are using 2.3 and want to experiment however, type php symfony help csv:import into the CLI from AtoM's root directory to see the full list of available options.

One thing to uderstand: the legacyID e.g. what I think in your sample is the idcolegial) is different than the internal information object ID assigned to each description. While the information object ID is an internal value assigned by the database (and never really shown to the user via the user interface), the legacyID, while *also* not shown in the interface, is not stored directly in the information object table in AtoM's MySQL database. Instead, it is a special number used during import for managing hierarchies. It is saved as part of the keymap table in AtoM's database as a property. See:

• https://www.accesstomemory.org/docs/2.3/user-manual/import-export/csv-import/#legacy-id-mapping-dealing-with-hierarchical-data-in-a-csv

So, the legacyID is saved as the source_id, and if you use the --source option when importing from the command-line, the --source value is added as the source_name to the database.

This means that the legacyID is not a unique value in the system - you could use 1, 2, 3, 4 etc to number them for the first import, and use the same values again for the next import if you wish. In 2.2 and earlier, legacyID values are mainly used for hierarchical relationships - e.g. if you want to import a fonds with children, then the children will have the fonds legacyID value in the parentID column.

In any case, if you can explain what you are trying to do more, I hope we can assist you further.

Regards,

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-users+unsubscribe@googlegroups.com.
To post to this group, send email to ica-atom-users@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/8a467257-37d9-415f-8d70-15300fe200c8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[alberto capellán]

Hello Dan

Finally i could modify the .CSV so i can check what you said. "Idcolegial" is a register just for our own service so it's unnecesary, also I inquired that some registers has " , "  and provoke several mistakes with CSV import.

But with all this changes i still have a problem, the tree seems bugged ( image ) and shows the same register multiple times.

Thanks for your time.

[Miguel Alfaro]

Hola Alberto:

Por lo que comentas, que se dupliquen los registros creo que se debe a que cada vez que subes el CSV, no eliminas los que ya están subidos a Atom y por eso se duplican. Por otro lado, los registros que tienen comillas te aconsejo que lo entrecomilles todo, es decir: Libro de "juntas" de Sevilla en el csv pon: "Libro de "juntas" de Sevilla". Si el idcolegial es un identificador propio vuestro, puedes intentar codificarlo en la plantilla como alternativeIdentifiers; en tu caso pienso que el idcolegial es el numero de colegial de tal persona, en tu ejemplo Juan de Amores sería el colegial 637, esta información debería consignarse en el registro de autoridad.

Saludos

--

You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-users+unsubscribe@googlegroups.com.
To post to this group, send email to ica-atom-users@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/96b03de3-d055-47c8-b90d-c36b12f4f2ab%40googlegroups.com.

[Dan Gillean]

Hi Alberto,

When records are duplicated in the treeview (but there are not actual duplicate records in the system from your imports), this often indicates a problem with the nested set. AtoM uses a nested set model to manage hierarchical data stored in the MySQL relational database - for example, archival descriptions (a fonds with child records).

Sometimes, especially when a long-running operation terminates improperly (such as an import), the nested set can become corrupted. There is a command-line task that will allow you to rebuild the nested set - from AtoM's root directory, try:

• php symfony propel:build-nested-set

See: https://www.accesstomemory.org/docs/2.3/admin-manual/maintenance/cli-tools/#rebuild-the-nested-set

You might also want to clear the application cache and restart your services - if you have followed our recommended installation instructions, you can use the following:

• sudo service nginx restart
• sudo service php5-fpm restart
• php symfony cc

One other possibility: check the sort settings on the treeview. There are known issues (see for example issue #6574) with the Sort by title, and Sort by title - identifier options, if you are missing even 1 title or identifier. We recommend using the Manual sort option. See:

• https://www.accesstomemory.org/docs/2.3/user-manual/administer/settings/#sort-treeview-information-object

Let us know if that helps!

Regards,

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

--

You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-users+unsubscribe@googlegroups.com.
To post to this group, send email to ica-atom-users@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/96b03de3-d055-47c8-b90d-c36b12f4f2ab%40googlegroups.com.

[alberto capellán]

Hi Dan.

Everything is right now, as you said the lack of sequence in the .csv provokes the bug  (issue #6574) , so we changed it and now de treeview is perfect.
Also we tested with interface import with 100 register and with CLI import with 2000 and the bug didnt reproduce.

Really grateful for the community support.