CSV-Import / levelofdescription / UTF-8

95 views
Skip to first unread message

Rogério de Souza Farias

unread,
Mar 9, 2021, 8:00:39 AM3/9/21
to AtoM Users
Hi there, sorry for bothering you. I am a complete newbie and I am facing several problems with the CSV-Import mechanism.

Problem 1: The first is the CSV encoding. I use Excel to edit CSV files (archival descriptions) and when I open it, it comes in weird characters. It seems atom uses UTF-8, but my computer does not support it. This will create the problem I am about to present you, but for now what routine do you suggest me to edit my CSV files to upload to ATOM using an Apple computer? 
Screen Shot 2021-03-09 at 9.47.24 AM.png

Problem 2: I use as template a csv file I exported from the system. The odd thing is that the column levelOfDescription is never filled and nowhere if the CSV file there is reference to fonds, series, etc.
I do not know if this is the source of problem number 2, but after I edit my file and import the CSV ("delete matches and replace with imported records") it creates a parallel branch. 

Problem 3: After several trials, I uploaded a CSV file with the wrong encoding and my plataform created a "phantom draft". I can see it, but I cannot delete it. When I conduct a global search and click on it, I get a 500 Internal Server Error.

Please, help me!


The levelOfDescription 
Screen Shot 2021-03-09 at 9.47.24 AM.png

Dan Gillean

unread,
Mar 9, 2021, 10:59:03 AM3/9/21
to ICA-AtoM Users
Hi Rogério, 

I will try to offer some tips and suggestions below - however, in a couple cases, it may be that you are encountering some known bugs in AtoM! We have fixes for some of these issues on the way in the upcoming 2.6.3 release, but not all. You may need to experiment a bit to find workarounds in the meantime. 

Problem 1

First, we strongly recommend that users make a database backup before performing large imports, especially if attempting to perform update or delete/replace imports! That way, if something goes wrong, you can easily load your backup and recover the previous state without having to do all of this troubleshooting. If you did make a backup first, the easiest way to resolve this will likely be to load it now! See: 
In terms of using spreadsheet applications: we also strongly recommend that users do NOT use Excel if possible! Also, your computer definitely supports UTF-8 display, as this is the de facto standard for encoding on the web and in most email clients - so if you're reading this message, it's probably encoded in UTF-8! The problem has to do with Microsoft's approach to standards - or rather, their strange refusal to use existing standards, instead implementing their own custom encodings and line ending characters. For example, by default Excel uses WinLatin, a custom character encoding that AtoM doesn't recognize - and Excel makes it very hard to change its defaults without any problems! 

The character encoding and line-ending requirements for AtoM are noted in our documentation here: 
Instead of Excel, I recommend that you try to install LibreOffice Calc and use that instead when preparing CSVs for import into AtoM. It's not as user-friendly or as nice looking as Excel, but it is free open source software, and the control it gives you over character encoding (and delimiter / separator characters) is far greater - you can set this before you open any CSV: 

calc-csv-options.png

Problem 2

What is the default installation culture of your site? (i.e. the language code used during installation - also available in the apps/qubit/config/settings.yml file. Also, what language were the descriptions created in? You can always check the original source language used when creating a description by entering edit mode, and looking in the Administration area to see the Source language:

administration-area.png
If levels of description are not exporting, I suspect that you might be encountering a bug with exports of translations and/or non-english records. See this bug ticket for more information: 
We hope to be able to fix these issues in a future release. In the meantime, if your descriptions are not in English, you could try going to the Levels of description taxonomy and adding translations for the Level of description terms if they don't already exist. Simply open the view page of a term (e.g. Fonds), flip the user interface to your desired language using the Language menu in the header bar, and then enter edit mode. If there is not already a translation, then the original English term will appear in a yellow box above the name field - you can add your translation directly to the edit field and save. See: 
I'm not sure if this will help or not, but it's worth a try to see. 

You can also try running a bulk CSV export from the command-line. Because this task exports everything, it doesn't have the same language issues - though if you just want a particular fonds, then you'd have to delete other rows out of the CSV after. See: 
As for using the delete and replace option: 

Both update import options in AtoM currently have some known issues and limitations with how they were originally implemented. Until they are improved, I strongly recommend that you read this section of the documentation on each option closely, to learn more about the known limitations: 
Additionally, if you are using one of the 2.6 releases (e.g. 2.6.0, 2.6.1, 2.6.2), then you may be encountering a known bug with the update import matching logic. We have a fix prepared for the upcoming 2.6.3 release, but in the meantime, you can find more information on this issue here: 
Finally, it may also be that the encoding issues you described in problem 1 are affecting things too, so hopefully resolving that will help!

Problem 3

Next, let's try to resolve the error with the phantom draft. I would recommend running some basic maintenance tasks first, to see if these resolve the issue. Try rebuilding the nested set, generating slugs, clearing the application cache, restarting PHP-FPM, and repopulating the search index. 

All of the following commands should be run from the root AtoM installation directory - if you have followed our recommended installation instructions, then this is typically: 
  • /usr/share/nginx/atom
If the commands won't run and you've confirmed you are in the right directory, try running them as the www-data user - this is the user to whom we assign all filesystem rights when installing and configuring AtoM. To do so, simply precede the commands below with sudo -u www-data. 

Rebuild the nested set: 
Generate slugs: 
Clear the application cache: 
Restart PHP-FPM
If you have memcached installed, it is good to clear that cache as well. This command should not have any ill effects if you don't have memcached installed, so you can always try it out: 
Repopulate the search index
  • php symfony search:populate
  • See: https://www.accesstomemory.org/docs/latest/admin-manual/maintenance/populate-search-index/
  • Note: If you have a lot of records, this task can take a long time to run, and will make records temporarily unavailable in the user interface as they are reindexed. Best to run this after business hours. However, you can also use some of the task options to reduce the time it takes by limiting how it runs - for example, using the --update option will not make existing records unavailable as the task progresses (it will progressively replace the index rather than deleting it first). The --exclude-types option can be used to target just certain entity types, rather than all of them, and the --slug option can be used just to target specific records, taking the slug as input. See the documentation for further details and options. 
If the issue is still not resolved, then the best thing to do next is to check the webserver error logs for more information about the error. If you have followed our recommended installation instructions, you can check the most recent error log entries with: 
Feel free to share any related error message you find here if you're not sure how to proceed, and we will try to offer suggestions on next steps. 

Hope this helps! 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056
@accesstomemory
he / him


--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/c89aad1e-c8c1-4db0-b022-ac01ed79ff88n%40googlegroups.com.

Rogério de Souza Farias

unread,
Mar 10, 2021, 3:22:42 PM3/10/21
to AtoM Users
Hi Dan. Thank you very much. I manage to solve the all problems. 

Our repository is in Portuguese. So, in the second problem, I discovered that the levelOfDescription field was exported only after I translated all fields into English. 

Thank you very much!

Reply all
Reply to author
Forward
0 new messages