Update AtoM 2.6 to 2.6 error with elasticsearch

[maxi...@gmail.com]

Hello There! How are you? I need some help please. I am updating atom and I have already completed all the steps in the documentation. However, when trying to search for a description in the search engine it gives me the following error.

On the other hand, if I click on any description that appears as a suggestion in the main menu, it gives me the following error.

What could be happening? Could I have made a mistake in some step?
I am thankful for any kind of help. Greetings

[Dan Gillean]

Hi there, 

I have recently assembled a general troubleshooting guide for all the most common issues encountered in this thread: 

• https://groups.google.com/g/ica-atom-users/c/wVrhL-TAaDY/m/Isu3jUY-AAAJ
  

Please read the first part, and provide answers to the general questions about your installation. Additionally, use the instructions there to check the webserver error logs, and please share any error message you find. 

There is also a section on Elasticsearch issues, which links to another forum thread with a lot more troubleshooting suggestions: 

• https://groups.google.com/g/ica-atom-users/c/tzvQRtkGMto/m/ezUvRyQHAQAJ
  

Using the information in that thread, please provide the output of the search:status task for us. You can also follow the rest of the recommendations in that thread (i.e. run the common maintenance tasks, try repopulating the search index, check the ES logs if that doesn't resolve the issue and share what you find, try stopping and restarting the ES service, etc), and let us know how it goes. 

Hopefully these suggestions will resolve the issue - and if not, then the additional information you provide should help us to suggest some good next steps for troubleshooting. 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/1a683270-09a7-41f6-8adb-6f4366564c69n%40googlegroups.com.

[maxi...@gmail.com]

Well, regarding the error 500 internal server error, I followed the instructions you provided me, when wanting to view a document unit the log captured the following error:

2024/02/15 11:34:46 [error] 14507#14507: *21358 FastCGI sent in stderr: "PHP message: Unknown record property "digitalObjectLink" on "QubitInformationObject"" while reading response header from upstream, client: 10.6. 242.224, server: atom.mininterior.gob.ar, request: "GET /index.php/familia-roca-3 HTTP/1.1", upstream: "fastcgi://unix:/run/php/php7.4-fpm .atom.sock:", host: "atom.mininterior.gob.ar", referrer: "https://atom.mininterior.gob.ar/index.php/informationobject/browse"

2024/02/15 11:34:47 [error] 14507#14507: *21360 FastCGI sent in stderr: "PHP message: Unknown record property "digitalObjectLink" on "QubitInformationObject"" while reading response header from upstream, client: 10.6. 242.225, server: atom.mininterior.gob.ar, request: "GET /index.php/familia-roca-3 HTTP/1.1", upstream: "fastcgi://unix:/run/php/php7.4-fpm .atom.sock:", host: "atom.mininterior.gob.ar"

On the other hand to answer the questions you ask:

• What is the full AtoM version number or your installation, as found in Admin > Settings? I installed stable version 2.8 but from the AtoM interface it displays 2.6 and I don't understand that.
• Did you follow exactly the recommended installation instructions for your version? If not, what changes have you made? I followed the recommended installation instructions to the letter.
• Are you using a custom theme plugin, and/or does your installation have any other local code customizations?
• If not, are you using the older default Bootstrap 2 Dominion theme, or the new Bootstrap 5 Dominion theme included in 2.7 and later? I am using a Bootstrap 2 theme.
• Does your AtoM installation meet the recommended minimum hardware requirements listed here? How much memory does your installation have? The server where I work has the minimum hardware requirements to be able to install AtoM. It has around 1TB of memory.
• Are there any important previous actions that might have caused this worth mentioning? For example, did you upgrade recently? Did someone attempt an import? When and why, to the best of your knowledge, did this error start happening? What was the triggering event before the error message? Don't do anything strange, or update, or anything.
• What, if anything, have you tried so far to investigate and/or resolve the issue? And what happened? I have been trying to resolve the error for 2 days, I read some group conversations, followed steps in the official documentation and I still can't solve it. I also followed all the steps at https://www.accesstomemory.org/en/docs/2.8/admin-manual/maintenance/cli-tools/#cli-regenerate-derivatives

About the other error, run the command "sudo php symfony search:status" and the output was:



Also, I previously ran the commands "php symfony propel:generate-slugs", "php symfony propel:build-nested-set" and "sudo systemctl restart php7.4-fpm" and still there were no changes

[Dan Gillean]

Hi again, 

A couple follow-up questions based on some of your responses: 

• Did you follow exactly the recommended installation instructions for your version? If not, what changes have you made? I followed the recommended installation instructions to the letter.

You say you followed the installation instructions to the letter, but your command-line prompt in the image you supplied for the search-status command makes it look like you are using Debian, rather than the recommended Ubuntu LTS distribution described in the docs. Are you in fact on Debian? This alone can potentially cause many issues as the available libraries and extensions supported in your distro may be different, or may be different versions than those used by default in the recommended Ubuntu LTS release. 

Are there any other changes we should know about?

• What is the full AtoM version number or your installation, as found in Admin > Settings? I installed stable version 2.8 but from the AtoM interface it displays 2.6 and I don't understand that.

I apologize in advance but I want to be very specific, because usually when we see such issues, it is because a step was skipped or the instructions not understood. 

First of all the Upgrading documents say essentially that to upgrade between major versions (e.g. 2.6 to 2.8), you need to: 

1. install a brand new AtoM intstance alongside the old one
2. Copy your data (database dump; uploads and downloads directory contents) from the old version
3. Drop and recreate the database in your new 2.8 installation
4. Load the 2.6 data into your new 2.8 installation
5. Run the upgrade task
6. Restart services and re-populate the search index

Because of changes in dependencies, database schema changes, and more, it is NOT possible to upgrade in place (i.e. over top of an existing installation) for upgrades between major versions. 

So, some questions based on this: 

• Our 2.8 installation docs outline 2 different methods of Installation - Option 1 is to download and install the tarballs we provide, and Option 2 is to install from our GitHub code repository. Which method did you use for the new 2.8 installation?
• Did you create a separate, brand new installation alongside your old 2.6 version when trying to ugprade? Or did you try to "upgrade in place" and overwrite the 2.6 version with 2.8?
• When performing the upgrade - i.e. loading your 2.6 data into the new site, did you definitely remember to drop and recreate the database FIRST, as outlined in step 4 of the upgrade docs in this section?
• After loading your sqldump from 2.6, did you definitely remember to run the upgrade task, as described here?

In the original troubleshooting post I sent you, there is also a whole section on common issues after upgrading that further outlines these issues. You might want to review that again: 

• https://groups.google.com/g/ica-atom-users/c/wVrhL-TAaDY/m/Isu3jUY-AAAJ
  

I am hoping that if you review all of these steps carefully, we may still find that one step of the process was overlooked, and completing it may resolve the issue. It seems like you have more than enough memory, and per the output of the search status task, you are running the correct Elasticsearch version and all your content appears to be properly indexed. 

The error message you found in the logs suggests to me that the database schema is not fully up-to-date for what is expected in 2.8, so hopefully reviewing the steps above, and perhaps trying to run the ugprade task again, will lead us to an answer. 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/1cee6bf6-954e-49b4-bf23-9049ec1c27c4n%40googlegroups.com.

[maxi...@gmail.com]

Hi Dan, how are you?
I tell you, I am an IT advisor to the General Archive of the Nation, the server I am working on is Debian and they have had it for some years now. The version 2.6 that worked until recently was installed and served on said server without problems. Is it completely necessary for the installation to be carried out in Ubuntu LTS?
On the other hand, I tell you that the download of version 2.8 is done through git from the official repository. In the location /usr/share/nginx I have version 2.6 (atom_old) and version 2.8 (atom) in the same location, is that a problem?

What I am going to try to do today is to redo the entire installation procedure for version 2.8 and then the update steps, so I would like to ask you which services should be stopped before starting the procedures. ?

I really appreciate all the help you are giving me since for me it is the first time that I serve a web application through a server.

Greetings

[maxi...@gmail.com]

I think I know where the problem is. When executing the command "php -d memory_limit=-1 symfony tools:upgrade-sql" the response is: "upgrade-sql Successfully upgraded to Release 2.6.4 v193" when it should be 2.8 right? What I can be doing wrong?

[Dan Gillean]

Hi again, 

A few suggestions and responses for you: 

 Is it completely necessary for the installation to be carried out in Ubuntu LTS?

It is not completely necessary, no - we definitely have users who have successfully installed on other distributions, including Debian. You can see all posts from the last couple years that have a Debian tag in the user forum here: 

• https://groups.google.com/g/ica-atom-users/search?q=label%3ADebian
  

The main limitation is that we cannot offer detailed troubleshooting advice when you encounter errors. Our team does all development, testing, and production deployment for our hosting clients using Ubuntu LTS releases, and this is what we support in the official documentation. This means that if you choose to use something different, it is possible but we may not be able to help resolve issues, and you will need to check to see if all required dependencies / packages / etc are available (including the correct version) during installation. 

On the other hand, I tell you that the download of version 2.8 is done through git from the official repository. In the location /usr/share/nginx I have version 2.6 (atom_old) and version 2.8 (atom) in the same location, is that a problem?

So, before attempting to install 2.8 following Option 2 (install from our git repository), you moved the 2.6 version from /usr/share/nginx/atom to /usr/share/nginx/atom-old - is that correct? Did you also update the MySQL database name, the Nginx configuration block, the Elasticsearch index name, the PHP pool name, etc - all the other parts of the 2.6 installation that would potentially conflict with a new installation, since they would also use the same generic name (atom) during installation?

I think I know where the problem is. When executing the command "php -d memory_limit=-1 symfony tools:upgrade-sql" the response is: "upgrade-sql Successfully upgraded to Release 2.6.4 v193" when it should be 2.8 right? What I can be doing wrong?

 

It sounds like what I described above might be the issue. If you did not either change all the dependency names of your old installation, or give the new dependencies a different name (e.g. atom-new or atom2), there may be conflicts between the two installations. 

Most importantly, from what you described, you now have two locations: 

• /usr/share/nginx/atom - location of the new 2.8 installation
• /usr/share/nginx/atom-old - location of the old 2.6 installation

The upgrade task should only affect the AtoM instance whose installation directory you are currently in, so if you are going to review the installation process for your 2.8 installation and then retry the upgrade, some reminders: 

1. Make sure that you have changed directories to the right location - i.e. /usr/share/nginx/atom  and not   /atom-old
2. Make sure your server has all the expected dependencies and versions - PHP 7.4, MySQL 8, all the required PHP extensions, etc
3. For safety, make sure you give new installation components a different name - say, atom2 (for the database name, the search index name, the PHP pool name, the Nginx configuration block, etc) to avoid any possible conflicts with the older 2.6 installation
4. Make sure, if you are installing from our code repository, that you are using the stable/2.8.x branch (and not accidentally installing from stable/2.6.x!
• After cloning the repo and doing a git pull --rebase, you can double-check this by running git rev-parse HEAD, which should return the most recent commit. For stable/2.8.x, this should be f045e6ed6929f62e0c8070e7ebc1bf8cbbc1981e (visible here)
5. Because the upgrade process for major versions is to install a whole new instance and THEN load your old data and run the upgrade task, we should be able to confirm that the 2.8 site is working first. You may have DNS issues to work out before you can use the old URL for the new site, but in the meantime, from the command-line at /usr/share/nginx/atom, you can try running php symfony tools:get-version - this should hopefully return a 2.8 version number! 
6. As for upgrading, don't forget! If you are loading in a sqldump from your 2.6 installation, you still need to drop and recreate the database FIRST. See step 4 in this section. 

Hopefully this will help! Please let us know how it goes!

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/4413d167-896a-4bfe-b2ed-6b72dc3a3bf8n%40googlegroups.com.

[maxi...@gmail.com]

Hello Dan, good morning. I installed again from scratch and was able to get version 2.8 of AtoM working. Now the problem I have is that the images and videos that I had loaded in the previous version are not displayed. I ran the command "php symfony digitalobject:regen-derivatives" as the installation step says but I still can't see the derivatives. What can you recommend?

Thank you.

[Dan Gillean]

Hi again, 

I'm glad to hear that you've made some progress! 

Did you migrate a copy of your uploads directory? The uploads directory is generally located right under the root AtoM installation directory. Per the upgrade instructions, before you make a sqldump of your old (2.6) data, we also recommend using  either rsync (or cp) to migrate a copy of the uploads directory to the new (2.8) installation. See: 

• https://www.accesstomemory.org/docs/2.8/admin-manual/installation/upgrading/#copy-your-old-data
  

Additionally, make sure that the Base URL of your new site is properly configured - see: 

• https://www.accesstomemory.org/docs/2.8/admin-manual/installation/upgrading/#set-site-base-url
  
• See also: https://www.accesstomemory.org/docs/2.8/user-manual/administer/settings/#site-information

I believe that you had a DNS record in place to give your production site a specific domain name (http://atom.mininterior.gob.ar) - if the IP address or any other important details have changed for the new site, be sure to review this and update it with your domain provider if needed! 

Other things you could possibly try: 

• Make sure the filesystem permissions are properly set: sudo chown -R www-data:www-data /usr/share/nginx/atom
  
• Clear the application cache (php symfony cc) and restart PHP-FPM (sudo systemctl restart php7.4-fpm)

And if it's still not working - what happens if you create a test description and try linking a new digital object? Does it link and upload as expected?

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/fdd5e651-2028-42f2-9cbf-01d5aae727dan%40googlegroups.com.

[maxi...@gmail.com]

Hi Dan, I already migrated the folder to the uploads directory, now I am realizing that when I run "php symfony digitalobject:regen-derivatives" it gives me the following error:

The strangest thing is that all the files inside the uploads directory and its corresponding directories have -rwxrwxrwx permissions. What could you recommend to me? The URL of the page remains the same and execute the commands you mentioned before.

Greetings

[Dan Gillean]

Hi, 

Did you try re-running the filesystem permissions command I suggested? 

• sudo chown -R www-data:www-data /usr/share/nginx/atom
  

I am far from an expert in unix permissions, but I believe that the user / owner matters in this case, even if the permissions were previously generally open. 

Let us know if it helps! If not, I will see if our team has any other suggestions. 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/962e4c04-219a-40e4-8a99-4f8ac7e199e2n%40googlegroups.com.

[maxi...@gmail.com]

yes, i did it, but look:

i don't know what else do

[maxi...@gmail.com]

Maybe it is because the owner is www-data but if you look at the status of php7.4-fpm it appears as follows:



Shouldn't it be the name of the owner www?

[Dan Gillean]

Interesting....

Can you try running the Symfony command specifically as the www-data user?

• sudo -u www-data php symfony digitalobject:regen-derivatives
  

If that doesn't work, I will see what the team has to offer for additional suggestions. 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/b68d0586-1a39-4b61-ba0a-667bf8f57ec0n%40googlegroups.com.

[maxi...@gmail.com]

Look, i change the owner to root and its a new error:

[maxi...@gmail.com]

I did it dan!! OMG I just had to first run "sudo chown -R www-data:www-data /usr/share/nginx/atom", then "php symfony cc", then "sudo systemctl restart php7.4-fpm" and finally "sudo -u www-data php symfony digitalobject:regen-derivatives"

Thank you very much for all the help and I hope this conversation helps someone who has the same error in the future. Greetings

[Dan Gillean]

So glad to hear you've gotten it sorted out - thanks for sticking with it, even when things were uncertain! 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/ce678c3c-81d7-408c-a62b-85cd86cc0255n%40googlegroups.com.

[maxi...@gmail.com]

Hi again! i need some help because i want to change the template of the finding aid. How can i do that?

[Dan Gillean]

Hi there, 

Finding aids in AtoM take the EAD 2002 XML that is generated from an archival description hierarchy, and then use an XSLT to transform the XML into either a PDF (or RTF) finding aid. As the docs note, there are two separate stylesheets for this - the Inventory Summary and the Full Details finding aid layouts. There's also an XSL file that contains some transformations that are common to both layouts (to avoid duplication in the code we maintain). 

You can find all the XSLT's here: 

• https://github.com/artefactual/atom/tree/qa/2.x/lib/task/pdf
  

Because users have provided a community version of our XSL files with French headers, the wiki includes an example of the commands to run after replacing the default XSLTs with your own. See: 

• https://wiki.accesstomemory.org/Community/Community_resources/Customization#Instructions_on_swapping_XSLT_files
  

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/e802fac4-bf84-4b8a-abfd-9d817ae3c752n%40googlegroups.com.

[maxi...@gmail.com]

Hi Dan! How are you? I need your help. I want to do a request GET to /api/informationobjects but i need the api key. How can i configurate the api key?

[Dan Gillean]

Hi there, 

First, you need to make sure that the arRestApiPlugin is enabled in Admin > Plugins. See: 

• https://www.accesstomemory.org/docs/latest/dev-manual/api/api-intro/#enable-the-api-plugin
  

Then, you can navigate to your own user account to generate an API key. See: 

• https://www.accesstomemory.org/docs/latest/dev-manual/api/api-intro/#generating-an-api-key-for-a-user
  

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/c45a84c4-1bd0-4f26-a9ee-1f1c6dd9b823n%40googlegroups.com.

[maxi...@gmail.com]

Hi Dan, how are you? I tell you that I was able to solve all the issues so far. But now another problem arises for me. In the related materials area, specifically the "related descriptions" dropdown menu is not working well. I entered the name and title of a description that I want to relate to another and an empty list was displayed that returned the value "undefined". What could be happening? Do I have to run "php symfony propel:build-nested-set"? I appreciate all the help you give me. Greetings

[Dan Gillean]

Hi there, 

Hmmm, it's hard to say what the issue is without knowing more specifics, but yes - starting with running some common maintenance tasks is a good idea. I would suggest you try: 

• Generate slugs - if you run this task with no added options, it will only generate slugs if they are missing
• Build nested set
• Restart PHP-FPM
• Clear application cache
• Repopulate search index

You can find links and instructions for all these commands starting from our Troubleshooting page: 

• https://www.accesstomemory.org/docs/latest/admin-manual/maintenance/troubleshooting/
  

Hopefully this will resolve the issue - let us know how it goes! 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/2c06f538-22a4-4d29-9596-36c3aa565a89n%40googlegroups.com.

[maxi...@gmail.com]

Hello Dan, how are you? I did the tasks that you gave me and yet the drop-down menu still does not work correctly. What I did do and found to work is:
1st write the name of the documentary unit that I am searching completely
2nd hit enter
3rd execute the command "sudo php -d memory_limit=-1 symfony search:populate"

By performing this step the unit is added to the suggestion list. However, the suggestions it makes are not so intuitive. I am attaching screenshots to see if they can give me a hand. Thank you

[Dan Gillean]

Hi again, 

I think that I see the issue now - thank you for sharing the screenshot, it actually helped a lot! 

I believe this is a translation issue. Your descriptions are in Spanish, and from what I can tell, it appears that YOU are properly in the Spanish interface when editing this record. 

However, see how most of the free-text field values appear in yellow boxes above the edit fields? That is what AtoM shows when it thinks you are adding a TRANSLATION. Meaning: when this record was originally created, the source culture for it was set to English, despite the fact that the actual content is in fact Spanish. Now, because of that, AtoM is only returning other records it thinks are in English in the autocomplete fields - meaning there is now probably a mix of "English" and Spanish records in your site. 

This can happen a few different ways, such as: 

• The default culture of the site is not set during installation - so it defaults to English
• In this case, all templates would be EN by default, and any descriptions created while the user interface is set to EN will be considered to have an English source culture
• The culture column is not populated in a CSV import - so again, it will default to English when not defined
• I am not actually sure if it defaults to English when blank or if it defaults to the installation culture, but: 
• If it defaults to EN then that would be a potential import problem, or 
• If it defaults to the installation culture but THAT was not defined during installation (so the installation culture defaults to EN), then the result would be the same
• The language menu is used to set the site to English, and THEN a user creates the description
• In all cases, whatever culture the user interface is in when a user begins creating a record, that culture will be used as the source culture for the description
• Meaning, even if the user is TYPING in Spanish, if AtoM thinks it is in the EN interface, it will treat the description as an English description. 

I am not sure which of these was the exact cause, but it's likely a combination of them. Either way, the problem is a result of AtoM thinking that some of your Spanish descriptions are in fact English, and the autocomplete fields only return results from the current culture by default. 

So! Short term, to find the right record in the autocomplete, you have two options: 

1. Exit edit mode. Use the Language menu to flip the user interface to English. Enter edit mode, and then try to find and link the right description in the autocomplete. Save.
2. Navigate to the description that you want linked. With the user inteface set to Spanish via the language menu, enter edit mode. Add a "translated" Spanish title in the title field. You can also "translate" whatever other fields you want, but the key one we need is the title for the autocomplete to work. Save, then go back to the original description and repeat your attempt to link the two via autocomplete

Resolving this issue long-term is a bit more complicated. 

First of all, does your site have real multilingual content? I.e. are there actual real English descriptions in your site, or are all records in your system intended to be in Spanish?

Second, can you double-check the installation culture value that is set in AtoM's configuration files? You can find this by accessing the apps/qubit/config/settings.yml configuration file, and looking for the default_culture value. It should be either es for Spanish, or en for English. See: 

• https://www.accesstomemory.org/docs/latest/admin-manual/customization/config-files/#apps-qubit-config-settings-yml
  

With more information, I can hopefully suggest some next steps for you. 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/cc271cc4-79e8-4ac4-a610-0b7b2b66efdbn%40googlegroups.com.

[maxi...@gmail.com]

Thanks for the information. All descriptions are made in Spanish, none have the English language. I leave you a screenshot of the settings.yml file

[Dan Gillean]

Okay, if they are all intended to be in Spanish, then that at least makes it slightly easier to troubleshoot! 

Unfortunately however, there is no easy way to do this. There are a few ways you could proceed, but without knowing more about your installation and how wide-ranging this issue is, it is hard to say how successful each possible method will be. 

Essentially, you could: 

1. Leave things as they are, and only fix them as needed by adding a "translation" to missing records as you find them, as I described in my last post. This is the least amount of work, and only resolves issues as you encounter them. Export behavior will be weird unless done from the command-line, and search/filter options based on language will not work as expected, but if that hasn't been an issue so far and you are not too concerned with the implications of the issue, you could take this minimal approach. 
2. Leave things as they are, and try to bulk add Spanish "translations" for all content where it is missing. There will still be some weirdness with Language controls, and possibly with exports from the user interface, but a) it's possibly less work, and b) it will resolve the immediate issue with missing records in drop-downs and autocompletes
3. Try exporting everything, updating the culture values in the exported CSVs, wipe your current data, and then reimport everything. More work than the previous options but still less work than the final option below. Will not work as a solution for some entities (such as any terms / access points or other records that don't have good import/export support for this kind of thing), meaning you will still need to do some manual translation. Additionally, the order of your imports will matter, to avoid reproducing this issue. 
4. Try to fix this at the database level, so that AtoM knows that these are all Spanish records, and not just Spanish writing added to an English record. This will be the most work, but has the best chance of permanently resolving the issue. 

I am going to attempt to describe as much of the process for option 4 as I can below, because this issue has come up before, and I'd like to try to write down as much as I have figured out for resolving it. However: 

• If you would prefer more information on any of the other 3 options, I can elaborate on any of them. Note that I have not personally tested ANY of these methods 100%.
• I do not have the knowledge to provide you with ALL steps in option 4 - it will require some research and experimentation on your own, meaning learning about or finding someone familiar with SQL queries and databases. In general, detailed step-by-step instructions for ANY of these solutions is getting beyond the level of support we can provide in the forum - I will do my best, but there are limits! 
• This means that we will not be responsible for the outcomes of any steps you take following the general advice shared here. Please be sure to back up your data before anything, and proceed at your own risk! 
• If you would like to discuss the possibility of paid support to resolve this, you can contact Artefactual off-list to discuss options further

So: For option 4, we are going to have to progress one table at a time as we review your data and try to properly update it to the correct culture. 

Initial steps and database schema overview

First things first: please make a backup of your database! We will be attempting to make changes directly in the MySQL database, so we want to make sure that if we get anything wrong, you have a backup you can load. 

All of these steps will require access to the MySQL command prompt. See the following link for instructions. Additionally, you will need to know the database username and password set during installation - but the link below also includes ways to check this information if you don't know or remember it: 

• https://www.accesstomemory.org/docs/latest/admin-manual/maintenance/common-atom-queries/#cli-access-mysql
  

Now, to backup the database, see: 

• https://www.accesstomemory.org/docs/latest/admin-manual/maintenance/common-atom-queries/#backing-up-the-database
  

Here is where things will get more complicated. An archival description is spread across a number of different tables in AtoM's database. If you would like a visual to accompany the following explanation, we maintain Entity Relationship Diagrams that describe AtoM's database schema on our wiki, here: 

• https://wiki.accesstomemory.org/Development/ERDs
  

Below I will try to give a summary of the various tables we will need to check and update, before providing some initial instructions on how to do so. I've also previously given a high-level overview of AtoM's data schema in this thread: 

• https://groups.google.com/g/ica-atom-users/c/ACZH5mTOd2E/m/57EnlvRaCgAJ
  

First, to handle any free-text field that can be translated, there are i18n tables (this is short for "internationalization" - bc in english this word has 18 characters, so developers regularly shorten it to i18n) for each major entity. So for example, non-translatable archival description fields (like the identifier, the internal object ID, relations to other entities and lower-level descriptions, etc) are stored in the main information_object table. Most free-text fields that can be translated (like the title, the scope and content, the extent and medium, etc) are found in the related information_object_i18n table. So, the first place we'll need to check and update the source_culture values will be the information_object_i18n table. However, there are other places we will need to consider as well. 

Next, it's helpful to know that AtoM's data model was originally built around the ICA standards - meaning that fields from other standards, or AtoM-specific fields that do not map exactly to ISAD(G) for example, will go into separate tables. The main ones we use for this type of overflow are the property tables - property and property_i18n. 

Additionally, there are separate notes tables in AtoM, so even some ISAD(G) note types will be there, rather than in the main information object tables, such as Archivist's Note, General Note, etc. So note_i18n will be another table to check and update. 

Finally, there are the related entities to consider. For example, every time you add a subject, place, or genre access point, this is creating a (or linking to an existing) term in one of AtoM's taxonomies. Every time you add a creator or a name access point, AtoM will create or link to an authority record. The archival institution name will create or link to a repository record. There are physical storage containers, accession records, functions, rights statements and rightsholder records, and more to consider. All of these related entities work similarly to the information object tables - there will be a base table for fields that do not require translation, and then an i18n table. There's even a relation and relation_i18n table that is used for tracking the connections between some records - for example, when relating two different authority records together. It will be good long-term to check all of these entities as well, as many new stub records can be created in AtoM during the description creation process, whether via the user interface or via import. 

Now, because AtoM is a multilingual application that can support multiple translations per record,  all of these tables should have a column that defines the culture for any given row. I think this is generally called source_culture across the base tables, and just culture in the i18n tables so that's what we'll look for first. 

So, now is the time to access the MySQL command prompt! 

Updating the main entities

In each table, we are going to want to: 

• In the base entity, perform a count of how many records have the source_culture value set to en - this step is optional, but gives us a sense of how many records are affected, so we can check that the update queries are working as expected
• Select these records, and update the source_culture value from en to es in the selected records
• Check the related i18n table's culture values for en records
• Update those to es as well

Now, if you are (or have access to) a developer, it would be possible to craft a script that does everything we are going to do below, across all of AtoM's tables. I do not have such skills, so I will describe the manual process!

Let's start with the information object records - the main part of your archival descriptions affected by this issue. Fortunately, we already have an example of this query in our documentation we can use, at the start of this section. We will update the selected culture in that example query to English, like so: 

• SELECT COUNT(*) FROM information_object WHERE source_culture='en';
  

That should tell us how many descriptions are affected. 

Now we will try to update them: 

UPDATE information_object io

SET io.source_culture TO "es";

You can then run the COUNT query again to see if it worked as expected. There shouldn't be any en records left! Next, we can try to do the same for the i18n table. 

• SELECT COUNT(*) FROM information_object_i18n WHERE culture='en';

Then: 

UPDATE information_object_i18n io18

SET io18.culture TO "es";

Essentially, from here, we are going to need to repeat this with pretty much ALL the main entity tables in AtoM! So, you will need the most recent Entity Relationship Diagram as a guide (the most recent one is a searchable PDF), because while I will list more examples below, it will be up to you to work out how to update the remaining tables. I would say that you should do this next for 

• The actor and actor_i18n tables
• The repository and repository_i18n tables

Access points

Now, this should address *most* of the translation issues in your site. Further changes are going to get more complicated, and we will need to be more careful as we proceed, and/or make some compromises.

First, let's do the easiest part of the hard part, haha - by which I mean, terms in taxonomies. AtoM does include some default (English) terms that are loaded as fixtures during installation. These terms we want to keep, because they are often the controlled values straight from the related standards templates, used in drop-down in edit pages. Consequently, we will only want to update some taxonomies. 

So, let's start with the access points, because for Subjects and Places at least, these taxonomies arrive empty when AtoM is first installed - meaning the only terms in there should be Spanish ones you have added. 

We will need to know the taxonomy ID for this, but fortunately, it's a stable number in our docs already: 

• Subjects taxonomy ID: 35
• Places taxonomy ID: 42

You can actually see an output of all taxonomy names and their IDs with: 

• SELECT id, name FROM taxonomy_i18n WHERE culture='en';
  
• See: https://www.accesstomemory.org/docs/latest/admin-manual/maintenance/common-atom-queries/#see-the-id-and-names-of-taxonomies-in-atom

So, let's update the Subject terms. First, we will do the base term table, using the taxonomy_id as our limiting clause on the select: 

UPDATE term t

SET t.source_culture TO 'en'

WHERE t.taxonomy_id='35';

Now we will try the same on the term_i18n table, but we will need to join it with the term table to use the same limiting WHERE clause: 

UPDATE term_i18n t18

JOIN term t ON t.id=t18.id

SET t18.culture TO 'en' 

WHERE t.taxonomy_id='35';

We can then repeat those two steps for Places, by swapping in 42 instead of 35 in the two example queries above. You can look at the output of available taxonomies, and consider if there are others that do not have default English terms - if so, and you have used them, then you can also repeat the update for these. For example: if you've added Actor Occupation access points (taxonomy ID: 80). If you're not sure, don't update it! Use the user interface instead - navigate to Manage > Taxonomies, find the right taxonomy, and look at the terms. You can always just flip the user interface to Spanish, enter edit mode, and add a Spanish translation if you're not sure.

Notes 

Now we are getting into the much more experimental areas! Proceed with caution! And note that I have not tested all of these queries, so if the previous steps worked, make another SQLdump backup of your database now, so we can roll back if something goes wrong! 

Let's consider the note tables next. Here's a quick tip: you can always use the DESCRIBE option to get a sense of what fields are in a table: 

mysql> DESCRIBE note_i18n;
+---------+-------------+------+-----+---------+-------+
| Field   | Type        | Null | Key | Default | Extra |
+---------+-------------+------+-----+---------+-------+
| content | text        | YES  |     | NULL    |       |
| id      | int         | NO   | PRI | NULL    |       |
| culture | varchar(16) | NO   | PRI | NULL    |       |
+---------+-------------+------+-----+---------+-------+
3 rows in set (0.00 sec)

mysql> DESCRIBE note;
+----------------+---------------+------+-----+---------+----------------+
| Field          | Type          | Null | Key | Default | Extra          |
+----------------+---------------+------+-----+---------+----------------+
| object_id      | int           | NO   | MUL | NULL    |                |
| type_id        | int           | YES  | MUL | NULL    |                |
| scope          | varchar(1024) | YES  |     | NULL    |                |
| user_id        | int           | YES  | MUL | NULL    |                |
| source_culture | varchar(16)   | NO   |     | NULL    |                |
| id             | int           | NO   | PRI | NULL    | auto_increment |
| serial_number  | int           | NO   |     | 0       |                |
+----------------+---------------+------+-----+---------+----------------+
7 rows in set (0.00 sec)

We can run a count of English rows in these note tables with: 

• SELECT COUNT(*) FROM note WHERE source_culture='en';
  
• SELECT COUNT(*) FROM note_i18n WHERE culture='en';

Here is where we need to pause. 

Now, note tables will potentially have some of your archival description content - as I mentioned earlier, even ISAD(g) fields like Archivists Notes, or General Notes, etc, can appear in this table. However, we don't want to bulk update ALL notes, because they are also used for other entities, such as the Scope notes on terms. You would think that in an all-Spanish site there should be no issue bulk updating all of them.... but don't forget: AtoM includes a number of (English) default taxonomy terms when installed. And many of these affect how the underlying system works, and/or control the available values in standards-based templates, including ISAD(G), so we don't want to bulk update everything! 

What follows below are just some examples - you may need to work out your own ways to proceed, since I haven't tried all of this myself. 

Here's one way (I think I might have found a better way below, but some of this might still be useful so I will leave it): 

Method 1 - note by note

We can try looking at some results from the note_i18n table, outputting 50 records at a time, and then skipping 50 in the next query to output 50 more. Perhaps we can even join with the note table and limit this to those affecting descriptions:

SELECT content,object_id FROM note_i18n no18 

JOIN note no ON no18.id=no.id 

WHERE no18.culture='en' AND no.scope='QubitInformationObject' 

LIMIT 50;

This would output only the actual note content, and the related description's object ID number for the first 50 results. If you wanted to skip the first 50 results and output the next 50, we could do it like so: 

SELECT content,object_id FROM note_i18n no18 

JOIN note no ON no18.id=no.id 

WHERE no18.culture='en' AND no.scope='QubitInformationObject' 

LIMIT 50, 50;

If we see notes that we know belong to our descriptions (and therefore need updating) we can try to use the related object as a select, to update that record: 

UPDATE note_i18n no18

JOIN note no ON no18.id=no.id 

SET no18.culture to 'en'

WHERE no.object_id='12345';

Only, replace the 12345 value with the actual object_id value you saw in the output from the query above. Then we repeat this but just on the note table (also replacing the 122345 placeholder with the actual object ID value:

UPDATE note no

SET no.source_culture to 'en'

WHERE no.object_id='12345

As you can imagine, this is going to be time consuming, unfortunately! So, let's try looking at this another way...

The (hopefully) better way to update notes

If we look in our docs, there is an example query that will output the IDs of the different note types. That might allow us to better restrict our updates to just those note types that we know are related to our descriptions. For example, if you are using the ISAD(g) description template, then really we just need to worry about General note and Archivists note, possibly the Language note. See the example query here: 

• https://www.accesstomemory.org/docs/latest/admin-manual/maintenance/common-atom-queries/#note-queries
  

So, General note has an ID of 125. Let's try crafting a query that will limit our selection to those notes used as General notes: 

UPDATE note_i18n no18

JOIN note no ON no18.id=no.id 

SET no18.culture to 'en' 

WHERE no18.culture='en' AND no.type_id='125'

UPDATE note no

SET no.source_culture to 'en' 

WHERE AND no.type_id='125'

We could repeat this for Archivist's notes by change the type_id value in the queries above to 124. 

Language notes will appear in other entities, but I think these should be the main entities without any default values, so you should be able to bulk update those notes as well. 

For the source notes and scope notes of taxonomy terms: 

• If you have added this info to access point taxonomies (like places and subjects) then we will need to deal with those. In that case, you will need to see if you can craft a way to join the term tables to the note tables (likely via the object table, which connects pretty much everything), so we can try to selectively ONLY update scope and source notes that affect the access point taxonomies we already updated. Let me know if this is in fact an issue, and I can try my best to help crafting something
• If you haven't  added scope /source notes to your access point entries, then you can probably ignore those :D

Other tables

There may still be other tables we need to consider, such as: 

• Physical storage containers that were created and linked to your descriptions
• Accession and Deaccession records, and their related Donors
• The property and relation tables
• There is an other_names_i18n table that might be used by any of your authority records  - fortunately, that one should be simple to update
• etc

For cases where you are not sure how to proceed, one easier alternative option will be to: 

• Make sure the user interface is set to Spanish via the language menu
• Navigate to the English-culture records you want to address and enter edit mode
• Add a "translation" in the Spanish interface, and save

At least this way, they will properly appear when you are searching and browsing in Spanish! And you can potentially correct the "English" versions later. 

Hopefully, you do not have too much mixed-language content in these areas, and hopefully the information included in this long message will help you understand how you can at least start investigating! 

Remember, back up often, proceed carefully, and let us know how it goes

After making updates

When you are done making updates via SQL: 

 

First, do NOT get rid of your backups just yet. Make sure the system works as expected for the next week at minimum, to confirm there not any unexpected low-level issues caused by the changes that take some time to appear. In general it is always wise to take regular backups of your production site. 

You will also need to run some of AtoM's maintenance tasks before using the system again. I would recommend basically running all of the most common ones, such as: 

• Rebuild the nested set
• Generate slugs
• Clear the application cache
• Restart PHP-FPM
• Re-populate the search index

You will find links and further instructions for all of these tasks from our Troubleshooting page: 

• https://www.accesstomemory.org/docs/latest/admin-manual/maintenance/troubleshooting/
  

Good luck!

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/a1899404-077d-44ce-931f-ae91f778aad0n%40googlegroups.com.

[maxi...@gmail.com]

Hi Dan, thank you for the information you gave me with the previous topic. I have a new problem: when editing a particular archival institution (this does not happen with all archival institutions) I get a 500 server error. When checking the error.log it gives me the following error: 

*1334 FastCGI sent in stderr: "PHP message: PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 135168 bytes) in /usr/share/nginx/atom /lib/QubitQuery.class.php on line 146" while reading response header from upstream, client: 10.6.242.223, server: atom.mininterior.gob.ar, request: "POST /index.php/archivo-general-de- la-nacion-argentina/edit HTTP/1.1", upstream: "fastcgi://unix:/run/php/php7.4-fpm.atom.sock:", host: "atom.mininterior.gob.ar", referrer: "https://atom.mininterior.gob.ar/index.php/archivo-general-de-la-nacion-argentina/edit"

I changed the memory_limit variable to 10G, although before it was at -1 and the error persisted. However, none of this worked. Do you know what could be happening? What could I do to fix it?
Thank you so much!

[Dan Gillean]

Hi there, 

How much memory does the server actually have? Changing the PHP memory_limit may not help if there is not enough actual memory available. That would be my first suggestion - check how much memory your AtoM installation server has, and if possible, increase that amount. The minimum we recommend for a small-to-medium production-ready server is 7GB of RAM. More is better if possible! 

Another thing to check: make sure that you have checked and adjusted any PHP execution limits in both: 

• PHP pool set up during installation - typically found at: /etc/php/7.4/fpm/pool.d/atom.conf
• See: https://www.accesstomemory.org/docs/latest/admin-manual/installation/ubuntu/#php-fpm
• Your general php.ini file for the server
• https://www.accesstomemory.org/docs/latest/admin-manual/installation/execution-limits/#the-php-ini-file
  

Make sure that after making any changes you have cleared the application cache and restarted PHP-FPM after: 

• php symfony cc
• sudo systemctl restart php7.4-fpm

Something else that could be causing issues.... when creating archival descriptions, are you manually linking every level of description in a hierarchy to the related archival institution? Or, are you linking once at the top level, and allowing for inheritance to handle links at lower levels? 

Both creators and archival institutions linked to descriptions will be automatically inherited to lower level records in the same hierarchy, unless an editor explicitly adds a different creator/institution at a lower level. The purpose of this is three-fold: 

1. It follows the ICA's general principles of description, which suggest that we should add information at the relevant level, and not repeat information at other levels unecessarily
2. It is supposed to simplify the creation of an archival hierarchy for editors, allowing you to focus just on the fields that matter / are unique at lower levels
3. Most relevant here: it helps with scalability and performance, by reducing the number of direct checks and updates that must be made when a related record is changed

If you are manually linking the repository name at every level, and you have 100 fonds, each with 100 lower level records under them, then AtoM needs to invididually check and update 10,000 related descriptions when you edit the related archival institution name. This is likely to time out and/or exhaust the memory of the system! However, if you are only linking at the fonds level and allowing inheritance to handle the lower levels, then AtoM only needs to make 100 related updates when you edit a repository name - a much less resource-intensive process. 

We have a command-line task that can automatically check and remove creator names where the default inheritance behavior would give the exact same result: 

• https://www.accesstomemory.org/docs/latest/admin-manual/maintenance/cli-tools/#unlink-creators-from-child-descriptions-and-reapply-inheritance-to-hierarchy
  

Unfortunately, we do not have any equivalent task right now for archival institutions, meaning you will need to check for yourself, and possibly make manual updates to your records if this is in fact an issue. 

An easy way to check without needing to enter edit mode on every single record: if you navigate to a lower level description in an archival unit's hierarchy, you can hover your cursor over the archival institution name in the view page. If it is being inherited, then a tooltip should show that says this, like so: 

Source is from our demo site, if you wanna try it yourself: https://demo.accesstomemory.org/records-of-1st-coniston-wold-cub-pack-group-charter-84-local-association-of-sudbury-ontario

If you DO find that you have linked your archival institution at all levels instead of just at the parent level: 

One option, if you have too many descriptions to make manual updates to them easy, would be to do something like: 

• MAKE A BACKUP FIRST, just in case there are unintended consequences to this suggestion! 
• Make a note somewhere outside of AtoM of all the top-level records (e.g. fonds, collections, etc) that should be linked to the archival institution 
• you could even filter the search results by repository, then add all the top-level records to the clipboard and export them without their lower levels, so you have a reference doc - we just need this so we know which ones will need to be manually corrected later
• Add one of your archival institutions record to the clipboard, and export as CSV
• Delete the archival institution in AtoM
• Reimport the archival institution
• Now you will need to manually re-link just the fonds-level records to the updated archival institution

Hopefully, if you have a lot of descriptions, this would reduce the work required to dozens or hundreds instead of thousands of manual updates! 

Keep in mind that this would also have other potential impacts that you should double-check first, such as: 

• The mixed culture issues in your AtoM site might impact this. If you try to export your institution and it is not complete, then you may need to take screenshots and then manually recreate the archival institution afterwards instead. AtoM has some known issues exporting non-English multilingual content via the user interface that our Maintainers are trying to address for the next release. Normally, the command-line exports bypass this issue and export all data regardless of culture, but we do not have a user-accessible command-line export for repository records in AtoM at present. 
• I confess I am not sure how the other known language issues will affect things. If you can, I would try to address that issue first, then this one. As always, back up often and be sure to do a good review of everything before deciding whether or not something worked!
• If you have been linking authority records to your archival institution as the maintainer, those links would be lost and would also need to be manually recreated, as they will also be lost when the original archival institution is deleted
• You will want to confirm after reimporting your archival institution that no data was lost in the roundtrip - for example, any access points, etc. Those terms should all still exist in AtoM when the archival institution is deleted, and should re-link to the existing terms when re-imported, but it will be good to double-check 

That's all I can think of for now! Let us know what you find, and how it goes! 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/b4bce34a-c532-44c1-b126-ca293f14c0a2n%40googlegroups.com.