DB and PHP errors after upgrading from 2.6.4 to 2.7.3

[Patrick Goetz]

We run AtoM in an LXD container, so upgrading amounts to

1. Spinning up a new Ubuntu 20.04 container

2. Following the installation steps outlined in the AtoM documentation

3. mysqldump the atom database in the container running 2.6.4

4. Load the mysql file into the 2.7.3 atom database in the new container.

This didn't work as well as one would hope, so based on a previous problem ticket on this forum, I ran the following commands:

 # php symfony propel:build-nested-set
 # php symfony cc
 # php symfony search:populate

This restored most of the functionality, but I'm still getting "500 Internal Server Error" when, for example, clicking through a record group.  Looking at the nginx error log, I"m seeing this when a query fails:

----------------
2023/10/09 05:57:01 [error] 226#226: *90 FastCGI sent in stderr: "PHP message:  [wrapped: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'digital_object.LANGUAGE' in 'field list']" while reading response header from upstream, client: 10.87.130.40, server: _, request: "GET /index.php/gc-01-r0009 HTTP/1.1", upstream: "fastcgi://unix:/run/php7.4-fpm.atom.sock:", host: "catalog-dev.episcopalarchives.org", referrer: "https://catalog-dev.episcopalarchives.org/index.php/informationobject/browse?topLod=0&sort=relevance&query="

2023/10/09 06:08:48 [error] 226#226: *132 FastCGI sent in stderr: "PHP message: The component does not exist: "object", "clipboardButton"" while reading response header from upstream, client: 10.87.130.40, server: _, request: "GET /index.php/joint-nominating-committee-for-the-election-of-the-presiding-bishop HTTP/1.1", upstream: "fastcgi://unix:/run/php7.4-fpm.atom.sock:", host: "catalog-dev.episcopalarchives.org", referrer: "https://catalog-dev.episcopalarchives.org/index.php/actor/browse"

----------------

The first error above makes it look like the database schema changed between versions 2.6.4 and 2.7.3? The second I"m not sure about.

Any ideas on what I can do to debug this further?  The 2.6.4 catalog works perfectly.

[Dan Gillean]

Hi Patrick, 

I think you're on the right track suspecting schema changes - we typically try to only include these in major releases, so following the existing Upgrading instructions is particularly important when upgrading between major release version (e.g. 2.6.x to 2.7.x). The process you describe doesn't sound like it accounts for schema changes between versions, which as you noted, looks like it could be the cause of the first error message there (the SQL error). Please see the existing upgrade instructions - it's very important when upgrading major versions that you: 

• Drop and recreate the database BEFORE you load your data (this also allows for changes in collation, SQL modes, etc)
• Run the tools:upgrade task, which applies any schema migrations included in the release, AFTER loading your SQLdump
• THEN clearing the cache, restarting services, and repopulating the index. 

See: 

• https://www.accesstomemory.org/docs/2.7/admin-manual/installation/upgrading/
  
• The step to drop and recreate the database is Step 4 of this section

I think if you purge your site (you can use the tools:purge command to flush all data - I would suggest using the --demo option as it's quicker, and any user / site information you enter when using this task will be overwritten when your sqldump is loaded anyway), and then do the steps above, it should likely resolve your issues. 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/3a9e0666-f40d-4a39-8cc1-11b00d2e1d0bn%40googlegroups.com.

[Patrick Goetz]

Thanks, Dan -

Running `php -d memory_limit=-1 symfony tools:upgrade-sql`, clearing the cache, and repopulating the search index solved almost all the problems.  There is still one "Browse by" menu item that isn't working correctly.  When I click on a Record to bring it up, I get a "500 Internal Server Error" which amounts to this:

2023/10/09 18:31:49 [error] 9458#9458: *156 FastCGI sent in stderr: "PHP message: The component does not exist: "object", "clipboardButton"" while reading response header from upstream, client: 10.87.130.40, server: _, request: "GET /index.php/fort-valley-high-and-industrial-school HTTP/1.1", upstream: "fastcgi://unix:/run/php7.4-fpm.atom.sock:", host: "catalog-dev.episcopalarchives.org", referrer: "https://catalog-dev.episcopalarchives.org/index.php/actor/browse"

That looks either a coding or package inclusion error.  Is the object "clipboardButton" part of the default AtoM distribution? I still have the 2.6.4 catalog running and this works fine there.  Just to be sure, I checked the nginx error logs on the 2.6.4 server and this message does not appear there.

[Dan Gillean]

Hi Patrick, 

I found an older thread in the forum with a similar issue that recommended making sure that the following line appears in your custom theme's _header.php file. See: 

• https://github.com/artefactual/atom/blob/qa/2.x/apps/qubit/templates/_header.php#L35
• Here it is in the ArchivesCanada custom theme, as an example: https://github.com/artefactual/atom/blob/qa/2.x/plugins/arArchivesCanadaPlugin/templates/_header.php#L87 
  

I would suggest trying that, and then recompiling your custom theme. If you've developed a custom theme I am hoping that means you already have the dependencies (nodejs, npm, lessc) installed - but if not, let me know and I will point you to those. Instructions for recompiling can be found in the Upgrading instructions here, using the ArchivesCanada theme as an example - just use the name of your custom theme plugin instead: 

• https://www.accesstomemory.org/docs/latest/admin-manual/installation/upgrading/#recompiling-after-making-modifications
  

To avoid other similar issues, make sure you have double-checked and applied the recommended changes for upgrading with a custom theme, found here: 

• https://www.accesstomemory.org/docs/latest/admin-manual/installation/upgrading/#upgrading-with-a-custom-theme-plugin
  

If that doesn't work, it will help to know a bit more about your installation, such as: 

• What is the full version number shown in Admin > Settings (so we can double-check that the schema version is correct for your installation)?
• Have you made any changes to the recommended installation dependencies (O/S distro and version; MySQL, ES, PHP versions in particular)? If yes, what exactly?
• Did you install following Option 1 (using the downloadable tarball from our website) or Option 2 (installing from the GitHub code repository)?
• Anything else about this particular installation that it might help us to know?

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/c93b6338-30b1-40ec-b7e9-51e4d0be765bn%40googlegroups.com.

[Patrick Goetz]

Hi Dan -

Thanks for the speedy response.  I've never had to recompile the custom theme for previous upgrades, so am in uncharted waters here. So yes, please point me to something documenting the necessary components in the toolchain for doing this. Also, I'm confused: what exactly is being compiled, given that this is a PHP app?

Just so that it's documented, the full version of the application is 2.7.3-192

The person who created the custom theme also made modifications to some other plugins:

       atom/plugins/arDacsPlugin/modules/arDacsPlugin/templates/editSuccess.php
       atom/plugins/arDacsPlugin/modules/arDacsPlugin/templates/indexSuccess.php
       atom/plugins/sfIsaarPlugin/modules/sfIsaarPlugin/templates/indexSuccess.php
       atom/plugins/sfIsaarPlugin/modules/sfIsaarPlugin/templates/editSuccess.php
       atom/apps/qubit/modules/informationobject/actions/notesComponent.class.php

Since the compilation instructions look like this:

    make -C plugins/arArchivesCanadaPlugin

i.e. assume all the customizations are in one folder, I"m wondering if I also need to address these additional customizations somehow?

[Dan Gillean]

Hi Patrick, 

Also, I'm confused: what exactly is being compiled, given that this is a PHP app?

AtoM uses the Bootstrap CSS framework for its front-end styling. We use LESS in AtoM to extend CSS and minimize the size of the stylesheets in our repository, and LESS compiles them to CSS when run. This is already done in the downloadable tarball, but when installing from our GitHub code repository, users need to first install the required dependencies, and then compile the stylesheets themselves. As such, you can find the prerequisites near the end of the Option 2 (install from our code repo) installation instructions: 

• https://www.accesstomemory.org/docs/latestadmin-manual/installation/ubuntu/#option-2-check-out-the-code-from-our-git-repository
  

Because we are assuming users are running the recommended version of Ubuntu and 20.04 includes nodejs by default, we skip that dependency - however, if you need to install nodejs and npm first, here is how you can do it on Ubuntu: 

• https://linuxize.com/post/how-to-install-node-js-on-ubuntu-20-04/
  

Then, the instructions from our docs to install the rest of the deps: 

• sudo apt install npm make
• sudo npm install -g "less@<4.0.0" n
• sudo n stable
• sudo npm install
• sudo npm run build

Once they are installed, we can recompile the base themes:

• sudo make -C /usr/share/nginx/atom/plugins/arDominionPlugin
• sudo make -C /usr/share/nginx/atom/plugins/arArchivesCanadaPlugin
  

You can also repeat that process for your custom theme, by using the name of the plugin instead of the two existing base theme plugins as above. 

Finally, note that after you have recompiled everything, you may want to clear the application cache (php symfony cc), restart PHP-FPM, and clear your web browser cache as well (or test in an incognito / private browser window, where the cache is typically disabled by default), just to ensure that you are seeing the most up-to-date version of your AtoM instance and not a cached copy from before the recompile. 

Just so that it's documented, the full version of the application is 2.7.3-192

Great! That is the expected database schema version for 2.7.3, meaning that there shouldn't be any further database issues, and the upgrade task completed successfully. Note that we have started adding the expected db schema version to the release notes, so you can always double-check against your own installation. See for example: 

• https://wiki.accesstomemory.org/Releases/Release_announcements/Release_2.7.3 

i.e. assume all the customizations are in one folder, I"m wondering if I also need to address these additional customizations somehow?

Normally, Symfony's plugin model allows you to add templates and files into a plugin that can extend or override the existing defaults. That's how the custom themes typically work - they extend the base Dominion theme, overriding it with customized templates (all found within the plugin directory structure) only where needed, otherwise falling back to the system theme defaults. This makes upgrading with a custom theme easier, because normally all your changes are restricted to a single plugin, so it's easier to move to a new installation and reapply. I am attaching a PDF we give to clients who maintain their own installations but come to us to develop theme plugins - it covers plugin installation, but also how a theme plugin is generally organized, which may be of interest to you. 

In your specific case, I would recommend reviewing that list of edited files against the notes in the 2.7 docs for upgrading with a custom theme: 

• https://www.accesstomemory.org/en/docs/2.7/admin-manual/installation/upgrading/#upgrading-with-a-custom-theme-plugin
  

And check to see if any of those files need to be reviewed and possibly updated. At a glance, I can see a mention of indexSuccess.php files, and making sure that any render_value calls in those files are updated to render_value_html. I suspect it's probably fine, but it would be good to double-check that. I also didn't do a thorough check of all the files you listed vs all the files in the upgrade docs, so do a quick check to see if any other changes need to be reviewed. 

Hope this helps!

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/069c1c3b-d9be-48c0-872d-71d5e36be088n%40googlegroups.com.

[Dan Gillean]

Hi again Patrick, 

One more important thing worth mentioning, particularly if you are taking over maintenance of a custom themed AtoM site but are not the developer who created it: 

As you may have noticed from recent release notes and forum posts, we are in the process of upgrading AtoM's Bootstrap version. The default Dominion theme in release 2.7 and earlier is built using Bootstrap v2.3.2, a version that has been deprecated and unsupported for a number of years now - but breaking changes in newer additions made it difficult for us to upgrade. 

We are now finally tackling this upgrade process, but are doing it across multiple releases so that we give users with custom themes time to adapt. 

I've written at a bit more length about this in the forum recently, as part of this thread: 

• https://groups.google.com/g/ica-atom-users/c/GjS0Ullh5G8/m/8AABh32dAgAJ

The TL;DR version is: 

• Bootstrap 5 changes a LOT of things, and no longer uses LESS either
• This means that any custom themes that extend the current BS2 base Dominion theme will need to be upgraded to use BS5 instead
• There is already a new BS5 Dominion theme included in release 2.7 so you can see what it's like, and have your theme developers begin studying how it is different
• Our public demo site is also now running the new BS5 theme by default
• In 2.7, it was included to give theme devs a chance to start exploring, but was not tested as much as the default BS2 templates
• In 2.8, BS5 will be the focus of testing and the supported version. The BS2 theme and templates will still be present but deprecated, meaning they will not be tested as much
• In the 2.9 release, BS2 will be fully removed from AtoM - meaning community users with a BS2-based custom theme will need to have upgraded the theme BEFORE upgrading to 2.9, or their themes will be broken
• When 2.8 is released, our team will assemble some resources to help community users with BS2 themes upgrade them. In the meantime, developers can simply study the Bootstrap 5 documentation to learn about its changes, as well as study our BS5 Dominion theme to see how we have implemented those differences. 

So, be aware: your custom theme will be fine in 2.8, but beyond that, you will need to plan to upgrade the theme before upgrading AtoM. Keep an eye on our documentation and this forum for further information, especially once 2.8 is released. 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

[Patrick Goetz]

Hi Dan -

I think maybe the instructions above beginning with

• sudo apt install npm make

Are missing a step?  I tried this:

• apt update
• apt install make nodejs npm

• npm install -g "less@<4.0.0" n

• n stable

and then when I tried this step:

• npm install

I get an error message about a missing module:

root@catalog-dev:/usr/share/nginx/atom# npm install
node:internal/modules/cjs/loader:1080
  throw err;
  ^

Error: Cannot find module 'semver'
Require stack:
- /usr/share/npm/lib/utils/unsupported.js
- /usr/share/npm/bin/npm-cli.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:1077:15)
    at Module._load (node:internal/modules/cjs/loader:922:27)
    at Module.require (node:internal/modules/cjs/loader:1143:19)
    at require (node:internal/modules/cjs/helpers:119:18)
    at Object.<anonymous> (/usr/share/npm/lib/utils/unsupported.js:2:14)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
    at Module._load (node:internal/modules/cjs/loader:960:12)
    at Module.require (node:internal/modules/cjs/loader:1143:19) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/usr/share/npm/lib/utils/unsupported.js',
    '/usr/share/npm/bin/npm-cli.js'
  ]
}

Node.js v18.18.2

A little googling suggests that this step is missing?

• npm init

Subsequent steps seem to want a package.json file, which  isn't created until `npm init` is run. But in any case I got Less installed and recompiled all the relevant themes.  The custom theme didn't have a Makefile, so I just ran lessc by hand.  This didn't resolve the issue, and I'm not sure how the css files would be relevant to this error message anyway. The customizations haven't changed from the 2.6.4 install, so the already compiled css files should be the same, or no?

Since only some objects give a "PHP message: The component does not exist: "object", "clipboardButton"" while reading response header from upstream" error" when clicked, I'm wondering how one can track a slug through the code in order to find out what component is causing this error?

I will also try spinning up a version which includes only the current database and not the customized theme to see what happens there. If after upgrading sql, regenerating the slugs, recreating the nested set, clearing the cache, and rebuilding the index, if I get the same error, wouldn't that implicate the base code?

[Dan Gillean]

Hi Patrick, 

Ugh, that's strange and I'm sorry that happened. It seems like a couple things are going on, of which at least one of them I have now realized belatedly, as I just ran into this with another user recently. 

I don't totally understand how all these extra dependencies are interrelated, but: there are some things that we don't include in the tarball but are in our code repository, mainly because we precompile everything in the tarball and if all goes well, then these other files are not needed. In addition to lessc and npm, we also use Cypress to handle installing a number of other dependencies on install, so we don't have to store a bunch of external binaries in AtoM anymore and worry about keeping them up to date - again, these are preloaded in the tarball if I recall correctly. Perhaps npm runs cypress? Sorry that I am a bit fuzzy on the specifics right now. 

In any case, from the last time this arose, I have learned that there are 2 additional files that npm is looking for to confirm other dependencies. See this thread: 

• https://groups.google.com/g/ica-atom-users/c/GjS0Ullh5G8/m/XkfQsUq3BQAJ
  

Hopefully if you copy those files to your local install, it will resolve at least some of the issues. 

Regarding the error: we confirmed that your database version number is correct, but: did you also drop and recreate the database  BEFORE loading your sqldump as the upgrade docs suggest? If not, then it's possible that you still have some outdated database columns that are causing this issue. Once again, exactly how this triggers a PHP error is beyond my archivist knowledge base, but I have seen similar issues before that were caused by skipping this step, so if you're going to test a parallel installation, do not skip the drop and recreate per the Upgrade instructions and see if it makes a difference. 

I'll see if I can gather additional suggestions for you. 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/48d3b2ba-86b0-4296-859c-55ed02186ca4n%40googlegroups.com.

[Dan Gillean]

Hi again, 

No real further suggestions at this time, but some corrections and clarifications: 

The reason I have been forgetting to mention those two files when telling people how to recompile themes is that we didn't use to have them. In 2.7 I believe, we started using Composer (NOT Cypress as I incorrectly said last time, which is a browser testing suite), a PHP dependency manager, as a way of reducing the number of dependency binaries we need to store in AtoM directly, keep up to date, etc. Composer uses the dependencies listed in the packages and package-lock JSON files that I pointed to in my last message, and will fetch and install the latest stable versions of them during installation. This is not needed in the tarball, where we precompile everything, but IS needed when following Option 2 and installing from our code repository. It's a bit of a side effect that this also complicates the process for tarball installations that add a custom theme and need to recompile later - but things will be changing again with the new Bootstrap 5 theme(s). 

Normally, the npm init command is used to create the package.json and package-lock.json files. Those files are needed for the rest of the commands to run... but if you just run that init command, then it will create BLANK copies of those files, meaning that they will not list the needed dependencies inside. 

So: it's not so much a missing command, as the files themselves that were missing in your installation. And: we will be changing this to account for Bootstrap 5 with 2.8, where we use SASS instead of LESS. 

In the meantime, let us know how things are going and if you've managed to sort things out locally!

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

[Patrick Goetz]

Hi Dan -

Sorry for the delayed response, I'm only able to work on this project intermittently. Regarding dropping and rebuilding the database, for every installation I start with a pristine Ubuntu 2-.04 container and install everything from scratch, so the database is empty initially. I'm assuming the

php symfony tools:install step starts with the correct database schema.

As an experiment, I created a new 2.7.3 installation and transferred only the old database, but not the custom theme. I then ran php symfony tools:upgrade-sql, told it to keep the missing theme, but use the  arDominion theme instead.  After clearing the cache and rebuilding the index, this works perfectly (i.e. no more The component does not exist: "object", "clipboardButton" error.  This seems to clearly indicate the problem is in the custom theme somewhere.  I'm going to try to transfer the theme again, being careful to precisely repeat the previous command sequence to see if maybe this was all just user error somehow. Else I'll have to try to start debugging the custom theme...

[José Raddaoui]

Hi Patrick,

You are on the right track, AtoM 2.7 included a major change on how the clipboard is managed, which involved changes on any template using the clipboard button component:

 https://github.com/artefactual/atom/pull/1166

No need to look at the full changes, but they will show you all the templates that were modified to use the new component. Some of those templates are probably overwritten by your theme plugin and they are still using the previous implementation:

<?php echo get_component('object', 'clipboardButton', array('slug' => $io->slug, 'wide' => true)) ?>

So you could search for `get_component('object', 'clipboardButton'` within your plugin an replace it with the new component call:

<?php echo get_component('clipboard', 'button', array('slug' => $io->slug, 'wide' => true, 'type' => 'informationObject')) ?>

Look for the template you are modifying in that PR as the type is a new parameter and there may be other parameters needed, or the slug may be passed from a different variable.

Best regards,

Radda.

[Patrick Goetz]

Thanks, Radda!  That was precisely the issue.  I had tracked the problem down to 2 particular theme modifications:

   atom/plugins/sfIsaarPlugin/modules/sfIsaarPlugin/templates/indexSuccess.php
   atom/plugins/sfIsaarPlugin/modules/sfIsaarPlugin/templates/editSuccess.php

and sure enough,  indexSuccess.php contained the faulty clipboardButton code.  I modified it as per your suggestion and now everything is working.

Thanks Radda and Dan for your help with this issue.