AtoM 2.6.4 - Issues after migration (database view and elasticsearch)

[Carolina Sacramento]

Hello,

I hope everyone is healthy and well.

We're facing two problems that I couldn't find in the group's history:

1)  Database view corrupted:

We recently migrated our AtoM installation to 2.6.4 version (about a month)

The system works perfectly, but a few days ago we can't perform any maintenance services, such as dumping the database or even cleaning the application cache (by command 'php symfony cc'). Regarding clearing cache the following error is displayed:

Configuration file "/usr/share/nginx/atom/vendor/symfony/lib/config/config/settings.yml, /usr/share/nginx/atom/apps/qubit/config/settings.yml" does not have a registered handler

We've investigated the source of the problem, and it looks like the database view identified by 'new_view' has been corrupted for some reason (and we don't know why... maybe related to issue 2).

We can't even access new_view via MySQL console. When running mysqldump, the procedure is interrupted in new_view, failing to back up important system tables (see attachments).

As the system is running perfectly for the users (including data insertion), I'm wondering if there is a way to recover or regenerate this view (some command or specific orientation) without having to restore an old backup or redo the migration process.

2) Elasticsearch huge log files   

Elasticsearch is generating a huge log. In a few days, it allocates all the machine's storage. We observe it took up 64GB of the disk In ten days. Is there any configuration/guidance regarding this issue? 

Did anyone face these problems or can help us? 

Thanks in advance.

Best Regards,

Carolina Sacramento

Oswaldo Cruz Foundation, Brazil

[Dan Gillean]

Hi Carolina, 

Unfortunately the attachments you mention did not come through with your message - will you please try to send them again?

Regarding the handler issue - we've seen this once before recently, here: 

• https://groups.google.com/g/ica-atom-users/c/Iw0glcWaIDM/m/Pwk1oSxxBAAJ
  

Unfortunately at the time, we did not figure out the cause of the issue - the original poster solved it by reinstalling. I did offer a few suggestions that might be worth trying (here). However, what I might suggest is that you open up the configuration file at /usr/share/nginx/atom/apps/qubit/config/settings.yml and compare it to the default installation version included in our code repository here:

• https://github.com/artefactual/atom/blob/stable/2.6.x/vendor/symfony/lib/config/config/settings.yml
  

Let us know what you find. I'm wondering if perhaps some part of the installation process timed out (such as when configuring the web installer), and this left some of the config files corrupted. We could try copying the contents of the default file into the corrupted one at apps/quibit/config/settings.yml if they do not match, and tnen restarting services (PHP-FPM, nginx, MySQL, ES) and clearing the application cache (php symfony cc) to see if this helps. 

Another way you might try resolving this would be to: 

• Make sure you have a backup of your data outside of AtoM
• Re-run the web installer, by returning to [baseURL]/index.php/sfInstallPlugin/loadData, and working through the steps again, and ensure you enter the correct credentials

However, the new_view thing - that's not a table name or anything I'm familiar with in AtoM. Is this what you named your AtoM MySQL database? Is there anything else installed on this server?

Finally, the ES logging issue - something else I don't recall seeing before. What is in these logs? Perhaps there is a source error that is being repeated that we can try to resolve. 

Any other information you can provide about your installation will help us offer better suggestions, such as: 

• What is the full version number listed in Admin > Settings?
• Did you follow our recommended installation instructions (e.g. Ubuntu 18.04, PHP 7.2, Nginx, MySQL 8.0, etc)? If no, what changes have you made locally?
• What is the default installation culture of your site? (typically configured during installation, and found in the apps/qubit/config/settings.yml file)
• During the upgrade, do you recall if you remembered to drop and recreate the database before loading your SQL dump? Did you remember to run the upgrade task?
• As previously mentioned, if you can share what you are seeing in the logs would be helpful
• Anything else you think it would be helpful for us to know about your installation environment, or any actions that led to these errors, etc.

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/e9e58b0f-93a6-4118-a94a-d6e29f28a576n%40googlegroups.com.

[Carolina Sacramento]

Hi Dan,

Thank you very much for your answers and guidance.

Sorry for the attached files. I don't understand what happened... I thought I had attached them to the Google Group's "new message" form. Again, sorry.

I will try to answer your questions. If I forgot something, please let me know.

1) About configuration files:

The settings.yml file has the same content as the default installation file.

2) You commented: "However, the new_view thing - that's not a table name or anything I'm familiar with in AtoM. Is this what you named your AtoM MySQL database? Is there anything else installed on this server?"

We used "atom" as the database name (as suggested in the AtoM documentation) and we didn't change the database schema. Also, there are no other products or services installed on the server.

But we noticed that new_view was in our last atom installation too (version 2.3.0 - 138). Atom tables and views are listed in print-03.png.

The other attached files (print-01.png and print-02.png) show the impossibility of accessing new_view and the dupping procedure error, respectively.

We will try to migrate again, eliminating this specific view from the database. Can you tell us if there is another non-standard table or view listed in print-03.png?

We've used atom since it was called an icaatom. ICA-AtoM 1.1 was the first stable version used by us. Before that, we tested and customized ICA-AtoM 1.0.8 (beta). But we've abandoned customizations in newer versions.

We'll keep you informed about the new migration process (after discarding new_view)

3) You commented, "Finally, the ES logging issue - something else I don't recall seeing before. What is in these logs? Perhaps there is a source error that is being repeated that we can try to resolve"

Log files are very large - daily logs are about 2 GB. I put the last ones in Google Drive and I'll send the link privately. Is there a way to clear elasticsearch logs (or do you recommend a simple manual removal?)

4) You asked, "Did you follow our recommended installation instructions (eg Ubuntu 18.04, PHP 7.2, Nginx, MySQL 8.0, etc)?"
Yes, for sure!

5) You asked: "During the upgrade, do you remember to drop and recreate the database before loading your SQL dump? Did you remember to run the upgrade task?"

Yes, all the procedure in the Wiki has been done exactly it was listed.

Some information about our environment:

AtoM Version: 2.6.4 - 184
Version: Ubuntu 18.04.5
PHP: 7.2.24
MySQL: 8.0.24
NGINX: 1.14.0
Default_culture: pt_BR

Thank you again for your help.

[José Raddaoui]

Hi Carolina,

The database problem could be a permissions issue. Those two views are not part of AtoM (maybe from your customization) and it's possible that the user you created to access the DB doesn't have enough permission over them. If you are not using them I'd try to delete them with the root user, otherwise maybe giving enough permissions to the AtoM user does the trick.

I've checked the Elasticsearch logs and it seems to be a lot of recurrent queries being logged (more than 10 per second in some cases), for example ...

......

  "from" : 0,

  "size" : 30,

  "query" : {

    "bool" : {

      "must" : [

        {

          "term" : {

            "subjects.id" : {

              "value" : "32158,41278,35279,38846,41284,40123",

              "boost" : 1.0

            }

          }

        },

        {

          "terms" : {

            "places.id" : [

              "318269"

            ],

            "boost" : 1.0

          }

        },

        {

          "term" : {

            "publicationStatusId" : {

              "value" : 160,

              "boost" : 1.0

            }

          }

        }

      ],

      "disable_coord" : false,

      "adjust_pure_negative" : true,

      "boost" : 1.0

    }

......

 

This query is pretty similar to the one used from the terms index page (places, subjects, genres) and in fact it's made over different terms and taxonomies However, the term highlighted is causing the following error ...

Caused by: java.lang.NumberFormatException: For input string: "32158,41278,35279,38846,41284,40123"

While the AtoM 2.5 release introduced a regression that has not been fixed yet and may be the reason of the problem ...

https://projects.artefactual.com/issues/13231

My first impression is that those queries are not triggered manually from AtoM's GUI, os I wonder if you have a different process making them that you may need to fix due to that regression.

Best regards,

Radda.

[Carolina Sacramento]

Hi José,

Thank you for your comments and orientations.

We dropped the new_view with the root user and solved the mysqldump problem, but unfortunately the error in cleaning the cache remains. We'll follow Dan's suggestion and redo the migration process (actually, we decided to redo all the installation + migration procedures in another machine).

Regarding the elasticsearch's log, we thank you for the deep investigation but maybe is better to check if the problem persists after reinstalling rather than proceeding with further investigation. Do you agree?

We'll keep you updated about the final results.

Thanks so much again and have a nice weekend.

Best regards,

Carolina Sacramento
Oswaldo Cruz Foundation, Brazil

--

You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/2a7e135f-556f-44ad-a85e-7afa3b56a9cen%40googlegroups.com.

--

Carolina

[deepak chauhan]

did you @carolina get any solution for this. we currently upgraded AtoM to 2.7.3 and are getting elasticsearch issues : 

Caused by: java.lang.NumberFormatException: For input string: "198438,198437"

at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043) ~[?:?]

at sun.misc.FloatingDecimal.parseDouble(FloatingDecimal.java:110) ~[?:?]

at java.lang.Double.parseDouble(Double.java:538) ~[?:1.8.0_402]

this is filling our logs very fast. please let me know if you found any solution for this.

thank you