No results in Admin->Description update (v.2.3.1)

[elizabet...@mcgill.ca]

Hello All,

We are testing out our new installation of AtoM (2.3.1, upgrade from 2.3.0) and I am trying to understand how results in the Description update option of the Admin menu are generated.

We currently have about 200 archival descriptions in the system, several archival institutions, and a couple of authority records, most of which were added over the past week, but no results are generated when using the 'Description update' menu item. I have tried a number of search options but the list beneath "Browse newest additions" remains steadfastly empty. There are no errors in the web log and no errors on the page itself. Is there a critical data field which we are not populating, or a background job which must be activated ?

Many thanks.

[Dan Gillean]

Hi Elizabeth,

Strange. There is nothing that should have changed with the Description updates module from version 2.3.0 to 2.3.1 - in our minor releases we tend to only include bug fixes and the like.

It is possible that the default time parameters on the display page are just outside of when the last data was updated - can you try using the date range filter on the page to broaden the scope to a whole month or more, to see if data is in fact displayed? I know I regularly encounter this when loading the sqldump of demo data for local testing - because the most of the sqldump's creation/modification dates haven'tt changed since at least 2015 when we last updated our sample data set, I have to remember to add a broader time range in the date range search to actually see results with that data on the Description updates page.

I hope that's what it is! Let me know if not and I will try to reproduce the issue on my end.

Cheers,

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-users+unsubscribe@googlegroups.com.
To post to this group, send email to ica-atom-users@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/bd7f543a-781a-4fc4-9749-d5a6edf98d13%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[elizabet...@mcgill.ca]

Hi Dan,

Many thanks for the suggestion.  Unfortunately, I see no input options for changing date ranges.  I attach a screenshot of what we see. 

Thanks 

On Wednesday, 12 April 2017 18:18:22 UTC-4, Dan Gillean wrote:

Hi Elizabeth,

Strange. There is nothing that should have changed with the Description updates module from version 2.3.0 to 2.3.1 - in our minor releases we tend to only include bug fixes and the like.

It is possible that the default time parameters on the display page are just outside of when the last data was updated - can you try using the date range filter on the page to broaden the scope to a whole month or more, to see if data is in fact displayed? I know I regularly encounter this when loading the sqldump of demo data for local testing - because the most of the sqldump's creation/modification dates haven'tt changed since at least 2015 when we last updated our sample data set, I have to remember to add a broader time range in the date range search to actually see results with that data on the Description updates page.

I hope that's what it is! Let me know if not and I will try to reproduce the issue on my end.

Cheers,

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

On Wed, Apr 12, 2017 at 5:07 PM, <elizabet...@mcgill.ca> wrote:

Hello All,

We are testing out our new installation of AtoM (2.3.1, upgrade from 2.3.0) and I am trying to understand how results in the Description update option of the Admin menu are generated.

We currently have about 200 archival descriptions in the system, several archival institutions, and a couple of authority records, most of which were added over the past week, but no results are generated when using the 'Description update' menu item. I have tried a number of search options but the list beneath "Browse newest additions" remains steadfastly empty. There are no errors in the web log and no errors on the page itself. Is there a critical data field which we are not populating, or a background job which must be activated ?

Many thanks.

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To post to this group, send email to ica-ato...@googlegroups.com.

[Dan Gillean]

Hi Elizabeth,

My apologies! I spend so much time working in our development environments over the course of the year between our major releases that sometimes I forget which release new features are added in. In our upcoming 2.4 release we have overhauled the Description updates page and added a Date range search widget. There is some draft new documentation, with a screenshot, in the 2.4 manual, if you are curious:

• https://www.accesstomemory.org/docs/2.4/user-manual/administer/search-updates/

Have you tried creating a test record and then checking the Updates page to see if it shows up? It would be good to establish whether the whole page is broken, or whether it is just specifically records prior to the upgrade that are not displaying. I couldn't tell from your original post if the new records were created since the upgrade or not.

There shouldn't be any special setting you need to turn on to see results here. We use MySQL's auto-populating timestamp fields to capture creation and modification dates I believe. While these shouldn't be lost during an upgrade, I suppose it is possible? If you are creating new records *since* the upgrade and still not seeing anything in the page, that is more concerning. I will try to reproduce in my local 2.3 test instance and get back to you, and see if our developers have any theories.

Regards,

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-users+unsubscribe@googlegroups.com.
To post to this group, send email to ica-atom-users@googlegroups.com.

Visit this group at https://groups.google.com/group/ica-atom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/673e0c22-b1b6-486d-a4b0-dedaa0f2ec00%40googlegroups.com.

[Elizabeth Thomson]

Hi Dan,

Thanks for the reassurances re the date range widget. So no worries there !

Regarding new records : almost all records in the system have been added within the past two weeks.  Our very first install was version 2.3.0 (about a month ago) after which we did a migration of development data from a development 2.3.0 installation which contained only user account information.  Following the migration of user accounts we did an upgrade to 2.3.1.  That was done about a month ago.  Since doing the upgrade we have been adding records to the system, with most activity happening within the past two weeks.  

I should perhaps mention a couple things. First, many - but not all - of the archival descriptions were created by CSV import and have no corresponding accession record. Second, we discovered after doing the import- when trying to do an update publication status with update descendants - that gearmand had been incorrectly configured and no atom-worker was running. This was fixed a week ago and the atom worker is available. I created a new archival description object just now, but the Description updates list remains empty.

[Dan Gillean]

Hi again Elizabeth,

Thanks for this additional context. Based on your other post in the forum I have also learned that you have an installation that is not based on our recommended install instructions (using RHEL 7.3 instead), and so I will have more questions about that, as I am beginning to worry that there is some installation/configuration issues affecting your AtoM site in these subtle ways.

I have just tested in my local 2.3 vagrant box, and for me, the Description updates module is working as expected - I also had a developer test in his local instance and he reported the same - so I don't think we're looking at a general bug in AtoM here. Noticing that you are at McGill, I also wondered if perhaps it was an internationalization issue - perhaps your installation culture was different than the culture you were creating records in and there was a bug in AtoM affecting this - but I managed to make French descriptions in my English-language-installation site, and see them show up in Description updates when viewed in both French and English, so culture fallback appears to be working fine as well.

I also wondered if it was an indexing isssue - but a) it sounds like you are seeing your results when importing/creating them, so they are getting indexed, b) I tried re-indexing in my local test site (and our developer manually deleted his index then re-indexed) and my Description updates results were still there after, and c) it seems that, prior to our upcoming 2.4 release, many of the queries on the description updates page use SQL rather than the index (see issue #10458).

So: I have a few theories at the moment. Both seem like a bit of a stretch even to me, but in working through the steps and in answering the questions I will pose, we will at least be eliminating some possibilities.

The first is that this is somehow caused by data corruption - perhaps due to timeouts during the CSV import process. Are you importing these CSV files via the User interface? If so, how many rows are they? Have you ever had an import time out mid-process before completing? If so, did you do any follow-up after that, or just try re-importing again?

In 2.3.x and earlier, imports via the user interface are performed synchronously via the browser, so a large import can time out, which leaves partial rows in the database, and therefore some corrupt data that can affect your installation in strange ways that sometimes take a while to manifest. Theory 1 is that this is somehow messing with your database, preventing the date/time stamps from either being recorded, or being read properly. Even if not the cause of this particular issue, it is something to watch out for. Here's a couple posts with some starting place tips on resolving data corruption issues

• https://groups.google.com/d/msg/ica-atom-users/peScbebX2xM/An39IPA-AwAJ
• https://groups.google.com/d/msg/ica-atom-users/ZElzuHT_g40/z-XFsDFIFAAJ
  

Next theory: I'm wondering if this might somehow be related to your installation environment. You mentioned that you have installed on RHEL 7.3 - Can you find out more particulars about your installation environment? For example:

• Database type and version (RHEL will install mariaDB out of the box I think; our instructions recommend percona 5.6)
• PHP Version
• Elasticsearch version
• What webserver are you using? Nginx? Apache? Something else?
• Did you install from the tarball available on our Downloads page, for from our GitHub code repository?
• Are you certain that all required PHP extensions and additional libraries and dependencies outlined in our docs were installed? If some aren't available on RHEL or will install different versions by default, this could cause issues
• Any other changes we should know about?

Finally, have you tried any of the basic maintenance tasks that we generally recommend when first encountering issues? Such as:

• Repopulate the search index: php symfony search:populate
• Restart your webserver, PHP-FPM - instructions will depend on your particular environment
• Clear the application cache: php symfony cc
• etc (see also the commands in the linked threads about data corruption)

Do any of these tasks fail? If so, how/what do they say?

Sorry to throw a wall of ideas at you. I've not seen this before, and figure we should just work through some of the common catch-all issues and eliminate them as possibilities if possible - and beyond that, we'll need more information about your particular environment to be able to offer further suggestions.

Regards,

 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-users+unsubscribe@googlegroups.com.

To post to this group, send email to ica-ato...@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/9e62f82f-9e93-4b53-89ba-f055f934651c%40googlegroups.com.

[Elizabeth Thomson]

Hi Dan,

Many thanks for the follow up and links.  To answer your questions in order:

Our default culture is set to 'en' and certainly all the records in the CSV import had a value of 'en' in the culture column.

The CSV import was done on the command line and there were no timeouts or errors reported. The import took under 5 seconds to import 198 records if I recall correctly.

The import was followed by :

php symfony cc

sudo php symfony search:populate  (I've run those two commands a couple of times since then).

Our packages/versions are:

mariaDB 5.5.52

PHP 5.5.38

elasticsearch 1.7.3-1

nginx 1.10.2

I don't have full installation details, but I know that the installation was done from the tarball and deployed by ansible.

Re your first link above, I have run php symfony search:populate a couple of times and also php symfony propel:build-nested-set

I ran the SQL suggested in your second link above (re missing objects), but there were no empty columns in the result.

Possibly there are permissions issues involved here: If I run php symfony search:populate without sudo I get complaints about being unable to open the qubit_cli.log. 

I won't be around next week, so I will try and get you more details when I come back.  Thanks indeed for your help on this!

[Dan Gillean]

Hi Elizabeth,

Many thanks for these further details. It is possible that there is a permissions issue - perhaps this is preventing the date/time stamps from being properly written, or read?

All of AtoM's subdirectories and files should, following our recommended installation instructions, belong to the www-data user. You can try resetting the permissions in your application with the following command, run from AtoM's root directory:

• sudo chown -R www-data:www-data /usr/share/nginx/atom

You might try that, and then try repeating a test to see if new descriptions start showing up on the Updates page.

Given the history of your site and your further information about the CSV imports, it does sound unlikely that this is related to data corruption - at least that is one big mess we don't have to dive deeper into for the time being.

I'll wait to see what else you discover, and see if I can garner further ideas from our team.

Regards,

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-users+unsubscribe@googlegroups.com.
To post to this group, send email to ica-atom-users@googlegroups.com.

Visit this group at https://groups.google.com/group/ica-atom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/803cc274-c7cc-4063-9479-60c129bfc8ff%40googlegroups.com.

[Elizabeth Thomson]

Hi Dan,

Sorry for the delay in getting back to you. We are still trying to get a couple of overlooked packages installed.

Here is what I have found out so far:

PHP APCu was not enabled - it has been now.

We still trying to get lessc libsaxon-java and ffmpeg installed.

I have checked permissions for the directory tree under atom, and everything owned by user and group nginx (the user and group defined in atom.conf in our installation)

I'll let you know how it goes once the three packages have been installed.

One other thing.  I notice that 'systemctl status atom-worker' invariably reports the atom-worker as being active but exited. Is this normal behaviour ?

Thanks indeed for all your help so far.

Best regards

[Elizabeth Thomson]

Just documenting, for completeness sake, we discovered that the issue was Maria db handling of timestamps on RHEL7.  Ref: https://groups.google.com/forum/#!topic/ica-atom-users/RVmPOQDEEPM

We've now installed mysql and the problem has gone away.

[Dan Gillean]

Interesting! Thanks so much for posting this update, Elizabeth. I'm glad you've gotten everything working! 

Regards, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-users+unsubscribe@googlegroups.com.
To post to this group, send email to ica-atom-users@googlegroups.com.

Visit this group at https://groups.google.com/group/ica-atom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/58e3845f-4716-4cf9-b2c7-83c5d3140f44%40googlegroups.com.