Search not escaping certain characters, and metadata is lost when exiting editing mode by cancelling instead of saving

[Jimmy Thelander]

Hi,

The National Library of Sweden have had several problems since we upgraded to version 2.7.1 of Access to Memory. The problems seem to be the same for all the archival descriptions in the system and for all users.

1. The search functionality doesn’t allow certain characters which results in an ElasticSearch error, ”Elasticsearch error: Elastica\Exception\ResponseException”. This error seems to be present on https://demo.accesstomemory.org/ as well. It seems certain characters aren’t escaped properly.

2. Any metadata is lost on a certain level of description in an archival description if you enter Edit mode and then press Cancel instead of Save. The data isn’t regenerated if you repopulate the search index (php symfony search:populate), and seems to be missing in the database as well. 

3. When you enter Edit mode on any level of description where the tree view also contains other descriptions on the same level and then press Save, the level you are editing automatically moves down to the bottom of the structure. Sometimes it seems to “move back” to the original position in the tree view after a while but we have not come across it enough to know whether it goes back every time. Maybe repopulating the search index (php symfony search:populate) is the cause of it moving back to the original position.

Any ideas what could be causing these issues, and how we can proceed in resolving them?

We are running the following versions:

- Access to Memory 2.7.1 (upgraded from 2.6.3 I believe)

- nginx 1.23.3

- PHP 7.4.30

- ElasticSearch 5.6.16

- MySQL 8.0.30

Best regards,

Jimmy Thelander

National Library of Sweden

[Dan Gillean]

Hi Jimmy, 

Thank you for these reports. Some initial feedback below: 

1. The search functionality doesn’t allow certain characters which results in an ElasticSearch error, ”Elasticsearch error: Elastica\Exception\ResponseException”. This error seems to be present on https://demo.accesstomemory.org/ as well. It seems certain characters aren’t escaped properly.

The public demo site has not yet been upgraded from version 2.6.4, so what you are seeing is not something new in 2.7.x. In fact, it's possible that you are encountering a known limitation of Elasticsearch: there are reserved characters that cannot be used in searches without manually escaping them, because they are also operators for expert searching. See: 

• https://www.elastic.co/guide/en/elasticsearch/reference/5.6/query-dsl-query-string-query.html#_reserved_characters

From that link: 

If you need to use any of the characters which function as operators in your query itself (and not as operators), then you should escape them with a leading backslash. For instance, to search for (1+1)=2, you would need to write your query as \(1\+1\)\=2.

The reserved characters are: + - = && || > < ! ( ) { } [ ] ^ " ~ * ? : \ /

Failing to escape these special characters correctly could lead to a syntax error which prevents your query from running.

< and > can’t be escaped at all. The only way to prevent them from attempting to create a range query is to remove them from the query string entirely.

Note that we have added a setting in AtoM that allows you to automatically escape characters - useful for example if you use slash characters in your reference codes. See: 

• https://www.accesstomemory.org/docs/latest/user-manual/administer/settings/#escape-special-characters-from-searches
  

However, PLEASE READ THE "IMPORTANT" ADMONITION in this section! It's not a perfect solution, and it is therefore very important that administrators understand how searching works in AtoM before enabling the setting. 

Is this what you're encountering? Does this answer your questions about search errors, or are you encountering a different error? If something different, please provide more details - including step by step instructions on how we can reproduce the issue. 

2. Any metadata is lost on a certain level of description in an archival description if you enter Edit mode and then press Cancel instead of Save. The data isn’t regenerated if you repopulate the search index (php symfony search:populate), and seems to be missing in the database as well. 

Can you please provide more information on this, so we can attempt to reproduce it. Perhaps some exact steps. I tried going into edit mode on a few existing descriptions in a local 2.7.x test site and then canceling without making any changes, but could not reproduce this issue - all metadata was still present. Thanks in advance!

3. When you enter Edit mode on any level of description where the tree view also contains other descriptions on the same level and then press Save, the level you are editing automatically moves down to the bottom of the structure. Sometimes it seems to “move back” to the original position in the tree view after a while but we have not come across it enough to know whether it goes back every time. Maybe repopulating the search index (php symfony search:populate) is the cause of it moving back to the original position.

I believe that this is a known issue that our maintainers are currently prepping a fix for. See: 

• https://projects.artefactual.com/issues/13640
  

As you noted, and as the ticket also notes, running the search populate task should resolve this for now - the problem is the nested set and the search index getting out of sync. We hope to include a patch in the next release. 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/8b309b76-4661-4f15-afce-8b9ab19b214bn%40googlegroups.com.

[Jimmy Thelander]

Hi Dan,

Thank you for your input and your help! It's greatly appreciated!

1. I'll take a look at the options for escaping the searches, maybe it wasn't carried over when upgrading for one reason or another. 

2. I've been able to reproduce it by simply doing it over and over and over. Clicking the item in the tree view, clicking Edit, clicking Cancel, and repeat. It'll happen like 1 out of 30 times or similar (just an estimate) on our installation. I haven't been able to find anything in the logs regarding this either. Any idea where I could be investigating further?

3. Then we'll wait for the patch for this issue. :)

Thanks again!

Best regards,

Jimmy 

[Dan Gillean]

Hi again Jimmy, 

2. I've been able to reproduce it by simply doing it over and over and over. Clicking the item in the tree view, clicking Edit, clicking Cancel, and repeat. It'll happen like 1 out of 30 times or similar (just an estimate) on our installation. I haven't been able to find anything in the logs regarding this either. Any idea where I could be investigating further?

This sounds like a bit of a difficult-to-reproduce edge case, but two possibilities for more information:

• Perhaps you could try repeating the tests while using Debug mode to see if anything further is captured? See: Debug mode docs
• If an error has been captured from MySQL, then there might be something in /var/log/mysql/error.log to review. Note that if you've made configuration changes then the error log file name or location could be different - see the MySQL documentation for more. 

That's all I can think of for now! I will check to see if our team has any further suggestions. 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/feb830c2-c975-497a-8ecf-82e52b2bc7e6n%40googlegroups.com.

[Jimmy Thelander]

Hi again Dan,

There are no errors from MySQL, but I managed to capture the `UPDATE` query from when this happens, but I can't seem to figure out why our instance of Access to Memory sometimes (not always) does this. Here is the log:

UPDATE object SET `CLASS_NAME`='QubitInformationObject', `UPDATED_AT`='2023-03-13 10:00:00', `SERIAL_NUMBER`=1 WHERE object.ID=92028 AND object.SERIAL_NUMBER IS NULL
UPDATE information_object SET `IDENTIFIER`=NULL, `LEVEL_OF_DESCRIPTION_ID`=NULL, `REPOSITORY_ID`=NULL, `DESCRIPTION_STATUS_ID`=NULL, `DESCRIPTION_DETAIL_ID`=NULL, `DESCRIPTION_IDENTIFIER`=NULL, `SOURCE_STANDARD`='ISAD(G) 2nd edition', `DISPLAY_STANDARD_ID`=NULL WHERE information_object.ID=92028
UPDATE property SET `OBJECT_ID`=92028 WHERE property.ID=163908
UPDATE property_i18n SET `VALUE`='N;', `ID`=163908 WHERE property_i18n.ID=163908 AND property_i18n.CULTURE='sv'
UPDATE property SET `OBJECT_ID`=92028 WHERE property.ID=163909
UPDATE property_i18n SET `VALUE`='N;', `ID`=163909 WHERE property_i18n.ID=163909 AND property_i18n.CULTURE='sv'
UPDATE property SET `OBJECT_ID`=92028 WHERE property.ID=163910
UPDATE property_i18n SET `VALUE`='N;', `ID`=163910 WHERE property_i18n.ID=163910 AND property_i18n.CULTURE='sv'
UPDATE property SET `OBJECT_ID`=92028 WHERE property.ID=163911
UPDATE property_i18n SET `VALUE`='N;', `ID`=163911 WHERE property_i18n.ID=163911 AND property_i18n.CULTURE='sv'
UPDATE information_object_i18n SET `TITLE`=NULL, `EXTENT_AND_MEDIUM`=NULL, `ARCHIVAL_HISTORY`=NULL, `ACQUISITION`=NULL, `SCOPE_AND_CONTENT`=NULL, `APPRAISAL`=NULL, `ACCRUALS`=NULL, `ARRANGEMENT`=NULL, `ACCESS_CONDITIONS`=NULL, `REPRODUCTION_CONDITIONS`=NULL, `PHYSICAL_CHARACTERISTICS`=NULL, `FINDING_AIDS`=NULL, `LOCATION_OF_ORIGINALS`=NULL, `LOCATION_OF_COPIES`=NULL, `RELATED_UNITS_OF_DESCRIPTION`=NULL, `INSTITUTION_RESPONSIBLE_IDENTIFIER`=NULL, `RULES`=NULL, `SOURCES`=NULL, `REVISION_HISTORY`=NULL, `ID`=92028 WHERE information_object_i18n.ID=92028 AND information_object_i18n.CULTURE='sv'
UPDATE status SET `OBJECT_ID`=92028, `STATUS_ID`=160 WHERE status.ID=48597

It sets everything to NULL for the object in question when clicking Cancel. Any ideas on what causes this? I've tried comparing our test environment to our production environment, but I can't seem to find any differences in configuration. 

Best regards,

Jimmy 

[Dan Gillean]

Hi Jimmy, 

Good catch on getting that SQL log. Very interesting... that said, after consulting with our maintainers, we are also a bit stumped on this. One of our team tried to reproduce this issue locally over a hundred times, unsuccessfully. He also tells me that AtoM's cancel button work essentially the same as the Back button in your browser - it's just a hyperlink back to the previous page, and should not be triggering any SQL calls at all.

Can you please tell me a bit more about the NLBS installation? For example, are there any changes that you've made locally from the recommended installation instructions - different database, webserver, operating system or version, PHP version, etc? From what you posted at the start of this thread, it seems no - but best to just ask directly in case there's something I'm missing. 

What about local customization - does your site have a custom theme plugin, or any other custom code changes that might influence this behavior?

Can you tell me the full version number listed in Admin > Settings, so we can ensure that the database schema version (the second number after the dash) matches what is expected? For releases 2.7 and 2.7.1 this should be v192. 

In the meantime, our team isn't sure how to proceed with this if we can't reproduce it. Fortunately it does sound like an edge case. Until we get to the bottom of this, might I recommend that you ensure that you set up a process to capture nightly backups, so you can roll back in case of serious data loss resulting from this? Hopefully we can figure out a way forward before this impacts your users irrecoverably. 

Regards, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/4d354c9b-c200-4b60-9fea-6977483aa1adn%40googlegroups.com.

[Jimmy Thelander]

Hi again Dan,

It seems we managed to solve this, but haven't been able to figure out what actually solved it. It seemed to coincide with the bug that causes posts to "jump" to the bottom of the tree view and with some other issue we had with our atom-worker process not running properly. But more than that I do not know.

Thank you so much for your help!

Best regards,

Jimmy Thelander

National Library of Sweden

[Dan Gillean]

Very strange! 

I appreciate the update, Jimmy, and while it would have been good to understand what was causing this strange issue, I'm glad to hear that it's resolved! 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/c0c02e8e-3b8b-4426-9b74-13a68304ef6dn%40googlegroups.com.

[gomes...@gmail.com]

Hi Dan,

Regarding to Jimmy's feedback #1: "The search functionality doesn’t allow certain characters which results in an ElasticSearch error, ”Elasticsearch error: Elastica\Exception\ResponseException”. This error seems to be present on https://demo.accesstomemory.org/ as well. It seems certain characters aren’t escaped properly."

I had been using version 2.4.0 v156 until yesterday (the machine is still online). After upgrading to version 2.7.2 yesterday, I noticed that the ‘Escape special chars from searches’ option in Settings no longer works.  It works fine in version 2.4.x. To confirm this issue, I visited https://demo.accesstomemory.org/ which is already on version 2.7.1 and added ‘/’ to the ‘Escape special chars from searches’ option. However, it also doesn't work there. I can use double quotation marks to avoid the error, but few people will remember to do that. Is there a workaround for this?

Thank you for your support!

[Dan Gillean]

Hi Gomes (and Jimmy), 

Thanks for this report. I've now filed the following issue ticket for our Maintainers to investigate: 

• https://github.com/artefactual/atom/issues/1605
  

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/1998316f-da89-481e-ab35-f880870249f0n%40googlegroups.com.