Re: [atom-users] Elasticsearch error with a '/' character in the search query

[Dan Gillean]

Hi Martin, 

The slash is a reserved character in Elasticsearch, so searching with it when it is not either in quotations or escaped will result in an Elasticsearch error. See:

• https://www.elastic.co/guide/en/elasticsearch/reference/5.6/query-dsl-query-string-query.html#_reserved_characters

Reserved characters

If you need to use any of the characters which function as operators in your query itself (and not as operators), then you should escape them with a leading backslash. For instance, to search for (1+1)=2, you would need to write your query as \(1\+1\)\=2.

The reserved characters are: + - = && || > < ! ( ) { } [ ] ^ " ~ * ? : \ /

Failing to escape these special characters correctly could lead to a syntax error which prevents your query from running.

< and > can’t be escaped at all. The only way to prevent them from attempting to create a range query is to remove them from the query string entirely.

If you would like to know more about what most of these special characters are reserved for - it's that they act as special search operators. See: 

• https://www.accesstomemory.org/docs/latest/user-manual/access-content/advanced-search/#boolean-operators

While the forward slash is not a boolean operator, it is apparently reserved by Elasticsearach for constructing regular expressions. 

One simple solution is to search for the phrase exactly - i.e. in quotations. Wood/tree may produce an error, but "Wood/tree" should not. 

Additionally, there is a setting that can also help to resolve this. See: 

• https://www.accesstomemory.org/docs/latest/user-manual/administer/settings/#escape-special-characters-from-searches

Please read the section carefully, including the part in the IMPORTANT admonition, to get a full picture of how this setting works, and its gotchas.

If you'd also like to take a deeper dive into how we explored the issue, and the search tests that were conducted before hitting on a configurable option that escapes the characters, you can read the details on the related issue ticket here: 

• https://projects.artefactual.com/issues/5850
  

Finally, note that another user has previously suggested a more permanent, code-based fix for this issue as well. We have not personally tested it (and the suggestion was for an older release when we used Elasticsearch 1.7, so there could be changes), but another forum user has and reported success. See:

• https://groups.google.com/d/msg/ica-atom-users/7BZ7xOkm7K4/1racvZ9sGAAJ

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

On Thu, Nov 26, 2020 at 3:44 PM Martin <m.k...@docuteam.ch> wrote:

A search with a '/' character in the search query does produce an elasticsearch error even on a empty AtoM installation with freshly generated search index, slugs and nested-set (see Vagrant file).

Elasticsearch error: Elastica\Exception\ResponseException

Steps to reproduce:

1. vagrant up

2. Type *wood/tree* in the search field (no error for *wood* or *tree*)

/var/log/nginx/error.log:

2020/11/26 17:22:17 [error] 648#648: *35 FastCGI sent in stderr: "PHP message: This request has been forwarded to a 404 error page by the action "search/autocomplete"" while reading response header from upstream, client: 10.0.2.2, server: _, request: "GET /search/autocomplete?query=wood%2Ftr&repos= HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.atom.sock:", host: "127.0.0.1:8000", referrer: "http://127.0.0.1:8000/"

Setup:

* AtoM version: 2.6.1

* Vagrant box: 2.6.0.2

* Vagrant version: 2.2.7

* OS: ubuntu 18.04

Vagrant file:

# -*- mode: ruby -*-

# vi: set ft=ruby :

Vagrant.configure("2") do |config|

  config.vm.box = "artefactual/atom"

  config.vm.box_version = "2.6.0.2"

  config.vm.network "forwarded_port", guest: 80, host: 8000, host_ip: "127.0.0.1"

  config.vm.provision "shell", inline: <<-SHELL

    cd /home/vagrant/atom

    git pull

    git checkout v2.6.1

    php symfony tools:upgrade-sql --no-confirmation

    php symfony propel:generate-slugs

    php symfony propel:build-nested-set

    php symfony search:populate

    php symfony cc

    systemctl restart nginx.service

    systemctl restart php7.2-fpm.service

    systemctl restart elasticsearch.service

    systemctl restart atom-worker.service

    systemctl restart memcached.service

  SHELL

end

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/ae9c8cb0-1e77-459a-88e6-4dc43822f4d4n%40googlegroups.com.

[Martin Kunz]

Dear Dan

Thank you very much for your quick reply. Sure I do understand now why I get this error.

Best regards

Martin

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CAC1FhZLHyCKKaKJwGtUK4_pEwAbLZw50qr2ue_m%3DqS60jdU3XQ%40mail.gmail.com.

--

Martin Kunz
Archivinformatiker

Docuteam GmbH
Informationsmanagement und Archivdienstleistungen
Im Langacker 16, CH-5405 Baden-Dättwil
T +41 56 511 11 48
www.docuteam.ch

Anwesenheit: Montag - Mittwoch

Digitale Archivierung in der Cloud: https://cosmos.docuteam.ch