why text_en doesn't work on type : "vegetable" in arangosearch example

[Muhammad Davatgar]

i was doing the example provided in the first page of arangosearch in manual of arangodb which is we have a set of documents about foods with two fields : name and type in the aql query when i set search analyzer(doc.type == "fruit" , "text_en") it just works fine , but when i change the value from "fruit" to be "vegetable" it just returns an empty array , text_en is set in indexing time and it is showing it in the configuration of the view , it only works if i change the value back to fruit or use "identity" instead of "text_en" why is that ? any solutions ? link to the manual : https://www.arangodb.com/docs/stable/arangosearch.html#search-expressions-with-arangosearch-functions

[Simran Spiller]

The text_en Analyzer has stemming enabled. You can observe its behavior by running the following AQL query:

RETURN TOKENS("vegetable", "text_en")

You will see that it stems the word to "veget". Your query compares the stored value "veget" to the string "vegetable", which will not match the document.

You should get a match if you use ANALYZER(doc.type == "veget", "text_en"). If not, then verify that the type field is actually indexed with the text_en Analyzer in the View definition.

To apply the same transformation to the search phrase as to the stored value, you can do:

ANALYZER(doc.type == TOKENS("vegetable", "text_en")[0], "text_en")

Also see https://www.arangodb.com/docs/stable/arangosearch.html#searching-for-tokens-from-full-text

Note that ANALYZER() specifies the Analyzer for doc.type but doesn't affect TOKENS().

To search for more than the first token from the search phrase (AND-combined), you can use:

 ANALYZER(TOKENS("healthy vegetable", "text_en") ALL IN doc.type, "text_en")

Also see https://www.arangodb.com/docs/stable/arangosearch-fulltext-token-search.html

You may also create a custom text Analyzer with stemming disabled, if that fits your use case better. See the examples of https://www.arangodb.com/docs/stable/analyzers.html#text