query with dash is treated as negation

[Michael Masouras]

Hi, I am trying to run this query which contains two constraints and it's returning unexpected results: the document contains neither the text term nor the tag

"FT.SEARCH" "il1" "@title:v-storm @tags:{w\\:couchride\\:t\\:suzuki}" "LIMIT" "0" "1" RETURN 2 title tags

1) (integer) 172

2) "p:114625"

3) 1) "title"

   2) "2017 Harley-Davidson\xc2\xae VRSCF - V-Rod Muscle\xc2\xae"

   3) "tags"

   4) "t:english,t:unitedstates,w:couchride:t:dealer,w:couchride:t:used,t:image,w:couchride:t:cruisers,w:couchride:t:harley-davidson"

I suspect the "-" (dash) is throwing off the query parser:

FT.EXPLAIN "il1" "@title:v-storm @tags:{w\\:couchride\\:t\\:suzuki}" "LIMIT" "0" "1" RETURN 2 title tags

"INTERSECT {\n  @title:UNION {\n    @title:v\n    @title:+v(expanded)\n  }\n  NOT{\n    INTERSECT {\n      storm\n      TAG:@tags {\n        w:couchride:t:suzuki\n      }\n    }\n  }\n}\n"

I have tried lots of alternatives - how do I escape the dash in the query so it's treated as a regular character?

Thanks

Michael

[Meir Shpilraien]

Hi Michael,

Pls try to use escaping on the "-", i.e: "FT.SEARCH" "il1" "@title:v\\-storm @tags:{w\\:couchride\\:t\\:suzuki}" "LIMIT" "0" "1" RETURN 2 title tags

Let me know if it helped.

[Michael Masouras]

I tried that earlier, and it returned zero results. 

"FT.SEARCH" "il1" "@title:v\\-strom @tags:{w\\:couchride\\:t\\:suzuki}" "LIMIT" "0" "1" RETURN 2 title tags

1) (integer) 0

I think the double escaping works in terms of parsing the query correctly:

"FT.EXPLAIN" "il1" "@title:v\\-strom @tags:{w\\:couchride\\:t\\:suzuki}" "LIMIT" "0" "1" RETURN 2 title tags

"INTERSECT {\n  @title:UNION {\n    @title:v-strom\n    @title:+v-strom(expanded)\n  }\n  TAG:@tags {\n    w:couchride:t:suzuki\n  }\n}\n"

This is the same explanation I get if I specify a non-dashed word in the title field of the query.

I am suspecting the problem might be with the indexer that's not considering "v-strom" to be one word?

Michael

--
You received this message because you are subscribed to the Google Groups "redisearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redisearch+...@googlegroups.com.
To post to this group, send email to redis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/redisearch/161ca3c3-4511-464b-94b8-924ba8e7b525%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Meir Shpilraien]

I see the problem, on the indexing stage the v-storm was separate to 'v' and 'storm' you need escaping on the indexing stage also. See the following example:

127.0.0.1:6379> FT.ADD idx doc1 1.0 FIELDS text test-1

OK

127.0.0.1:6379> FT.SEARCH idx test-1

1) (integer) 0

127.0.0.1:6379> FT.ADD idx doc3 1.0 FIELDS text test\-1

OK

127.0.0.1:6379> FT.SEARCH idx test\-1

1) (integer) 1

2) "doc3"

3) 1) "text"

   2) "test\\-1"

The issue in this case is that the escaping appears in the value and I am not sure this is the expected behaviour (will open an issue on that).

[Michael Masouras]

Thanks for the explanation Meir. There are a couple of problems with this approach (as you note yourself):

a. It doesn't work with FT.ADDHASH. I now have to parse, escape and specify in the command every field I want to be indexed.

2. The returned text is "test\\-1", so I am guessing I will have to also un-escape escaped characters when I read the results back?

Having to escape and unescape the actual content is unexpected from the side of the developer - but escaping the query is a reasonable expectation since you need to cater for special characters in the query language. 

If you do open a bug, can you please paste the link here so I can follow? 

Thanks again

Michael

--

You received this message because you are subscribed to the Google Groups "redisearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redisearch+...@googlegroups.com.
To post to this group, send email to redis...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/redisearch/d2e72f1b-697c-4c92-9465-1260cb0e5a52%40googlegroups.com.

[Meir Shpilraien]

I do agree this is not best approach to handle this issue.

We talked about it today and we are planning to allow customising the tokenisation chars (and possibly allow more tokenisation methods), so you can tell the tokeniser not to split on '-'.

But this will only be possible in future releases. Until then you will have to settle with this suggested workaround.

[Itamar Haber]

Tracking issue: https://github.com/RedisLabsModules/RediSearch/issues/478

To unsubscribe from this group and stop receiving emails from it, send an email to redisearch+unsubscribe@googlegroups.com.

To post to this group, send email to redis...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/redisearch/6c4206f4-70c5-4356-ace1-ac252d424bf3%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Itamar Haber Technicalist Evangenly Phone: +972.54.567.9692

[David Snowberger]

Hi, it looks like this issue may not have seen any movement and I wanted to see what plan there may be and whether my understanding is correct. So just to clarify, there is currently no way to search for hyphenated values unless the hyphens have been escaped in the data because the tokenization does not include adding the complete string as a term, and there is also not a way to specify that hyphens should not be separators including during index creation time - is that correct?

So given:

127.0.0.1:20000> ft.add index doc1 1.0 fields field1 some-value

OK

127.0.0.1:20000> hgetall doc1

1) "field1"

2) "some-value"

It is impossible to match on "some-value", regardless of query escaping.

Is it on the agenda to change how this works or is the recommendation still to add escaping to the data, and then do escaping in the query as well?

[Mark Nunberg]

In the future we may add options to ignore certain characters as separators and treat them as actual terms. It is a non-trivial feature to add, however, but is most definitely in the roadmap

-- 
You received this message because you are subscribed to the Google Groups "RediSearch Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to redisearch+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/redisearch/04572b0e-c8f6-42a8-8c66-10bb68b2c534%40googlegroups.com.

[David Snowberger]

Hey Mark ok, thank you for the update.

On Thursday, October 17, 2019 at 1:22:43 AM UTC-7, Mark Nunberg wrote:

In the future we may add options to ignore certain characters as separators and treat them as actual terms. It is a non-trivial feature to add, however, but is most definitely in the roadmap

To unsubscribe from this group and stop receiving emails from it, send an email to redis...@googlegroups.com.