Re: Characters like ',[,],! and etc. => "syntax error, unexpected $end near"

[Alexander Sologub]

Forgot to mention - Sphinx returns errors (and Thinking Sphinx firing exceptions) when search term consists of separators ONLY.

Example:

@some_field  ![]''

вторник, 25 декабря 2012 г., 16:56:07 UTC+3 пользователь Alexander Sologub написал:

Hi there!

We are using Thinking Sphinx for fulltext search in our project, but now we face that nifty problem.

As I red in Sphinx documentation, characters missing in charset_table should be treated like separators.

Also, I call Riddle.escape('my_gorgeous_request') before calling ::search method on my models.

But I don't get expected result (at least empty one). I end up with ThinkingSphinx::SphinxError.

So, the question is - How can I solve this?

Should I add all undesired symbols to charset_table? But it's kind of stupid to copy all UTF-8 table there...

Maybe there's some kind of configuration option I don't know about (for Sphinx itself?)

Or maybe, I'm doing something wrong?

[Pat Allan]

Hi Alexander

Can you share the controller action that's making this search call? I'd like to get the full context, which may provide some clues as to why you're getting this error.

Cheers

--
Pat

> --
> You received this message because you are subscribed to the Google Groups "Thinking Sphinx" group.
> To view this discussion on the web visit https://groups.google.com/d/msg/thinking-sphinx/-/HdceEdphtfIJ.
> To post to this group, send email to thinkin...@googlegroups.com.
> To unsubscribe from this group, send email to thinking-sphi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/thinking-sphinx?hl=en.

[Alexander Sologub]

Sure, here's the code:

    search_params = get_location_filter.present? ? { conditions: { near: get_location_filter } } : {}

    search_params[:with] = { category_ids: get_category_filter } if get_category_filter.present?

    @events = Event.search(search_params).sx_limit(7)

    render

Also, here's declaration of sx_limit scope:

sphinx_scope(:sx_limit) { |limit| { limit: limit } }

get_location_filter helper just returns unescaped query from user input.

Also, here's declaration of :near field:

indexes [location.country, location.city, location.country_full_name], as: :near

среда, 26 декабря 2012 г., 16:13:18 UTC+3 пользователь Pat Allan написал:

[Pat Allan]

I think the get_location_filter is the important part, as that's providing the query text, which is what Sphinx is complaining about.

-- 

Pat

To view this discussion on the web visit https://groups.google.com/d/msg/thinking-sphinx/-/Umh7IOqKangJ.

[Alexander Sologub]

Here's the code:

  def get_location_filter

      cookies[:event_location_near_filter]

  end

I mean, it returns simple string from user input. It's not Sphinx query.

Examples: 

1) 'Moscow', 'London' - they don't cause exceptions at all.

2) '\\Moscow!@$'' (or any combination of unknown symbols + at least one symbol from charset_table) - still no exceptions.

3) ' #$%^' - input contains unknown symbols only - ThinkingShpinxError => (Sphinx itself returns 'unexpected $end near' first symbol of this field).

I used "binding.pry" inside of ThinkingSphinx source code to determine conditions of exception firing.

Query itself looks like this: (for user input @@$%!%!) 

"@near \\@\\@\\$%\\!%\\!"

As i said before, I've used Riddle.escape to escape Sphinx-specific symbols.

I tried to enter some scary letters from UTF-8 table - they give the same result:

If query consists from unknown characters only (not listed in charset_table) then exception fires.

четверг, 27 декабря 2012 г., 4:17:51 UTC+3 пользователь Pat Allan написал:

[Pat Allan]

Wait, so where is Riddle.escape being used?

> To view this discussion on the web visit https://groups.google.com/d/msg/thinking-sphinx/-/uimBqDRXVBEJ.

[Alexander Sologub]

def conditions_as_query

      return '' if @options[:conditions].blank?

      ' ' + @options[:conditions].keys.collect { |key| "@#{key} #{Riddle.escape(options[:conditions][key]})"}.join(' ')

end

It's in search.rb file (lib/thinking_sphinx/search.rb) in ThinkingSphinx source files (I forked this gem and made these modifications).

However, it's the way I tried to fix problem described above. I get errors in both clean and modified implementations.

четверг, 27 декабря 2012 г., 16:11:18 UTC+3 пользователь Pat Allan написал:

[Pat Allan]

So, if you add a normal character in with all of these special characters, and run it all through Riddle.escape, it behaves correctly?

> To view this discussion on the web visit https://groups.google.com/d/msg/thinking-sphinx/-/-7OqpwNOO2sJ.

[Alexander Sologub]

Yes, the only case it throws exception: query with separators only.

пятница, 28 декабря 2012 г., 9:48:07 UTC+3 пользователь Pat Allan написал:

[Pat Allan]

Okay, so this is an issue with Sphinx itself. I think your options are as follows:

* Don't stress about this bug. What are the odds of it actually occurring?
* Or: Check for queries that are only special characters, and change them to empty strings?
* Or: Add a rescue handler for these errors, and retry with an empty string instead. This could hide other errors though.

I would recommend one of the first two options.

Cheers

--
Pat

> To view this discussion on the web visit https://groups.google.com/d/msg/thinking-sphinx/-/xpYFzeFPSgUJ.

[Alexander Sologub]

Agreed, I've already implemented second option.

Since it's Sphinx problem, i'm not sure that pull request to Thinking Sphinx will be appropriate.

However, I can do it if you're interested =)

Thanks

пятница, 28 декабря 2012 г., 13:14:18 UTC+3 пользователь Pat Allan написал:

[Pat Allan]

I think this is a rare enough situation that a patch isn't required. The offer is appreciated though :)

> To view this discussion on the web visit https://groups.google.com/d/msg/thinking-sphinx/-/tSYQ3CJZ2KEJ.