Riddle::ResponseError from crazy search engine bots

[agibralter]

My exception notification app is going crazy with
Riddle::ResponseError from eager search engine bots querying for
random pages... I guess googlebot, msnbot, etc. all parse out page
numbers in query strings (page=...). Is there a good way to deal with
this? Could Riddle have a Riddle::OutOfBounds exception class so that
I could tell hoptoad to ignore them? I don't want to tell hoptoad to
ignore all Riddle::ResponseErrors because other types of
ResponseErrors might actually be problematic and require attention.

"Riddle::ResponseError: searchd error (status: 1): offset out of
bounds (offset=1470, max_matches=1000)"

[James Healy]

I have a before_filter in my ApplicationController that sets the page
param to 50 if it is set to greater than 50. It's a little hacky to have
this ine very request I suppose, but it's effective at controlling the
issue.

As an alternative you could get it to return an instant 404 if param is
more than 50.

-- James Healy <ji...@deefa.com> Sun, 21 Feb 2010 17:32:36 +1100

[agibralter]

Hmm well it also depends on per_page... I guess I could check if page
* per_page > 1000 in my app, but then what happens if I change the
max_matches... I feel like this might be better handled in TS or
Riddle by throwing a more specific exception. Then apps could handle
that exception however they want: e.g. rescue_from
Riddle::OutOfBounds ... => return 404, etc.

[Pat Allan]

Hi Aaron

I like the idea of Riddle::OutOfBounds - could you create a ticket on Riddle's GitHub project for this, so I don't forget to implement it?

Cheers

--
Pat

> --
> You received this message because you are subscribed to the Google Groups "Thinking Sphinx" group.
> To post to this group, send email to thinkin...@googlegroups.com.
> To unsubscribe from this group, send email to thinking-sphi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/thinking-sphinx?hl=en.
>

[agibralter]

Cool -- done. 1st ticket! :)

Thanks,
Aaron