pagination issue with memcached

[badnaam]

I memcached (with Dalli client) some of my search results like..

def self.city_search city, per_page, page, order
Rails.cache.fetch("#{city}_#{page}_#{order}") do
Post.search(:include => [:poster, :category],:geo =>
location_lat_lng_radian(city),
:with => {"@geodist" => 0.0..(Constants::PROXIMITY *
Constants::METERS_PER_MILE), :status => Post::STATUS['approved']},
:latitude_attr => :lat, :longitude_attr
=> :lng, :per_page => per_page, :page => page, :order =>
(ModelHelpers.determine_order_search order))
end
end

##In controller
@search_results = Search.city_search params[:city], params[:per_page]
|| Constants::LISTINGS_PER_PAGE, params[:page] || 1, params[:order] ||
'distance'

The result is presented in the view with pagination like..

<%= page_entries_info @search_results, :entry_name => "Search Result"
%>
<%= will_paginate @search_results, :class => 'pagination' %>

The first hit i.e. when the block is exececuted and the return type of
the collection is "ThinkingSphinx::Search" this all works fine. The
2nd time when there is a cache hit and the collection returned from
the city_search method is of type "Array" I get the following error.

If I convert the results of the block to an array so that the view
always get an Array (regardless of cached or non cached hit), the
first hit i.e. non-cached hit, get the same exact error.

Will paginate's page_entires_info expect a collection..so why am I
seeing this? Not sure if this is a will paginate or thinkingsphinx
issue.

undefined method `total_pages' for #<Array:0xa068c40>
/usr/local/lib/ruby/gems/1.8/gems/will_paginate-2.3.14/lib/
will_paginate/view_helpers.rb:171:in `page_entries_info'
/home/username/Apps/myapp/app/views/searches/_search_results.html.erb:
7:in
`_run_erb_app47views47searches47_search_results46html46erb_locals_object_paginate_search_results'
/home/username/Apps/myapp/app/views/searches/index.html.erb:4:in
`_run_erb_app47views47searches47index46html46erb'
/home/username/Apps/myapp/app/controllers/searches_controller.rb:28:in
`index'

[badnaam]

Update

OK, the issue is related to the face that the collection needs to be
of type Will_Paginte::Collection. TS Search by default returns the
results of this type, that why it works when the block is executed,
but memcached returns just an array and hence it doesn't work.

I tried converting the memcached Array to a will_paginate:collection
type like...

if results.class == ThinkingSphinx::Search
return results
else
pag_results = WillPaginate::Collection.create(page,
per_page) do |pager|
pager.replace(results)
unless pager.total_entries
# the pager didn't manage to guess the total
count, do it manually
pager.total_entries = results.count ##this is bad
end

end
return pag_results
end

But the problem is results does not contain the entire search count,
only the first paginated lot and without doing the search all again, I
can't calculate the exact count, which would defeat the purpose of
memcaching it, since everytime there is a cache hit I would have to
run the search again just to find the total count.

Bump!

[Jim Ruther Nill]

I think there's no way around this. memcache the object and memcache the total count.

--
You received this message because you are subscribed to the Google Groups "Thinking Sphinx" group.
To post to this group, send email to thinkin...@googlegroups.com.
To unsubscribe from this group, send email to thinking-sphi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/thinking-sphinx?hl=en.

--
-------------------------------------------------------------
visit my blog at http://jimlabs.heroku.com

[Pat Allan]

Hmm, this is odd. I think memcached calls freeze on objects it caches... but that shouldn't convert the search to an array (although it does freeze the underlying array).

Maybe Jim's suggestion is the best approach as a quick fix? Cache both the results and the total? Although it won't fix pagination - because WillPaginate expects a bit more information than that...

If you want to debug further, I'd start by seeing what memcached does when you cache an object.

--
Pat

[badnaam]

I asked the creator of the dalli gem (the memcache client) that I am
using. According to him "*Some things cannot be marshalled, like
sockets or cursors, and it sounds like TS is performing lazy queries
so it's not getting the total result count unless you actually ask for
it."

Is this what's going on here?

Thanks

On Sep 21, 6:23 am, Pat Allan <p...@freelancing-gods.com> wrote:
> Hmm, this is odd. I think memcached calls freeze on objects it caches... but that shouldn't convert the search to an array (although it does freeze the underlying array).
>
> Maybe Jim's suggestion is the best approach as a quick fix? Cache both the results and the total? Although it won't fix pagination - because WillPaginate expects a bit more information than that...
>
> If you want to debug further, I'd start by seeing what memcached does when you cache an object.
>
> --
> Pat
>
> On 21/09/2010, at 12:06 PM, Jim Ruther Nill wrote:
>
> > I think there's no way around this. memcache the object and memcache the total count.
>

> > For more options, visit this group athttp://groups.google.com/group/thinking-sphinx?hl=en.
>
> > --
> > -------------------------------------------------------------
> > visit my blog athttp://jimlabs.heroku.com

[Pat Allan]

I'm not sure about the logic there, but perhaps it's related - and yes, TS does perform lazy queries by default (this is necessary for sphinx scopes).

If you don't want the queries lazily evaluated, pass :populate => true in your search request, see if that helps with the caching.

Cheers

--
Pat

[badnaam]

Pat - no that didn't really make any difference, I still get an array
back and any "total_entries", total_pages type calls still return
undefined method errors.

[badnaam]

somomething like this..

Rails.cache.fetch("#{city}_#{page}_#{order}") do

Post.search(:populate => true, :include => [:poster, :category],:geo

=> location_lat_lng_radian(city),
:with => {"@geodist" => 0.0..(Constants::PROXIMITY *
Constants::METERS_PER_MILE), :status => Post::STATUS[:approved]},
:latitude_attr => :lat, :longitude_attr
=> :lng, :per_page => per_page, :page => page, :order =>
(ModelHelpers.determine_order_search order))
end

On Sep 21, 5:10 pm, Pat Allan <p...@freelancing-gods.com> wrote:

[badnaam]

any thoughts on this?

[Pat Allan]

Sorry, not yet... I was running an event this past weekend, so have been struggling to keep up with the list. I think I'll need to look at Dalli to see how it's handling objects when caching, and see why it's not storing the ThinkingSphinx::Search object...

--
Pat

[Ben Greenberg]

Anyone come up with a more elegant solution to this problem? It's still an issue with Dalli and built-in memcache libraries.

[Pat Allan]

Hi Ben

Last time I tried to reproduce this, I couldn't get any errors to occur. If you're having it happen regularly (all the time?), then perhaps create a sample application with the minimal setup required to get the error happening - that'd certainly make it easier for me to debug and fix.

Cheers

--
Pat

> --
> You received this message because you are subscribed to the Google Groups "Thinking Sphinx" group.

> To view this discussion on the web visit https://groups.google.com/d/msg/thinking-sphinx/-/oAM13_vWQ80J.