I'm returning a fairly large data set from sphinx for a client with a lot of filter attributes. I'm trying to get just the active record IDs.
I'm seeing significant delays and a lot of memory usage. In debugging this I noticed that in search.rb's compose_ids_results, the @results object contains not only the active record IDs but pretty much my entire index with all the filter attributes. This is a lot of data and very slow for sphinx to return to my rails app.
I'm wondering if there is a way to tell sphinx not to return everything it knows about each document and just return the sphinx_internal_id?
On Tuesday, October 2, 2012 8:27:16 PM UTC-4, Eric wrote:
> I'm returning a fairly large data set from sphinx for a client with a lot > of filter attributes. I'm trying to get just the active record IDs.
> I'm seeing significant delays and a lot of memory usage. In debugging > this I noticed that in search.rb's compose_ids_results, the @results object > contains not only the active record IDs but pretty much my entire index > with all the filter attributes. This is a lot of data and very slow for > sphinx to return to my rails app.
> I'm wondering if there is a way to tell sphinx not to return everything it > knows about each document and just return the sphinx_internal_id?
RIght, I'm using search_for_ids and it works; Thinking Sphinx only fetches the list of IDs not ActiveRecord objects. But when it loads the list of matching records from sphinx, the underlying protocol returns an enormous hash (in my case) for each document like this:
I have a lot of attributes associated with each document in the index, so the result is enormous. I debugged the thinking sphinx code, looking at search.rb's compose_ids_results and it actually includes the text of each full text field as well as all the attributes, no idea why. I really need to figure out how to avoid this as it adds significant overhead and I don't see a need for returning all this data.
> On Tuesday, October 2, 2012 8:27:16 PM UTC-4, Eric wrote:
>> I'm returning a fairly large data set from sphinx for a client with a lot >> of filter attributes. I'm trying to get just the active record IDs.
>> I'm seeing significant delays and a lot of memory usage. In debugging >> this I noticed that in search.rb's compose_ids_results, the @results object >> contains not only the active record IDs but pretty much my entire index >> with all the filter attributes. This is a lot of data and very slow for >> sphinx to return to my rails app.
>> I'm wondering if there is a way to tell sphinx not to return everything >> it knows about each document and just return the sphinx_internal_id?
On Wednesday, October 3, 2012 5:05:32 AM UTC-7, Eric wrote:
> RIght, I'm using search_for_ids and it works; Thinking Sphinx only fetches > the list of IDs not ActiveRecord objects. But when it loads the list of > matching records from sphinx, the underlying protocol returns an enormous > hash (in my case) for each document like this:
> I have a lot of attributes associated with each document in the index, so > the result is enormous. I debugged the thinking sphinx code, looking > at search.rb's compose_ids_results and it actually includes the text of > each full text field as well as all the attributes, no idea why. I really > need to figure out how to avoid this as it adds significant overhead and I > don't see a need for returning all this data.
> Thanks, > Eric
> On Tuesday, October 2, 2012 9:00:21 PM UTC-7, Chris wrote:
>> On Tuesday, October 2, 2012 8:27:16 PM UTC-4, Eric wrote:
>>> I'm returning a fairly large data set from sphinx for a client with a >>> lot of filter attributes. I'm trying to get just the active record IDs.
>>> I'm seeing significant delays and a lot of memory usage. In debugging >>> this I noticed that in search.rb's compose_ids_results, the @results object >>> contains not only the active record IDs but pretty much my entire index >>> with all the filter attributes. This is a lot of data and very slow for >>> sphinx to return to my rails app.
>>> I'm wondering if there is a way to tell sphinx not to return everything >>> it knows about each document and just return the sphinx_internal_id?
> I'm returning a fairly large data set from sphinx for a client with a lot of filter attributes. I'm trying to get just the active record IDs.
> I'm seeing significant delays and a lot of memory usage. In debugging this I noticed that in search.rb's compose_ids_results, the @results object contains not only the active record IDs but pretty much my entire index with all the filter attributes. This is a lot of data and very slow for sphinx to return to my rails app.
> I'm wondering if there is a way to tell sphinx not to return everything it knows about each document and just return the sphinx_internal_id?
> Thanks much, > Eric
> -- > You received this message because you are subscribed to the Google Groups "Thinking Sphinx" group. > To view this discussion on the web visit https://groups.google.com/d/msg/thinking-sphinx/-/fn4acDteSTYJ. > To post to this group, send email to thinking-sphinx@googlegroups.com. > To unsubscribe from this group, send email to thinking-sphinx+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/thinking-sphinx?hl=en.
Thanks pat. As always, you have the answer- sphinx_select does the trick.
I wanted to mention a caveat; although full-text searching and attribute filtering worked great, I needed to include the fields I wanted to order by. So when I build my query, I essentially parse the order clause and add the fields to the sphinx_select clause and everything works perfectly.
For a large page, this makes an enormous gigantic difference.
On Wednesday, October 3, 2012 5:42:31 AM UTC-7, Pat Allan wrote:
> The :sphinx_select option should do the trick - it's much like a SELECT > option for a SQL query... so, try adding this option:
> :sphinx_select => 'sphinx_internal_id'
> Not sure off the top of my head whether other attributes will be required, > but probably not given you're using search_for_ids.
> Cheers
> -- > Pat
> On 03/10/2012, at 10:27 AM, Eric wrote:
> > I'm returning a fairly large data set from sphinx for a client with a > lot of filter attributes. I'm trying to get just the active record IDs.
> > I'm seeing significant delays and a lot of memory usage. In debugging > this I noticed that in search.rb's compose_ids_results, the @results object > contains not only the active record IDs but pretty much my entire index > with all the filter attributes. This is a lot of data and very slow for > sphinx to return to my rails app.
> > I'm wondering if there is a way to tell sphinx not to return everything > it knows about each document and just return the sphinx_internal_id?
> > Thanks much, > > Eric
> > -- > > You received this message because you are subscribed to the Google > Groups "Thinking Sphinx" group. > > To view this discussion on the web visit > https://groups.google.com/d/msg/thinking-sphinx/-/fn4acDteSTYJ. > > To post to this group, send email to thinkin...@googlegroups.com<javascript:> > . > > To unsubscribe from this group, send email to > thinking-sphi...@googlegroups.com <javascript:>. > > For more options, visit this group at > http://groups.google.com/group/thinking-sphinx?hl=en.
