is there a way to make TS not configure ranged queries for deltas?

[Johann Tagle]

Our setup:

- multiple app servers, no sphinx installed.

- one dedicated server running sphinx

- TS is still used to generated the sphinx config file, then loaded to the sphinx server

- TS will set delta=1 for new/updated records, but we let cron jobs on the sphinx server do the full and delta index updating

Everything is working, except I see an inefficiency with the way the delta index is updated for one model.  This model has around 5M records, with about 10k being updated daily.  Just before writing this, I ran the statement defined by sql_query_range and saw that the values for the min and max ids are at the opposite extremes, meaning the delta reindexing will practically scan through the whole table when there are just 5k records with delta=1.  We indexed the delta column and so doing a simple "select * from table where delta=1" is quite fast.

Is there anything we can put within define_index or somewhere so that it will generate the config for the delta source to not use ranged queries?

Thanks

Johann

[Johann Tagle]

I guess the solution for me is to add an sql_range_step with a high value in the source definition of the delta.  Hmmm maybe it would be good if TS can allow index configuration settings for main and delta.

[Pat Allan]

Hi Johann

Yeah, currently there's no way to have settings just for the delta index. TS's DSL doesn't allow for it… I'm open to suggestions on how it could, but this would very likely be something that I add to TS 3, not earlier versions.

You could just set the sql_range_step to be quite large generally? May not have too much of an impact for full indexing (indeed, in some situations can speed things up).

Cheers

--
Pat

> --
> You received this message because you are subscribed to the Google Groups "Thinking Sphinx" group.
> To view this discussion on the web visit https://groups.google.com/d/msg/thinking-sphinx/-/w4FxEspJkoUJ.
> To post to this group, send email to thinkin...@googlegroups.com.
> To unsubscribe from this group, send email to thinking-sphi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/thinking-sphinx?hl=en.

[Johann Tagle]

Thanks Pat and sorry for late reply - got caught up with the holidays (Happy New Year!).  For the new support maybe something like:

define_index do

.

.

.

  enable_delta do

     #put delta-specific settings here

  end

end

Thanks again!

Johann

[Pat Allan]

So, I've realised this is actually possible - albeit with a little bit of duplication - with Thinking Sphinx v3 (a release candidate is out, and I should have the full release out today too, with a bit of luck):

# app/indices/article_index.rb

# Whatever you normally use with :delta is the argument here.
processor = ThinkingSphinx::Deltas.processor_for(true)
options = {:with => :active_record, :delta_processor => processor, :delta? => false}

ThinkingSphinx::Index.define_for :article, options do
# core index definition
indexes title, body
has created_at, user_id
end

ThinkingSphinx::Index.define_for :article, options.merge(:delta? => true) do
# delta index definition
indexes title, body
has created_at, user_id

set_property :sql_range_step => 10_000_000
end

What are your thoughts on this?

--
Pat

> To view this discussion on the web visit https://groups.google.com/d/msg/thinking-sphinx/-/M9dpgxcEZY4J.