question about custom weight / ordering

[Travis J I Corcoran]

I've been using TS for a while now.  Great product - thanks!

I'm getting complaints from our users that the ranking of results is not great.  OK - Sphinx allows  a lot of options.

I've figured out WeightEnumeratorMask and I've figured out how to use :ranker => SPH04 and that works decently...but there's so much more that can be done.  At least at the Sphinx level.

Reading the Sphinx docs here http://sphinxsearch.com/docs/current.html#ranking-factors

I see that I might want to sort by, say 

    5 * weight() + sum(lcs) + 2 * wordcount() 

I've tried to specify :ranker => 'expr', but then I get the error

ThinkingSphinx::SphinxError: missing ranker expression (use OPTION ranker=expr('1+2') for example) - SELECT  *, sum(word_count) as custom  FROM `title_core` WHERE MATCH('Batman') AND `sphinx_deleted` = 0 ORDER BY `custom` DESC LIMIT 0, 20 OPTION ranker=expr; SHOW META

where do I put this 

OPTION ranker=expr('1+2')

?

It doesn't go in the :select argument, because if I try that I get an error.

tl;dr can someone supply an example that uses  SPH_RANK_EXPR and a moderately complex custom ranking algorithm?

Many  thanks!

Travis

[Pat Allan]

Hi Travis

You got pretty close - the :ranker option in a search call ends up in the OPTION sections… so, :ranker => “expr(1+2)” will work.

Cheers

— 

Pat

--
You received this message because you are subscribed to the Google Groups "Thinking Sphinx" group.
To unsubscribe from this group and stop receiving emails from it, send an email to thinking-sphi...@googlegroups.com.
To post to this group, send email to thinkin...@googlegroups.com.
Visit this group at http://groups.google.com/group/thinking-sphinx.
For more options, visit https://groups.google.com/d/optout.

[Travis J I Corcoran]

Pat,

Many thanks for your response.

I think I went down that path at one point and couldn't make it work.

Obviously the expression you're suggesting (1+2) is just an example, but let's go with that.

When I do this:

ret = Title.search("Batman", :ranker     => 'expr(1+2)')

I get the error

 ThinkingSphinx::SyntaxError: sphinxql: syntax error, unexpected CONST_INT, expecting QUOTED_STRING near '1+2); SHOW META' - SELECT * FROM `title_core` WHERE MATCH('Batman') AND `sphinx_deleted` = 0 LIMIT 0, 20 OPTION ranker=expr(1+2); SHOW META

How do I get rid of that error?

Can you give me an example of a more complicated expression that you've actually used?

Many thanks,

Travis

[Travis J I Corcoran]

I've learned something.

This:

ret = Title.search("Batman", :ranker     => "expr(1)")

yields an error, but this

ret = Title.search("Batman", :ranker     => "expr('1')")

works.

Further experimentation shows that the entire expression needs to be quoted.

Two samples:

# use each item's id as its rank. Because we prefer high ranks over low, items are returned in

# order of descending id

#

ret = Title.search("Batman", :limit => 10000, :ranker     => "expr('id')").map(&:id)

# use the expression 1000-id as rank. Because we prefer high ranks over low, items are returned in

# order of ASCENDING id

ret = Title.search("Batman", :limit => 10000, :ranker     => "expr('10000 - id')").map(&:id)

[Pat Allan]

A little odd that quoting is required by Sphinx, but good to know you’ve got it working :)

[Travis J I Corcoran]

Is there any way to build a ranking expression that references the string length of the text field?  e.g. if I'm searching for "Fred", I'd like to rank a record that has a full text of "Fred" over one that has a full text of "there are several words in this document including the word Fred".

I've been staring at the Sphinx docs http://sphinxsearch.com/docs/current.html#document-factors and I don't see anything.  If not, I might have to add a column to my records in SQL, then tweak the before_save callback in ActiveRecord to update that, then use that field in the :ranker expression...

[Pat Allan]

This may be useful:

http://sphinxsearch.com/docs/current.html#conf-index-field-lengths

You should be able to use this by adding `set_property :index_field_lengths => true` to your index definition, and then, once you’ve run rake ts:rebuild, there’s an attribute with the same name as your field which is the length of the field.

However, this can’t be used if you’ve also got :sortable on any fields. For that scenario, you’d need to create this attribute manually, but you don’t need to have a column for it - it can be generated on the fly. An example presuming the column for the field in question is ‘name’:

  has “LENGTH(name)”, :as => :name_length, :type => :integer

And then you can use name_length as an attribute.

— 

Pat

On 16 May 2014, at 7:47 am, Travis J I Corcoran <tjic...@gmail.com> wrote:

Is there any way to build a ranking expression that references the string length of the text field?  e.g. if I'm searching for "Fred", I'd like to rank a record that has a full text of "Fred" over one that has a full text of "there are several words in this document including the word Fred".

I've been staring at the Sphinx docs http://sphinxsearch.com/docs/current.html#document-factors and I don't see anything.  If not, I might have to add a column to my records in SQL, then tweak the before_save callback in ActiveRecord to update that, then use that field in the :ranker expression...

[Terence Lin]

Hi Pat,

I'm still on TS 2.0.14, am I actually able to use rank_mode => "expr(1+2)" in older version of TS?

Thanks,

Terence

IMPORTANT NOTICE:  This message, including any attachments (hereinafter collectively referred to as "Communication"), is intended only for the addressee(s) named above.  This Communication may include information that is privileged, confidential and exempt from disclosure under applicable law.  If the recipient of this Communication is not the intended recipient, or the employee or agent responsible for delivering this Communication to the intended recipient, you are notified that any dissemination, distribution or copying of this Communication is strictly prohibited.  If you have received this Communication in error, please notify the sender immediately by phone or email and permanently delete this Communication from your computer without making a copy. Thank you.

[Pat Allan]

I’m not sure it *will* work, but try the following options in your search query:

  :rank_mode => :expr, :rank_expr => “1+2”

— 

Pat

[Terence Lin]

Thanks Pat, but it doesn't work, I guess I have to upgrade to TS 3.x in order to use this rank_mode:SPH_RANK_EXPR then. Many thanks though :)

[Terence Lin]

Hi Pat,

Corrections, it worked :)

but I need to include  require 'riddle/2.1.0' on the model though. Thanks for pointing the direction.

Thanks,

Terence