Thinking Sphinx Delta Indexes Not working

[Strafe]

I know this has been covered before, but I couldn't find something in
the other documents which helped me, so I thought i'd make a thread
here.

So im running Rails 3.0.9 on Ruby 1.9.2.

Thinking Sphinx worked great, then I decided I wanted to implement
Delta Indexes.

So I added the column to my model

:delta, :boolean, :default => true, :null => false

Then I added the property to my model

define_index do
set_property :delta => true
end

Then I ran my migrations

Then I ran a rake ts:rebuild

Started my rails server, and everything seemed to be working.

However, when I create or destroy objects they do not appear in the
search.

It would seem that everything is working fine, the delta index is in
the model, it's set to true when I create a record. I see the Sphinx
stuff in the console without errors when I create a record.

And when I run the index_delta command in the IRB it seems to find all
the new records.

> Hardware.index_delta
Sphinx 2.0.1-beta (r2792)
Copyright (c) 2001-2011, Andrew Aksyonoff
Copyright (c) 2008-2011, Sphinx Technologies Inc (http://
sphinxsearch.com)

using config file 'C:/ruby192/cec_inventory/config/
development.sphinx.conf'...
indexing index 'hardware_delta'...
WARNING: collect_hits: mem_limit=0 kb too low, increasing to 14592 kb
collected 2 docs, 0.0 MB
sorted 0.0 Mhits, 100.0% done
total 2 docs, 31 bytes
total 0.038 sec, 803 bytes/sec, 51.86 docs/sec
total 2 reads, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg
total 9 writes, 0.000 sec, 0.1 kb/call avg, 0.0 msec/call avg
rotating indices: succesfully sent SIGHUP to searchd (pid=7772).
[delta index stuff here]

Even if i run a ts:reindex when the searchd is running, it doesnt find
the new records, it only set's their delta to false. They don't show
up. The only way I can get them to show is if I run a ts:rebuild.


Can you guys help me? Sorry If i accidentally left out some details.

[Pat Allan]

Hi

Could you run us through how you're testing the deltas in the command line? It's worth noting that Sphinx takes a second to load the new data, so don't run a search *straight* after creating a new object - I usually use `sleep 0.5` in my tests to account for this.

Also, can you provide the full define_index block?

Cheers

--
Pat

> --
> You received this message because you are subscribed to the Google Groups "Thinking Sphinx" group.
> To post to this group, send email to thinkin...@googlegroups.com.
> To unsubscribe from this group, send email to thinking-sphi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/thinking-sphinx?hl=en.
>

[Strafe]

Step 1. I have premade records which are indexed initially.

irb(main):001:0> Hardware.last
=> #<Hardware id: 3, [...] delta: false >

Step 2. I create a new record

irb(main):001:0> Hardware.last
=> #<Hardware id: 4, [...] delta: true >

Step 3. I checked for defined indexes

irb(main):003:0> Hardware.define_indexes
=> true

Step 4. I check out delta index

irb(main):004:0> Hardware.index_delta

Sphinx 2.0.1-beta (r2792)
Copyright (c) 2001-2011, Andrew Aksyonoff
Copyright (c) 2008-2011, Sphinx Technologies Inc (http://
sphinxsearch.com)

using config file 'C:/ruby192/cec_inventory/config/
development.sphinx.conf'...
indexing index 'hardware_delta'...
WARNING: collect_hits: mem_limit=0 kb too low, increasing to 14592 kb

collected 1 docs, 0.0 MB

sorted 0.0 Mhits, 100.0% done

total 1 docs, 18 bytes
total 0.060 sec, 295 bytes/sec, 16.44 docs/sec

total 2 reads, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg
total 9 writes, 0.000 sec, 0.1 kb/call avg, 0.0 msec/call avg

rotating indices: succesfully sent SIGHUP to searchd (pid=1824).
=> [#<ThinkingSphinx::Deltas::DefaultDelta:0x414b910 [........] ]



Here is my define index block

define_index do
# fields
indexes equipment_manufacturer, :sortable => true
indexes equipment_model, :sortable => true
indexes serial_number
indexes equipment_building
indexes equipment_location

set_property :delta => true
has created_at, updated_at
end



Let me know if you need anything else.

[Pat Allan]

If you have deltas defined, then the indexing should happen already - you don't need to run define_indexes or index_delta.

Can you show us what data you're using in the new record, and what your test search query is?

--
Pat

[Strafe]

Creating Record

irb(main):003:0> Hardware.create(:equipment_manufacturer =>
"Dell", :equipment_model => "1337", :serial_number => "12345678")

Sphinx 2.0.1-beta (r2792)
Copyright (c) 2001-2011, Andrew Aksyonoff
Copyright (c) 2008-2011, Sphinx Technologies Inc (http://
sphinxsearch.com)

using config file 'C:/ruby192/project/config/

development.sphinx.conf'...
indexing index 'hardware_delta'...
WARNING: collect_hits: mem_limit=0 kb too low, increasing to 14592 kb
collected 1 docs, 0.0 MB
sorted 0.0 Mhits, 100.0% done

total 1 docs, 16 bytes
total 0.029 sec, 551 bytes/sec, 34.46 docs/sec

total 2 reads, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg
total 9 writes, 0.000 sec, 0.1 kb/call avg, 0.0 msec/call avg

rotating indices: succesfully sent SIGHUP to searchd (pid=7588).
=> #<Hardware id: 11, equipment_model: "1337", serial_number:
"12345678", equipment_building: nil, equipment_loca
tion: nil, delta: true, created_at: "2011-11-14 13:20:37", updated_at:
"2011-11-14 13:20:37", equipment_manufacturer: "Dell">

Then I run a Hardware.search with no parameters and it doesnt show up.

[Pat Allan]

I'm afraid I don't know what the cause of the problem is here… it looks like you don't even have 20 records, so it's not a matter of pagination getting in the way… the delta processing happens, the new record seems to be recorded in it.

What happens if you run - from the command line - a manual search:
search --config config/development.sphinx.conf -i hardware_delta dell

Does it get returned?

--
Pat

[Benjamin Zumhagen]

This is what I get when I run that command.

irb(main):006:0> search --config config/development.sphinx.conf -i hardware_delta dell

SyntaxError: (irb):6: syntax error, unexpected tIDENTIFIER, expecting keyword_do or '{' or '('

search --config config/development.sphinx.conf -i hardware_delta dell

                      ^

(irb):6: syntax error, unexpected tUMINUS, expecting $end

search --config config/development.sphinx.conf -i hardware_delta dell

                                                ^

        from C:/ruby192/lib/ruby/gems/1.9.1/gems/railties-3.0.9/lib/rails/commands/console.rb:44:in `start'

        from C:/ruby192/lib/ruby/gems/1.9.1/gems/railties-3.0.9/lib/rails/commands/console.rb:8:in `start'

        from C:/ruby192/lib/ruby/gems/1.9.1/gems/railties-3.0.9/lib/rails/commands.rb:23:in `<top (required)>'

        from script/rails:6:in `require'

        from script/rails:6:in `<main>'

[Benjamin Zumhagen]

Also, I am running this on a windows environment.

[Pat Allan]

You need to run this command in your shell, not in script/console.

--
Pat

[Benjamin Zumhagen]

I ran that command in the IRB shell

[Pat Allan]

We have a confusion of terms here…

On a *nix box the shell I'm talking about would be bash, zsh, etc. On Windows, I guess it's cmd.exe?

[Benjamin Zumhagen]

Ooohh, okay.

[geoffw8]

Hey, so I am having this EXACT issue and when I run the shell command I get:

gw:WTL2 gwt$ search --config config/development.sphinx.conf -i product_delta top

Sphinx 2.0.4-release (r3135)

Copyright (c) 2001-2012, Andrew Aksyonoff

Copyright (c) 2008-2012, Sphinx Technologies Inc (http://sphinxsearch.com)

using config file 'config/development.sphinx.conf'...

index 'product_delta': search error: .

All seems to be well coming in:

Product Load (1.3ms)  SELECT `products`.* FROM `products` WHERE (guid = 'BLFA1176M1851' AND feed_id = 1) LIMIT 1

  Brand Load (0.4ms)  SELECT `brands`.* FROM `brands` WHERE `brands`.`name` = 'Bench' LIMIT 1

   (0.1ms)  BEGIN

  SQL (0.6ms)  INSERT INTO `products` (`data omitted')

  Feed Load (0.5ms)  SELECT `feeds`.* FROM `feeds` WHERE `feeds`.`id` = 1 LIMIT 1

  Category Load (0.2ms)  SELECT `categories`.* FROM `categories` WHERE `categories`.`id` IS NULL LIMIT 1

  Subcategory Load (0.2ms)  SELECT `subcategories`.* FROM `subcategories` WHERE `subcategories`.`id` IS NULL LIMIT 1

   (0.4ms)  COMMIT

Sphinx 2.0.4-release (r3135)

Copyright (c) 2001-2012, Andrew Aksyonoff

Copyright (c) 2008-2012, Sphinx Technologies Inc (http://sphinxsearch.com)

using config file '/Users/geoffwright/Documents/rails/WTL2/config/development.sphinx.conf'...

indexing index 'product_delta'...

collected 0 docs, 0.0 MB

collected 0 attr values

sorted 0.0 Mvalues, 100.0% done

total 0 docs, 0 bytes

total 0.014 sec, 0 bytes/sec, 0.00 docs/sec

total 5 reads, 0.000 sec, 25.5 kb/call avg, 0.0 msec/call avg

total 5 writes, 0.000 sec, 0.2 kb/call avg, 0.0 msec/call avg

rotating indices: succesfully sent SIGHUP to searchd (pid=1488).

  Sphinx Query (2.1ms)  

  Sphinx  Found 0 results

 

Any ideas what I might be doing wrong? I'd have thought collected should be > 0

Cheers

Geoff

[Pat Allan]

Hi Geoff

You omitted the data for the new product - can you confirm that the delta flag was set to true there? Also, what version of Thinking Sphinx are you using?

Cheers

--
Pat

> To view this discussion on the web visit https://groups.google.com/d/msg/thinking-sphinx/-/yQzDyqDPR3cJ.

[geoffw8]

Hi Pat

(Can't believe I didn't see this, I was just writing a new post!!)

Yes, the delta flag for the product was set to true (1 in the db).

I am using version 2.0.13 of TS and Sphinx 2.0.6-release (r3473)

Here is the full output:

  Product Load (0.8ms)  SELECT `products`.* FROM `products` WHERE (guid = '1411077050' AND feed_id = 2) LIMIT 1

  Brand Load (0.6ms)  SELECT `brands`.* FROM `brands` WHERE `brands`.`name` = 'Schuh' LIMIT 1

   (0.1ms)  BEGIN

  SQL (2.5ms)  INSERT INTO `products` (`additional_data_1`, `additional_data_2`, `additional_data_3`, `brand`, `category_id`, `colour_id`, `cost_id`, `created_at`, `dead`, `deeplink`, `delta`, `description`, `feed_id`, `feed_log_id`, `gender_id`, `get_photo_status`, `guid`, `image`, `image_attempts`, `is_duplicate`, `last_checked`, `merchant_link`, `name`, `old_prices`, `parent_id`, `pattern_id`, `permalink`, `photo_content_type`, `photo_file_name`, `photo_file_sze`, `photo_updated_at`, `photograph_id`, `price`, `processed`, `rank`, `sale`, `sale_on`, `subcategory_id`, `tags`, `updated_at`) VALUES ('Womens Footwear', 'Women\'s Footwear', NULL, 'Schuh', NULL, NULL, NULL, '2013-01-02 14:17:08', 0, 'http://www.awin1.com/pclick.php?p=696651025&a=93579&m=2044', 1, 'Youll get more attention showing these off than sticking your thumb out! The Hitch Hike is one of our own and features a black suede upper and two functional buckles. This tough little ankle boot sits on a 6cm heel and a cleated sole.', 2, 13125, NULL, NULL, '1411077050', 'http://www.schuh.co.uk/images/product/141107/1411077050_exlg.jpg', NULL, NULL, NULL, 'http://www.schuh.co.uk/womens-black-schuh-hitch-hike/1411077050/', 'Schuh Hitch Hike', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, '85.00', NULL, NULL, 0, NULL, NULL, NULL, '2013-01-02 14:17:08')

  Feed Load (1.0ms)  SELECT `feeds`.* FROM `feeds` WHERE `feeds`.`id` = 2 LIMIT 1

  Category Load (0.5ms)  SELECT `categories`.* FROM `categories` WHERE `categories`.`id` IS NULL LIMIT 1

  Subcategory Load (0.5ms)  SELECT `subcategories`.* FROM `subcategories` WHERE `subcategories`.`id` IS NULL LIMIT 1

   (2.4ms)  COMMIT

Sphinx 2.0.6-release (r3473)

Copyright (c) 2001-2012, Andrew Aksyonoff

Copyright (c) 2008-2012, Sphinx Technologies Inc (http://sphinxsearch.com)

using config file '/Users/geoffmwright/Documents/rails/WTL2/config/development.sphinx.conf'...

WARNING: no such index 'product_delta', skipping.

total 0 reads, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg

total 0 writes, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg

  Sphinx Query (3.9ms)

  Sphinx  Found 0 results

And I just checked in the db, and this product does have delta set to 1

Any ideas on where to look first? Its not working on my local, or prod environments.

Thanks for your help all, its much appreciated

Kind regards,

Geoff 

[Pat Allan]

Hi Geoff

This is different in the Sphinx output - but I guess you may have disabled deltas at some point in the meantime?

> WARNING: no such index 'product_delta', skipping.

Run ts:rebuild and give it another spin, just to confirm.

--
Pat

> To view this discussion on the web visit https://groups.google.com/d/msg/thinking-sphinx/-/eGZsYthFdacJ.

[geoffw8]

Hi Pat,

Woops, my bad. Here's a link to a gist after I rebuilt: https://gist.github.com/9a6e0127491b48173416

Let me make sure I'm getting this right. So I did the migration, added "set_property :delta => true" into my index block and rebuilt.

What I do is delete all the records from our database - rebuild so the index is empty and we have no results - and start redownloading the products into the database - what I would expect without typing rake ts:index (or giving ANY additional command) is for the products to be immediately available. Is this the correct workflow?

I did just go through this again (on a new laptop) and it hasn't worked again, and the delta value is marked as 1 for all the products.

Arghh! Its driving me mad. Would appreciate any pointers...!

Thanks,

Geoff

[Pat Allan]

Hi Geoff

Sounds like you're doing everything right. Can you add your Sphinx configuration file to that gist - though you should remove any database usernames and passwords from the sources.

--
Pat

> To view this discussion on the web visit https://groups.google.com/d/msg/thinking-sphinx/-/IVzzLxsn6aYJ.

[geoffw8]

Hey Pat

https://gist.github.com/4d62474260ceaa42f6f6

There ya go. Thats the dev config... Let me know if you need anything else.

Thanks again for this!

Regards

Geoff

[Pat Allan]

Thanks Geoff.

I think I've found the culprit - you've got additional WHERE clauses in your index definition of dead IS NULL and image_attempts = 1, yet neither of those are true in the SQL INSERT when the product is created. That'd be why the indexer isn't picking up your new records. Best to either remove those conditions, or ensure the expected values are covered by them.

Cheers

--
Pat

> To view this discussion on the web visit https://groups.google.com/d/msg/thinking-sphinx/-/cxGroG3ceWEJ.

[geoffw8]

Pat it would appear you are absolutely right.

That did throw a bit of a spanner in the works as we need to process stuff as it comes in, twinning this with your delayed gem seems to have done the job. Gives us enough time to process.

Thanks!

Geoff