Issues with delta indexes

[gobigdave]

I have two issues with delta indexes. First, I can not get them to
update from my dev browser. I do use Passenger even in development,
but I verified that searchd and apache(httpd) are running as the same
user. I also have bin_path set in my sphinx.yml file. If I add new
items from my console, then I can search for the new records fine.
Even when adding records from my browser, my log file has:

WorklingDeltaWorker#index:
{:delta_index_name=>"person_delta", :uid=>"workling_delta_workers:index:d9ba9a4bbec22daca144db294206d7b2", :core_index_name=>"person_core", :document_id=>242557}
ThinkingSphinx::Search.search_for_id is deprecated. Please use
ThinkingSphinx.search_for_id instead.
Querying Sphinx:

Any other ideas on what I can check?

Two classes for reference:

class WorklingDelta < ThinkingSphinx::Deltas::DefaultDelta

def index(model, instance = nil)
return true unless ThinkingSphinx.updates_enabled? &&
ThinkingSphinx.deltas_enabled?
return true if instance && !toggled(instance)

doc_id = instance ? instance.sphinx_document_id : nil
WorklingDeltaWorker.asynch_index(:delta_index_name =>
delta_index_name(model), :core_index_name => core_index_name
(model), :document_id => doc_id)

return true
end

end

class WorklingDeltaWorker < Workling::Base

def index(options = {})
logger.info("WorklingDeltaWorker#index: #{options.inspect}")
ThinkingSphinx::Deltas::DeltaJob.new(options
[:delta_index_name]).perform
if options[:document_id]
ThinkingSphinx::Deltas::FlagAsDeletedJob.new(options
[:core_index_name], options[:document_id]).perform
end

return true
end

end

If you are not familiar with workling, it allows for simple async
processing. In development mode, however, everything is executed
synchronously.


The second issue is that the index does not update when I change an
attribute. Here is some of my index:

define_index do

indexes first_name, last_name, :sortable => true
indexes [title, title1], :as => :title, :sortable => true
indexes description

has "CRC32(people.state)", :as => :state, :type => :integer
has "CRC32(people.country)", :as => :country, :type => :integer
has "CRC32(people.area_code)", :as => :area_code, :type
=> :integer

set_property :delta => WorklingDelta
end

When I create a new record from the console, I see the following:

using config file '/Users/dave/projects/msc/config/
development.sphinx.conf'...
indexing index 'person_delta'...
collected 1 docs, 0.0 MB
collected 0 attr values
sorted 0.0 Mvalues, 100.0% done
sorted 0.0 Mhits, 100.0% done
total 1 docs, 12 bytes
total 0.235 sec, 51.03 bytes/sec, 4.25 docs/sec
rotating indices: succesfully sent SIGHUP to searchd (pid=32480).
=> true

However, if I update the state, country, or area_code, then nothing is
output, and the index is not updated. Note, I have the state, country,
and area_code as attributes because I want to filter on multiple
values. Is this a known issue? Do I need to move these to indexes and
use 'OR' searches to make this happy?

[gobigdave]

FYI, I verified that when Workling/Starling are running in async mode
my delta indexes are being properly updated. What is strange is that
the workling, httpd, and searchd processes are all running with the
same user. I've also verified the path several times, but I will do so
again. Very strange...

Also, I moved the state, country, and area_code attributes to indexes,
and now updates to the object properly cause the delta index to be
updated. Looking at the thinking_sphinx code it seems that attribute
updates should have caused deltas to be updated, but I'm thinking that
it may not work with calculated attributes.

[Pat Allan]

Hmm, it could be a bug.

Sphinx can update some attributes (integers, timestamps, booleans) -
and so if you only change columns that are tied to those types of
attributes, Thinking Sphinx will not invoke the delta indexing.

However, if there are any associations used, or manual SQL, then delta
indexing should always be fired. And given that it works via script/
console, then that suggests to me that there's nothing wrong with that
code.

Which doesn't really help.

--
Pat

[gobigdave]

Even when running in the console, calculated attribute modification
did not cause delta index update.

person = Person.new
person.last_name = "Smith"
person.save # caused delta index
person.area_code = "617" # assume you have has CRC32("area_code"), :as

=> :area_code, :type => :integer

person.save # did not cause delta index

As I said, I moved these attributes to fields and updated my advanced
search code to use conditions.

I still can't make Passenger happy in development mode even when
everything is running as the same user and the path is right. At least
I'm pretty sure they are.

My config/sphinx.yml for dev:

development:
morphology: stem_en
max_matches: 100000
bin_path: /usr/local/bin


-Dave

[Pat Allan]

Okay, sounds like it is a bug, maybe what I was mulling in the
previous email is what's at fault. Will try to find some time to
investigate.

--
Pat

[baldrailers]

I have the same issue.

It's working fine on my localmachine(Leopard) but when i try it on
production server(Ubuntu), both DEV and PROD environment is not
triggering the delta update. I try to force the delta update to true
but nothing happen.

I've tried updating the attribute on script/console and it triggers
the update on delta.

Thanks!

On Aug 28, 8:10 am, Pat Allan <p...@freelancing-gods.com> wrote:
> Okay, sounds like it is a bug, maybe what I was mulling in the  
> previous email is what's at fault. Will try to find some time to  
> investigate.
>
> --
> Pat
>
> On 27/08/2009, at 8:50 PM, gobigdave wrote:
>
>
>
> > Even when running in the console, calculated attribute modification

> > did not causedeltaindex update.

> >>> mydeltaindexes are being properly updated. What is strange is that

> >>> the workling, httpd, and searchd processes are all running with the
> >>> same user. I've also verified the path several times, but I will  
> >>> do so
> >>> again. Very strange...
>
> >>> Also, I moved the state, country, and area_code attributes to  
> >>> indexes,

> >>> and now updates to the object properly cause thedeltaindex to be

> >>> updated. Looking at the thinking_sphinx code it seems that attribute
> >>> updates should have caused deltas to be updated, but I'm thinking  
> >>> that
> >>> it may not work with calculated attributes.
>
> >>> On Aug 26, 11:31 pm, gobigdave <gobigd...@gmail.com> wrote:

> >>>> I have two issues withdeltaindexes. First, I can not get them to

> >>>> The secondissueis that the index does not update when I change an

> >>>> attribute. Here is some of my index:
>
> >>>>   define_index do
>
> >>>>     indexes first_name, last_name, :sortable => true
> >>>>     indexes [title, title1], :as => :title, :sortable => true
> >>>>     indexes description
>
> >>>>     has "CRC32(people.state)", :as => :state, :type => :integer
> >>>>     has "CRC32(people.country)", :as => :country, :type => :integer
> >>>>     has "CRC32(people.area_code)", :as => :area_code, :type
> >>>> => :integer
>

> >>>>     set_property :delta=> WorklingDelta

> >>>>   end
>
> >>>> When I create a new record from the console, I see the following:
>
> >>>> using config file '/Users/dave/projects/msc/config/
> >>>> development.sphinx.conf'...
> >>>> indexing index 'person_delta'...
> >>>> collected 1 docs, 0.0 MB
> >>>> collected 0 attr values
> >>>> sorted 0.0 Mvalues, 100.0% done
> >>>> sorted 0.0 Mhits, 100.0% done
> >>>> total 1 docs, 12 bytes
> >>>> total 0.235 sec, 51.03 bytes/sec, 4.25 docs/sec
> >>>> rotating indices: succesfully sent SIGHUP to searchd (pid=32480).
> >>>> => true
>
> >>>> However, if I update the state, country, or area_code, then nothing
> >>>> is
> >>>> output, and the index is not updated. Note, I have the state,
> >>>> country,
> >>>> and area_code as attributes because I want to filter on multiple

> >>>> values. Is this a knownissue? Do I need to move these to indexes  

[Pat Allan]

Just to confirm (all with regards to the server), file permissions are
okay, you're not seeing any errors in the logs, and it does fire off
the delta index in script/console?

--
Pat

[Lf]

Hi Pat,

I faced similar situation: when I create a model, it doesn't fire
delta indexing under mongrel; but *does* under script/console :)

Best,
Anton

[Pat Allan]

Hi Anton

Just to confirm - are the mongrels running as the same user that
invokes the indexing/daemon rake tasks?

--
Pat