Delta Indexing problems

[stasch]

I know you've seen this one a thousand times. My situation is
somewhat unique. The database is shared. The indexer and the
database are on the same server. All adds/updates to the models that
are being indexed take place on the three front end servers.
Searching is working across all four servers, but delta indexing is
not taking place. I am using the simplest form of delta indexing. I
did all of the stuff that was recommended for delta indexing problems
without success.

I found this error in my Apache error.log on one of the front end
servers:

FATAL: failed to lock /opt/local/var/db/sphinx/contact_info_delta.spl:
Permission denied, will not index. Try --rotate option.

I'm pretty sure that's just an old message from when I was
experimenting because I am currently not running searchd on the front
end servers. Should I be?

I found this error in my Apache error.log on the database server:

FATAL: failed to lock /opt/local/var/db/sphinx/contact_info_delta.spl:
Resource temporarily unavailable, will not index. Try --rotate option.

On both of the machines the contact_info_delta.spl was owned by the
proper user and group but the permissions had been changed to -
rwx------. I know for a fact that they were originally -rw-r--r--.
Is this how sphinx locks the index?

Any idea why delta indexing is not working?

Thanks,

Stan

[James Healy]

The default delta indexing method requires the sphinx daemon and indexes
to be on the same machine as the rails process.

In a multi frontend setup like you're running I'd suggest using the
DelayedJob delta method instead. That way your queue worker can be
running on your database server and have access to the index files.

One alternative deployment method some people have used (have a search
of the list archives) is to run a searchd daemon on each front end. This
requires you regularly build the indexes on your database server then
somehow sync them to your frontends. DelayedJob is probably an easier
first step.

-- James Healy <ji...@deefa.com> Sat, 14 Nov 2009 11:48:25 +1100

[Pat Allan]

Hi Stan

There's two key issues:

* Because Sphinx is on a separate server, Thinking Sphinx on the web
servers can't tell if it's running or not, and therefore assumes it
isn't by default. This is why it doesn't use the --rotate option for
delta indexes.

* If you have separate indexes on each web server, then they won't be
in sync - some will have all the delta changes, others will not.

The best process is, as James has suggested, to use the delayed
deltas, and have that all running on your database server. Also run
all Thinking Sphinx tasks on this server. This way, all the updates
happen in one place, and you shouldn't hit any syncing issues. To
avoid the first problem, you'll need to add the following line:
ThinkingSphinx.remote_sphinx = true

Hope this helps - if it's not clear, let us know :)

--
Pat