Multi-server setup with distributed indices

[Jarkko Laine]

Has anyone been successful in setting up Thinking Sphinx in a multi-
server distributed setup using a searchd daemon on every app server?
Is it even possible?

I can find references everywhere that sphinx supports distributed
indices but have been unable to find clear how-to instructions for
setting such a setup up, especially in context of Thinking Sphinx.

Basically our dream setup would be that each app server would have its
own searchd which would then talk to each other, using their own delta
indices.

We could in the current state of affairs just set up one searchd
running on the db server, but that would mean losing the delta index
option and the distributed indexing sounds like something we'll be
needing in the future anyway.

Cheers,
//jarkko

[Josh]

We have been doing this successfully for about six months. We wrote a
shell script that:

- Runs our indexes on our database server, using a copy of our
production
sphinx config
- Renames the resulting files with the ".new" filename
- Copies these files to each of our app servers, where searchd is
running
- Sends a SIGHUP to searchd, which does a seamless load of the new
indexes

We haven't seen any problems with this approach.

- Josh

[DrMark]

Hi Jarkko,

We have also been doing this. We run searchd on each web server. This
helps us scale more linearly. We are doing it the same way as Josh.
The only downside to this approach is your web servers have to be able
to transfer files between them and the database server, which could be
viewed as a minor security issue.

Best of luck!

DrMark

[Jarkko Laine]

On 26.3.2009, at 20.35, DrMark wrote:

>
> Hi Jarkko,
>
> We have also been doing this. We run searchd on each web server. This
> helps us scale more linearly. We are doing it the same way as Josh.
> The only downside to this approach is your web servers have to be able
> to transfer files between them and the database server, which could be
> viewed as a minor security issue.

Hi Mark (and Josh),

and thanks for your help!

If you're running searchd on both app servers, couldn't they be set up
so that the index is distributed between the two and they talk to each
other over the network? At least that's how I envision the distributed
indices Sphinx supports work. Or would that not work with TS? Delta
indices might cause some trouble with that, I could imagine.

Cheers,
//jarkko

--
Jarkko Laine
http://jlaine.net
http://dotherightthing.com
http://odesign.fi

Check out my latest book, Unobtrusive Prototype, fresh off the
Peepcode oven:
http://peepcode.com/products/unobtrusive-prototype-js

--
Jarkko Laine
http://jlaine.net
http://dotherightthing.com
http://www.railsecommerce.com
http://odesign.fi