God, Thinking Sphinx, Capistrano and the Universe

[vanderkerkoff]

Hello everyone, I can't get god to start my TS and it's starting to
drive me a bit mad(madder). I'm following Tims article here,
http://openmonkey.com/articles/2008/09/configuring-god-to-monitor-sphinxs-searchd
and having some problems.

I've included all my config files and if anyone is nice(mad) enough to
have a look through this for me I'd be eternally grateful. I'm
emailing Tim as well to see if he can help, Tim, sorry mate.

Symptoms

I run a cap deploy, TS restarts and reindexes.

I want god to monitor TS and restart it if it goes outside it's memory
limits or if something 'mental' happens and searchd stops for some
reason.

After deploying, I run sudo god log gaol-1 and it says that it's up.
I check ps aux | grep searchd, and it is.

To test if god will restart I've tried two options.

1. sudo killall searchd

searchd stops, god notices that it's stopped, but it fails to restart
it with the error start command exited with non-zero code = 1

2. sudo god stop gaol-1

searchd stops, god is now not monitoring it

so I run sudo god start gaol-1 and it starts monitoring it, but never
starts searchd, error start command exited with non-zero code = 1

The command it says it is trying to run, /usr/local/bin/searchd --
pidfile --config /var/www/rails/gaol/current/config/
staging_production.sphinx.conf, can be run from anywhere on the system
when I'm logged into the system as the same user in the uid setting

Here's the god watch

God.watch do |w|
w.group = "gaol-sphinx"
w.name = "gaol-1"

w.interval = 30.seconds

w.uid = "username"
w.gid = "groupname"

w.start = "searchd --config /var/www/rails/gaol/current/
config/staging_production.sphinx.conf"
w.start_grace = 10.seconds
w.stop = "searchd --config /var/www/rails/gaol/current/
config/staging_production.sphinx.conf --stop"
w.stop_grace = 10.seconds
w.restart = w.stop + " && " + w.start
w.restart_grace = 15.seconds

w.pid_file = "/var/www/rails/gaol/shared/log/searchd.production.pid"

w.behavior(:clean_pid_file)

w.start_if do |start|
start.condition(:process_running) do |c|
c.interval = 5.seconds
c.running = false
end
end

w.restart_if do |restart|
restart.condition(:memory_usage) do |c|
c.above = 100.megabytes
c.times = [3, 5] # 3 out of 5 intervals
end
end

w.lifecycle do |on|
on.condition(:flapping) do |c|
c.to_state = [:start, :restart]
c.times = 5
c.within = 5.minutes
c.transition = :unmonitored
c.retry_in = 10.minutes
c.retry_times = 5
c.retry_within = 2.hours
end
end
end

Here's config/sphinx.yml

production:
bin_path: "/usr/local/bin"
searchd_file_path: "/var/www/rails/gaol/shared/sphinx"
config_file: "/var/www/rails/gaol/current/config/
staging_production.sphinx.conf"
searchd_log_file: "/var/www/rails/gaol/shared/log/searchd.log"
query_log_file: "/var/www/rails/gaol/shared/log/searchd.query.log"
pid_file: "/var/www/rails/gaol/shared/log/searchd.production.pid"


Here's the static production.sphinx.conf file (we run the index
command with INDEX_ONLY=true)

indexer
{
}

searchd
{
address = 127.0.0.1
port = 3312
pid_file = /var/www/rails/gaol/shared/log/searchd.production.pid
}

source app_core_0
{
type = mysql
sql_host = sql database server
sql_user = user
sql_pass = password
sql_db = database_name
sql_sock = /var/run/mysqld/mysqld.sock
sql_query_pre = SET NAMES utf8
sql_query = SELECT `apps`.`id` * 1 + 0 AS `id` ,CAST
(`apps`.`given_names` AS CHAR) AS `given_names`,CAST(`apps`.`surname`
AS CHAR) AS `surname`,CAST(`apps`.`date_of_birth` AS CHAR) AS
`dob`,CAST(`apps`.`personalid` AS CHAR) AS `personalid`,CAST
(GROUP_CONCAT(DISTINCT `choices`.`decision` SEPARATOR ' ') AS CHAR) AS
`decision`, CAST(GROUP_CONCAT(DISTINCT `courseinstances`.`entry_year`
SEPARATOR ' ') AS CHAR) AS `app_year`, CAST(GROUP_CONCAT(DISTINCT
`courses`.`facultycode_id` SEPARATOR ' ') AS CHAR) AS `faculty`, CAST
(GROUP_CONCAT(DISTINCT `courses`.`coursetitle` SEPARATOR ' ') AS CHAR)
AS `Coursetitle`,CAST(`courses`.`campus` AS CHAR) AS
`campus`,`apps`.`id` AS `sphinx_internal_id`, 4045616687 AS
`class_crc`,'4045616687' AS `subclass_crcs`,0 AS
`sphinx_deleted`,IFNULL(`apps`.`given_names`, '') AS
`given_names_sort` FROM `apps` LEFT OUTER JOIN `choices` ON
choices.app_id = apps.id LEFT OUTER JOIN `courseinstances` ON
choices.courseinstance_id = courseinstances.id LEFT OUTER JOIN
`courses` ON courses.id = courseinstances.course_id WHERE `apps`.`id`
>= $start AND `apps`.`id` <= $end GROUP BY `apps`.`id` ORDER BY NULL
sql_query_range = SELECT IFNULL(MIN(`id`), 1), IFNULL(MAX(`id`), 1)
FROM `apps`
sql_attr_uint = sphinx_internal_id
sql_attr_uint = class_crc
sql_attr_uint = sphinx_deleted
sql_attr_str2ordinal = given_names_sort
sql_attr_multi = uint subclass_crcs from field
sql_query_info = SELECT * FROM `apps` WHERE `id` = (($id - 0) / 1)
}

index app_core
{
source = app_core_0
path = /var/www/rails/gaol/shared/sphinx/app_core
min_infix_len = 3
enable_star = 1
}

index app
{
type = distributed
local = app_core
}

Here's the section of the cap deploy script that restarts sphinx on
deploy

run "cd /var/www/rails/gaol/current; RAILS_ENV=production rake
thinking_sphinx:stop"
run "cd /var/www/rails/gaol/current; RAILS_ENV=production rake
thinking_sphinx:index INDEX_ONLY=true"
run "cd /var/www/rails/gaol/current; RAILS_ENV=production rake
thinking_sphinx:start"

[Matt Davies]

Another thing I've tried is adding the --pidfile switch to the searchd
command and changing the name of the file from
staging_production.sphinx.conf to production.sphinx.conf and all
mentions of it in the config files

still no dice :-(

2009/9/16 vanderkerkoff <ton...@gmail.com>:

[Jesse Newland]

I literally just finished setting this up, including using God to
manage the thinking_sphinx delayed_delta task. Here's how I have
things setup:

http://gist.github.com/188053

You might find it useful to set the 'log' attribute on a watch - any
output from the process started will be logged there. I bet you'll
find an error message in that log that leads you to the resolution of
your problem!

Regards -

Jesse Newland
---
je...@railsmachine.com
404.216.1093

[Matt Davies]

Hi Jesse

You'll never guess, adding the log attribute has showed that there's a
problem accessing the searchd.log file :-)

Thanks so much, I haven't worked the fix out yet but it's a hell of a
lot better than start command exited with non-zero code = 1

Thanks again.

Matt

2009/9/16 Jesse Newland <je...@railsmachine.com>:

[Matt Davies]

Just to let you know I got it all working Jesse, thanks a million for
your help, it's really appreciated.

I've based my solution on your example but my way is much less
advanced, more stupider I'd say :-)

Thanks again

Matt

2009/9/16 Matt Davies <ton...@gmail.com>: