High CPU usage when Upstart starts Thinking Sphinx

[Andrew Stewart]

(I tried to post this to the CPU Usage? topic but Google Groups
doesn't seem to allow me.)

I use Foreman to export an Upstart configuration for Thinking Sphinx.
Here's the line from my Procfile:

thinking_sphinx: bundle exec rake thinking_sphinx:start
NODETACH=true

I'm using Thinking Sphinx 2.0.10.

Then on my server, when I do `ps wwaux | grep thinking_sphinx`, I see
this:

rails 22942 0.0 0.1 35636 1244 ? Ss 10:37 0:00
su - rails -c cd /var/www/apps/sparkle/releases/20111118082310; export
PORT=5100; export RAILS_ENV=production; bundle exec rake
thinking_sphinx:start NODETACH=true >> /var/www/apps/sparkle/shared/
log/thinking_sphinx-1.log 2>&1
rails 22943 0.2 0.2 19780 2308 ? S 10:37 0:00 -
su -c cd /var/www/apps/sparkle/releases/20111118082310; export
PORT=5100; export RAILS_ENV=production; bundle exec rake
thinking_sphinx:start NODETACH=true >>
/var/www/apps/sparkle/shared/log/thinking_sphinx-1.log 2>&1
rails 23041 97.3 6.6 251448 67900 ? Rl 10:37 0:12
ruby /var/www/apps/sparkle/shared/bundle/ruby/1.9.1/bin/rake
thinking_sphinx:start NODETACH=true

The CPU is at 97.3% (3rd row, 3rd column). It never seems to come
down from the high nineties.

Please can you help me fix this?

Thanks in advance,

Andy Stewart

[Pat Allan]

Hi Andrew

What happens if you run searchd manually instead of via the rake task?

searchd --config path/to/production.sphinx.conf --nodetach

--
Pat

> --
> You received this message because you are subscribed to the Google Groups "Thinking Sphinx" group.
> To post to this group, send email to thinkin...@googlegroups.com.
> To unsubscribe from this group, send email to thinking-sphi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/thinking-sphinx?hl=en.
>

[Andrew Stewart]

Hi Pat,

On 18 Nov 2011, at 12:13, Pat Allan wrote:
> What happens if you run searchd manually instead of via the rake task?
>
> searchd --config path/to/production.sphinx.conf --nodetach

I patched my Upstart configuration to run searchd directly instead of via Rake...it worked perfectly and barely took any CPU at all.

(I also realised that my pid_file location in sphinx.yml was unwriteable; once I'd fixed that, Sphinx was able to run. Maybe that was pinning the CPU?)

When I revert to the Rake method the CPU starts at >95% but declines over several minutes. After 5 minutes my process list looks like:

$ ps wwaux | grep sphinx
rails 31907 0.0 0.1 35636 1240 ? Ss 11:39 0:00 su - rails -c cd /var/www/apps/sparkle/releases/20111118082310; export PORT=5100; export RAILS_ENV=production; bundle exec rake thinking_sphinx:start NODETACH=true >> /var/www/apps/sparkle/shared/log/thinking_sphinx-1.log 2>&1
rails 31912 0.0 0.2 19780 2312 ? S 11:39 0:00 -su -c cd /var/www/apps/sparkle/releases/20111118082310; export PORT=5100; export RAILS_ENV=production; bundle exec rake thinking_sphinx:start NODETACH=true >> /var/www/apps/sparkle/shared/log/thinking_sphinx-1.log 2>&1
rails 32048 4.8 8.1 285084 83512 ? Sl 11:39 0:16 ruby /var/www/apps/sparkle/shared/bundle/ruby/1.9.1/bin/rake thinking_sphinx:start NODETACH=true
rails 32136 0.0 0.0 4144 564 ? S 11:39 0:00 sh -c /usr/local/bin/searchd --pidfile --config "/var/www/apps/sparkle/releases/20111118082310/config/production.sphinx.conf" --nodetach
rails 32139 0.0 0.4 37828 5040 ? S 11:39 0:00 /usr/local/bin/searchd --pidfile --config /var/www/apps/sparkle/releases/20111118082310/config/production.sphinx.conf --nodetach

So the CPU is down to 4.8%. Much better than 95% but not as good as bypassing Rake altogether. Speaking of Rake, I wonder why it is still hanging around?

Cheers,
Andrew

[Pat Allan]

I need to change the detach version of the task to use Kernel.exec, I think - that way, the process changes to searchd, instead of searchd being run within rake.

If you want to have a stab at patching, you're certainly welcome to give it a shot :)

--
Pat

[Andrew Stewart]

On 18 Nov 2011, at 12:52, Pat Allan wrote:
> I need to change the detach version of the task to use Kernel.exec, I think - that way, the process changes to searchd, instead of searchd being run within rake.
>
> If you want to have a stab at patching, you're certainly welcome to give it a shot :)

I'll try to give it a go later today...

Thanks for the help!

Cheers,
Andrew

[dkam]

For other people who are searching for an upstart script, this is what I used.  Any improvements recommended? 

-------

description "Sphinx Upstart script"

start on startup

stop on shutdown

console output

respawn

script

    env RAILS_ENV=production

    exec sudo -u deployment_user -i /usr/local/sphinx/bin/searchd --config /var/www/site/config/production.sphinx.conf --nodetach

end script