[noob] Where do we put code for long running process?

[Joshua Partogi]

Hi all,

I have another noobie question. After looking at last night
presentation about Rails 3, I am wondering where do we put piece of
code that runs on a separate thread for a long time every 5 seconds?
e.g

Thread.new do
while true
queue = check_whether_there_is_something_new_in_the_queue

while queue > 0
Trends.create(:data => queue.data)
end

sleep 5
end
end

1. I don't really like using cron to call `script/rails runner` as
that will load the whole rails library which will choke the server
because this process is run every 5 seconds.
2. I don't know about resque (which was mentioned by Ryan) yet, but
since resque is a separate webapp, would resque know anything about
the models in my Rails app?

Any input is highly appreciated.

Kind regards,
Josh

--
http://twitter.com/scrum8

[Warren Seen]

Yeah, don't write your own jobs queue... use something like resque, delayed_job, background_job, etc.

There's a reasonably good comparison here: http://4loc.wordpress.com/2010/03/10/background-jobs-in-ruby-on-rails/

> --
> You received this message because you are subscribed to the Google Groups "Ruby or Rails Oceania" group.
> To post to this group, send email to rails-...@googlegroups.com.
> To unsubscribe from this group, send email to rails-oceani...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/rails-oceania?hl=en.
>

[Ivan Vanderbyl]

Hi Josh,

This is a problem I had recently. Using Resque would probably be overkill, as would delayed_job. 

From what I gather you want to keep this script running in the background? What I would do is use something like bluepill (http://github.com/ivanvanderbyl/bluepill) and set it to daemonize your script, which could be a rake task as it would then be able to load up your entire rails stack (once) and have access to all your models etc. 

Ivan

On 13/10/2010, at 10:42 AM, Joshua Partogi wrote:

[Clifford Heath]

On 13/10/2010, at 10:42 AM, Joshua Partogi wrote:

> ...where do we put piece of

> code that runs on a separate thread for a long time every 5 seconds?

If you have the choice of server, I'd consider using Thin and doing
this using EventMachine timers.

Clifford Heath.

[Joshua Partogi]

Thanks Ivan,

This is sweet as it would only load the rails stack once and have
access to my models. Just like what I need. Let me try it out.

Cheers,
Joshua

[Ivan Vanderbyl]

No worries, if you run into any trouble getting it going let me know.

Ivan

[Joshua Partogi]

Hi Cliff,

Thanks for the suggestion. Didn't thought of that also. My next
question would be, where do I put the code to start the em timers?
Initializers? Thanks for the enlightment.

Kind regards,
joshua.

--
http://twitter.com/scrum8

[Clifford Heath]

On 13/10/2010, at 10:59 AM, Joshua Partogi wrote:
> Thanks for the suggestion. Didn't thought of that also. My next
> question would be, where do I put the code to start the em timers?
> Initializers? Thanks for the enlightment.

It's a good question - I've done this stuff in async_sinatra not Rails
yet.
See my fork of that gem, there's an example of using an EM::Channel.

I'd start it in initializers (check that works though!) but I think
I'd want to
keep some global indicator of the timer being set, so that any code
which
accesses the Trends can verify that it's still running, and kicks it
off again.

Come to that, you might be able to start it on first use and just keep
it
running... depending on whether you need can or need to create the
interim data retroactively.

Clifford Heath.
http://github.com/cjheath

[Warren Seen]

Seriously?

IMO, it's overkill to write your own jobs queue management, plus a worker, rather than just using delayed_job's queue and worker - or any other lib for that matter. 

Anything else is reinventing the wheel, only your wheel probably ends up being an octagon or something less than round...

[Ben Hoskings]

I'd consider using delayed_job. I don't think it's overkill. Polling sucks (at least, app-exposed polling does), and with delayed_job you can queue your trend creation in the background when the appropriate thing happens, instead of continually checking if it's required.

def trendy_action
something_trendy
Trend.send_later :data => data
end

Using threads in any way within a rails instance is a bad idea, because it will cause problems with passenger and friends. It might seem to work in development for a bit, but you'll definitely have problems down the road.

―ben_h

[Ivan Vanderbyl]

Disagree, at the core of it he is doing exactly what delayed_job does to look for jobs (polls database every 5 seconds), except if using delayed job he would then have that overhead. Sure there is no reason why you couldn't use delayed job and reschedule the job after each run. 

However delayed job like any background task has the possibility of crashing, which then brings you back to something like bluepill to restart your workers.

Matter of taste.

Ivan

[Warren Seen]

Perhaps I wasn't entirely clear.

I'm not suggesting just sticking the polling of the queue into DJ, I'm suggesting replacing it entirely.

Instead of putting data into the queue to begin with, all you do is make a call with send_later. You then get rid of a whole bunch of queue management code you need to test and maintain, and get back to what I understand is the crux of the problem... offloading a long-running task into the background.The tradeoff to the DJ overhead is not having to manage the work queue. 

(Sidenote: We're talking about rails... isn't it a bit late to start worrying about overhead NOW? :D)

For an inexperienced developer (The subject does say [noob]), implementing your own job queue (unless you can clearly describe why existing queues don't meet your needs) is not a matter of taste, it's a code smell. 

Cheers,

Warren

[Joshua Partogi]

HI Warren,

Thanks for the feedback. From what I understand delayed_job puts in a
job in a queue with the assumption that the apps that created the job
is the same rails apps that is going to consume the queue. CMIIW. My
problem is the queue is created by another external system that is
non-rails. At this point I want to evaluate bluepill and see how it
fits.

Cheers,
Josh.

--
http://twitter.com/scrum8

[Joseph Rodriguez]

+1 for EM + bluepill

+++ lib/trend_muncher.rb +++

EventMachine::PeriodicTimer.new(5) do

queue = check_whether_there_is_something_new_in_the_queue
while queue > 0

logger.info "Munching #{queue.size} trends"

Trends.create(:data => queue.data)
end

end


+++ config/muncher.pill +++

RAILS_ROOT = File.expand_path("#{File.dirname(__FILE__)}/..") unless
defined?(RAILS_ROOT)

Bluepill.application("MuncherEnvironment") do |app|
app.working_dir = RAILS_ROOT

app.process("TrendMuncherDaemon") do |process|
process.start_command = "ruby lib/trend_muncher.rb"
process.stop_command = "kill -INT {{PID}}"
process.pid_file = RAILS_ROOT + "/tmp/pids/trend_muncher.pid"
process.stdout = RAILS_ROOT + "/log/muncher.log"
process.daemonize = true

process.checks :flapping, :times => 2, :within =>
30.seconds, :retry_in => 60.seconds
process.checks :mem_usage, :every => 10.seconds, :below =>
150.megabytes, :times => [3,5]
end
end

and then...

`sudo bluepill load config/muncher.pill`

Good luck!