Control how many process runs based on conditions

[christian.s.roy]

Hello all,

I have a queue (I am using redis) with things pilling up.  Sometimes things gets pilled up real fast and sometimes real slow.

I have a ruby script that loops forever and empty the queue, one item at a time.

When there is less than 5000 in the queue, I want only one process.
When there is between 5000 and 9999 items in the queue, I want two process until the queue is down to 4000 or less.
When there is between 10000 and 14999 items in the queue, I want three process until the queue is down to 8000 or less.
and so on...

I want the number of processes to go up and down depending on how many things are in the queue.  Is it possible with god?  If not, is there an alternative or a way around?

Here is my god config file that was hoping to work but, I cannot use :stop in transition  (E [2012-09-06 14:00:18] ERROR: Invalid state :stop. Must be one of the symbols :init, :up, :start, :restart)

rails_env   = ENV['RAILS_ENV']  || "production"
rails_bin   = `which rails`.chomp
rails_root  = ENV['RAILS_ROOT'] || File.dirname(File.dirname(File.realdirpath(__FILE__)))

require File.expand_path('environment.rb', File.dirname(__FILE__))

num_process = 5
num_process.times do |num|
  God.watch do |w|
    w.name = "process_queued_track_hits-#{num}"
    # w.start = "#{rails_root}/script/process_queued_track_hits.sh ${rails_env}"
    w.start = "exec #{rails_bin} runner -e '#{rails_env}' 'Bonus.process_queued_track_hits'"
    w.env = {
      'RAILS_ROOT' => rails_root,
      'RAILS_ENV' => rails_env
    }
    w.dir = rails_root
    w.interval = 30.seconds

    # restart if memory gets too high
    w.transition(:up, :restart) do |on|
      on.condition(:memory_usage) do |c|
        c.above = 350.megabytes
        c.times = 2
      end
      on.condition(:cpu_usage) do |c|
        c.above = 50.percent
        c.times = 5
      end
      on.condition(:cpu_usage) do |c|
        c.above = 80.percent
        c.times = 2
      end
    end

    # determine the state on startup
    w.transition(:init, { true => :up, false => :start }) do |on|
      on.condition(:process_running) do |c|
        c.running = true
      end
    end

    # determine when process has finished starting
    w.transition([:start, :restart], :up) do |on|
      on.condition(:process_running) do |c|
        c.running = true
        c.interval = 5.seconds
      end

      # failsafe
      on.condition(:tries) do |c|
        c.times = 5
        c.transition = :start
        c.interval = 5.seconds
      end
    end

    # start if process is not running
    w.transition(:up, :start) do |on|
      on.condition(:process_running) do |c|
        c.running = false
      end
    end

    w.stop_if do |on|
      on.condition(:lambda) do |c|
        c.interval = 30.seconds
        c.lambda = lambda do
          $redis.llen("track_hits_queue") < num * 4000
        end
      end
    end

    w.transition([:stop, :init], :start) do |on|
      on.condition(:lambda) do |c|
        c.interval = 30.seconds
        c.lambda = lambda do
          $redis.llen("track_hits_queue") > num * 5000
        end
      end
    end

  end
end

If not possible, I can pass a parameter to  Bonus.process_queued_track_hits(#{num * 5000}) so that all process always run and my loop check if its time to start emptying the queue or not.

Thanks.