Introducting Kaboose Queue

[Mazdak Rezvani]

= Kaboose Queue (kabqueue)

The Kaboose Queue system is designed to handle asynchronous generic
tasks (or messages). The main function of this system is to take the
load off the request cycle. We use this system to send out e-mail,
perform intensive image or video tasks, asynchronous network
operations, etc.

We had a few important things in mind when desiging this system:
* Performance
* Scalibility
* Reliability

The first item is obvious. The system needs to be fast and relatively
resource friendly. For this reason, we chose the Starling queue system
from Twitter (www.twitter.com).
The second item, scalibility, means that the system should:
* Be distributed. Many machines submitting tasks, and many machines
processing these tasks.
* Handle a large number of requests. Each photo upload spawns resize
operations for frequently used sizes. Each resize spawns off
additional S3 uploads, for example.
The third item requires the system to handle failures, retry common
errors, and notify us in case of major failures. Also, the system
needs to surive the server dying and coming back and temporary network
errors.

With the help of Starling, and our previous experience with a database-
backed message queue system, I think we've achieved our goals. We look
forward to the community's feedback.

== Requirements

Starling:
sudo gem install starling
sudo gem install daemons

== Installation

script/plugin install http://kabqueue.googlecode.com/svn/trunk/kabqueue

== Example usage:
First, you will need to create a processor file(+foo.rb+) in
+RAILS_ROOT/app/processors+ folder. It should be structured like so:

class FooProcessor < Kaboose::Processor

processes :some_model

def process
some_model.some_method
end
end
end

Each processor class *must* implement a process method that defines
the action that needs to be processed by the queue.
This method has access to the _@task_ instance variable that is an
instance of Kaboose::Task. The processes macro creates an accessor
method as a shortcut for accessing
ActiveRecord models specified by model_id option in the task. See
Kaboose::Processor's self.processes for more info.

== Configuration
You will need a kqueue.yml file in your apps config folder to specify
the address and namespace of the system, for example:

address: 127.0.0.1:22122
namespace: some_namespace

== Running the Kaboose Queue system
Just run ./script/queue_processor to get a list of options.

If you are using monit, here's what worked for us:

check process queue-processor with pidfile /path/to/
queue_processor.pid
group qtp
start program = "PATH/queue_processor start -d -e production -c
CWD -u USER -g GROUP"
stop program = "PATH/queue_processor stop -d -e production -c CWD"

== Contact Info, Credits

Blog: http://labs.kaboose.com/

Google Code Page: http://code.google.com/p/kabqueue/

Google Group: http://groups.google.com/group/kaboose_labs

By: Mazdak Rezvani. Kaboose Inc. (www.kaboose.com)

Contributors: Ahmed El-Daly, Talha Syed, Narsa Chelluri

Original non-starling
version: Terry Gregory, Mazdak Rezvani. BubbleShare
(www.bubbleshare.com)