Why we didn't use Delayed Job for Ruby Storm

[Stephen Creedon]

Hi all,

At the talk last Thursday I was asked why we opted for RabbitMQ over Delayed Job. It was a good question that didn't really get a good answer from me so I wanted to answer it here:

1. Delayed Job is mostly intended for serialising Ruby, and in particular ActiveRecord classes so that they can be de-serialised and used in a separate process. We work in hashes simply because we found it much faster to retrieve , use and store the hashes we convert from our XML (or sometimes Json) than to encapsulate each object in a class then serialise/deserialise as they are needed. This is especially true of ActiveModel classes with all of their associated callbacks.

2. Delayed Job simply allows a process to pop a job off of a queue then run it. RabbitMQ provides much more sophisticated methods for routing messages. You can:

• Split the workflow by sending one message to multiple workers. When data has been refined to our 'common' object then we send a message to our elastic search server, a message to our photograph processing service and a message to our web-app so that three actions result from a single message.
  
• Fan out the workflow in round-robin fashion by sending the message to one of a group of workers.
• You can assign a key to a message and route that message to specific workers based on the value of the key. E.g. if  key > 100 then send the message to worker X if it's < 50 send the message to worker Y etc.

In other words RabbitMQ can become a part of your business logic rather than simply a way of deferring a long running task to a separate process.

3. Last - but not least - at peak we've seen a single RabbitMQ queue handle over 2500 MESSAGES A SECOND - and that's just one of 8 queues so typically Rabbit is handling 3000-5000 messages a second for us in total. We haven't tested Delayed Job but I very much suspect it won't be able to compete on performance.

Having spent the last few years working on huge, creaking Rails apps that seem to generate argument and headache much more than fun we are on a mission to enable Rails and Ruby developers to break their apps into smaller components (rather than just talk about it). Our belief is that nothing should be so big you can't afford to throw it away and RabbitMQ has enabled us to keep things small, simple and always a pleasure; and a pleasure we want to share. If this interests you then do get in touch. We're close to releasing a blog on this subject so email me and we'll let you know when it's ready.

Best regards,

Steve Creedon