Queue Abstraction Layer

[Anthony Eden]

I want to take a moment to talk about the notion of a queue
abstraction layer in Rails and put a stake in the ground with my
opinions on the topic while at the same time getting some feedback.

My understanding is this: at the moment ActiveWorker (which can be
seen in this repo: http://github.com/joshbuddy/rails ) is the current
track for how to integrate workers (and queues to an extent). There's
nothing wrong with this approach, however my position is that workers
!= queues, and I don't necessarily want to define my workers in the
context of Rails, yet I do want an abstraction layer for queues. The
current design of ActiveWorker has the queues essentially as adapters,
similar to how ActiveRecord works, so I can understand leaving this as
part of ActiveWorker, but I do wonder if it's a good idea to couple
these things together. Personally I don't think it is but then again
others may disagree.

Here's my proposal: there should be a queue abstraction layer (call it
ActiveQueue if you like) with the following high-level requirements:

* It should provide a generic interface to queues (messages queues and
publish/subscribe)
* It should provide a standard interface for drivers to implement and
should provide a few basic driver implementations
* It should allow other drivers to be added in through simple
configuration (config/queue.yml)
* It should define an interface for serializing messages (and provide
JSON and XML serialization as the default?)
* It should NOT define how to handle work units (i.e. this is not for
defining asynchronous workers)
* It should be used under the hood by ActiveWorker

I actually have a need for this right now and would be willing to
implement it to scratch my own itch, but I didn't want to do so
without first getting some feedback from the core community to see if
this is both interesting and useful.

Thoughts?

Sincerely,
Anthony Eden

--
GMU/IT d- s: a32 C++(++++)$ UL@ P--- L+(++) !E W+++$ !N o? K? w--- !O
M++ V PS+ PE Y PGP t+ !5 X- R tv b++ DI+ D++ G- e++ h---- r+++ y++++**

http://anthony.mp

[Paul Campbell]

+1 on the concept ... I feel that Rails has been without this for a
while. With the recent surge of activity on queue services, I think
this is an awesome suggestion ... it would be great to have a standard
Rails-ish way of setting up multiple work queues.

I think that a good starting point for where to look for inspiration
is http://github.com/defunkt/resque#readme ... which has a ton of
really good design choices.

If there was a common interface to queues such as Resque that could be
abstracted in front of any choice of queue backend, I feel like it
would be a huge win for Rails.

Paul

> --
>
> You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
> To post to this group, send email to rubyonra...@googlegroups.com.
> To unsubscribe from this group, send email to rubyonrails-co...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=.
>
>
>



--


Paul Campbell
pa...@rushedsunlight.com
- - - - - - - - - - - - - - - - - - -
blog http://www.pabcas.com
twitter http://www.twitter.com/paulca
github http://www.github.com/paulca
phone +353 87 914 8162
- - - - - - - - - - - - - - - - - - -

[mbiker]

+1

I would love to see the related scheduled job be part of this as well
since some processes can only have a single instance running at a
time.

On Nov 22, 7:57 pm, Anthony Eden <anthonye...@gmail.com> wrote:
> I want to take a moment to talk about the notion of a queue
> abstraction layer in Rails and put a stake in the ground with my
> opinions on the topic while at the same time getting some feedback.
>
> My understanding is this: at the moment ActiveWorker (which can be

> seen in this repo:http://github.com/joshbuddy/rails) is the current

[Joshua Peek]

Interesting ideas.

I think it be "ActionQueue" since its not a model layer (not sure).

Maybe both belong in the same "pack" similar to ActionPack.

So we have some root package called "ActionBackgroundShit" and it
contains AW and AQ.

actionbackgroundshit/lib/action_worker/*
actionbackgroundshit/lib/action_queue/*

> --
>
> You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
> To post to this group, send email to rubyonra...@googlegroups.com.
> To unsubscribe from this group, send email to rubyonrails-co...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=.
>
>
>



--

Joshua Peek

[Amol Hatwar]

Queues != Model. Josh, you are right indeed. +1 from me for AW & AQ :-)

Cheers,

Amol

[Marc Byrd]

Seems to me it's analogous to caching from a service-oriented architecture pov - ActiveSupport::Queue like ActiveSupport::Cache ?  Interface/implements approach?  AMPQ-compliant interface?  Starling?  Also suggest that there may be different levels of the interface, starting with very simple transient push/pop all the way to full AMPQ. 

This would allow work around ActiveWorker to focus exclusively on worker challenges like polling, starvation, backoff, sleep, wake, deployment, running in context, etc.

m