Another option might be Celery. This is maintained by Pivotal, in the
person of Ask Solem cc'd here. It is popular for jobs work, and web
site creation. Celery is designed to work with Redis and Rabbit and
Mongo.... "take your pick".
Links
>
http://celeryproject.org/
>
https://devcenter.heroku.com/articles/background-jobs-queueing
Requirements:
> Durability
Depending on broker, with RabbitMQ can be transient or durable,
can also support RabbitMQ publisher confirms.
> Performance
Depending on broker, but with RabbitMQ and on my generic office desktop PC,
a single worker (1 thread) can do 100k jobs a second using
non-persistent messages.
> Scalability
Depending on broker, workers communicate by message-passing.
> Retry w/ back-off
Supported
> Scheduling/recurring jobs
Supported (one-off, interval or crontab expressions)
> Ability to query job status
Celery supports pluggable "result backends", which let you track the status
of a job. Many options supported, both RPC and persistent storage
(RabbitMQ, Redis,
SQL, …)
> Admin interface
Flower (
github.com/mher/flower) is a web-based real-time monitor and
admin interface
for Celery.
Monitoring is not the same as result backends, and workers can
enable/disable monitoring
at runtime.
Note that Celery does not let you modify the queue directly, e.g. you
cannot reorder messages in the queue, or delete a specific message.
Celery does let you "revoke" a message, so that the message is ignored
by workers, and you can implement most
of these operations without directly modifying the queue.
Jobs in Celery are considered a stream, i.e. they may be in-flight, in
the queue, or
reserved by a worker, so these operations are very difficult to
implement using direct access at scale.
> Database dependencies
Does not have any external service requirements except for what you
chose as a broker (and optionally a result backend).
> Queues local / non-local to the Cloud Controller box
Not sure what this means, but I suspect it depends on the broker used.
> Monitoring (statsd).
There are plugins for many monitoring tools, I think I have seen 3rd
party statsd scripts.
> Licensing
Celery is BSD licensed (3-clause)
Note that Celery is Python-centric, as this is the language it's written
in and what most of the users use it with.
It can be used from other languages using webhooks or by other means.
Native support for other languages is being planned (you can already
do this, but it does require an effort)
a