This topic has been hidden because it was flagged for abuse.

Showing 1-17 of 17 messages
This message has been hidden because it was flagged for abuse.
Re: [vcap-dev] Background job system for Cloud Controller Matthew 7/25/13 10:39 PM
bosh has been using resque with rufus scheduler for a while and it's been working well. It's a part of the infrastructure that quietly works and doesn't require much attention.
Re: Background job system for Cloud Controller David Williams 7/26/13 6:41 AM
A cold chill went down my spine when you mentioned LGPL. :)  My vote would be Resque based on Matthew's comments about it already being parts of the infrastructure today and being MIT licensed.  In the spirit of not deviating from cf-release's currently licensing model, CF components should be Apache, MIT, or BSD (no LGPL or GPL).

Is there any use case or requirement that Resque doesn't meet?
Re: Background job system for Cloud Controller Luke Bakken 7/26/13 7:11 AM

If it's good enough for GitHub ....
This message has been hidden because it was flagged for abuse.
Re: Background job system for Cloud Controller 7/26/13 7:37 AM
Author of Sidekiq here.

You'll find Sidekiq to be MUCH faster than Resque and far more RAM efficient, we're talking 10-20x.  Sidekiq has a lot more functionality built in and all integrated cleanly together.  The documentation is also far more extensive.

I have every intention of supporting Sidekiq for years to come because of that fancy Pro version you mentioned.

Hope this helps.

PS MRI 1.9+ uses native threads, just with a GIL.

Re: [vcap-dev] Background job system for Cloud Controller Jesse Zhang 7/26/13 8:58 AM
On the top of my head Resque sounds like a better fit because it's forking. We've had terrible experience with Ruby handling lots of string objects (send_file anyone?), and worked around the issue by fronting the cloud controller with nginx. Uploading files sounds precisely like that. The last thing we want is something that "works in dev" and passes all the tests but leaves a large chunk of uncollected memory in the main process on an unexpected Saturday 3 weeks later.

Cloud Foundry

On Thu, Jul 25, 2013 at 10:10 PM, Matthew Kocher <> wrote:

Re: Background job system for Cloud Controller 7/26/13 9:40 AM
To add more fuel to the fire, check out Qless -

Its new but has most of your requirements already built in.  It uses the lua scripting built into redis to ensure that its data structures remain consistent in highly concurrent environments, so ++ on your reliability requirement

I'm in the process of replacing resque with it, and so far am very happy (not in production yet, but we process about 6M resque jobs a day).
It uses the fork model of resque, which I like due to better isolation between jobs, but shouldn't be too hard to do things in a threaded fashion if you prefer.

I've ported a couple of the resque plugins we use to it as well:

Re: Background job system for Cloud Controller 7/26/13 10:36 AM
Resque maintainer here.

Just so you're all aware, Resque is working on a big 2.0 release, so things are changing. That said, huge companies use Resque to run a zillion jobs per day, just like Sidekiq. We do have a much, much bigger installed base. Compnies like GitHub, 37signals, LivingSocial, and many more. At a conference last week, a user told me about how they do a few thousand jobs per minute through Resque.

That said, Mike's comment about memory usage is true: you will use less memory with Sidekiq, due to threading. Resque 2.1 is slated to have N:M processes:threads, which will let you choose whatever model you want. I still think the forking model is more useful by default. His comment about the documentation is also unfortunately true at the moment, though we recently achieved 100% API documentation on master, and are now working on guide-style docs.

If there's anything I can do to help, please let me know. Whichever choice you pick, both Resque and Sidekiq are great, so you're in good hands either way.
Re: [vcap-dev] Re: Background job system for Cloud Controller Alexis Richardson 7/26/13 1:02 PM
Another option might be Celery.  This is maintained by Pivotal, in the
person of Ask Solem cc'd here.  It is popular for jobs work, and web
site creation.  Celery is designed to work with Redis and Rabbit and
Mongo.... "take your pick".



> Durability
Depending on broker, with RabbitMQ can be transient or durable,
can also support RabbitMQ publisher confirms.

> Performance
Depending on broker, but with RabbitMQ and on my generic office desktop PC,
a single worker (1 thread) can do 100k jobs a second using
non-persistent messages.

> Scalability
Depending on broker, workers communicate by message-passing.

> Retry w/ back-off

> Scheduling/recurring jobs
Supported (one-off, interval or crontab expressions)

> Ability to query job status
Celery supports pluggable "result backends", which let you track the status
of a job.  Many options supported, both RPC and persistent storage
(RabbitMQ, Redis,
SQL, …)

> Admin interface
Flower ( is a web-based real-time monitor and
admin interface
for Celery.

Monitoring is not the same as result backends, and workers can
enable/disable monitoring
at runtime.

Note that Celery does not let you modify the queue directly, e.g. you
cannot reorder messages in the queue, or delete a specific message.
Celery does let you "revoke" a message, so that the message is ignored
by workers, and you can implement most
of these operations without directly modifying the queue.
Jobs in Celery are considered a stream, i.e. they may be in-flight, in
the queue, or
reserved by a worker, so these operations are very difficult to
implement using direct access at scale.

> Database dependencies
Does not have any external service requirements except for what you
chose as a broker (and optionally a result backend).

> Queues local / non-local to the Cloud Controller box
Not sure what this means, but I suspect it depends on the broker used.

> Monitoring (statsd).
There are plugins for many monitoring tools, I think I have seen 3rd
party statsd scripts.

> Licensing
Celery is BSD licensed (3-clause)

Note that Celery is Python-centric, as this is the language it's written
in and what most of the users use it with.
It can be used from other languages using webhooks or by other means.

Native support for other languages is being planned (you can already
do this, but it does require an effort)

This message has been hidden because it was flagged for abuse.
Re: [vcap-dev] Re: Background job system for Cloud Controller Christopher Ferris 7/27/13 4:44 AM
I must admit to the same reaction to the mention of LGPL. Of course, it depends on a number of considerations as to whether it really presents an issue. My preference would be to explore alternative options to an LGPL licensed component.

Sent from my iPhone
Re: Background job system for Cloud Controller Brian Martin 7/29/13 9:56 AM

On Friday, July 26, 2013 1:10:52 AM UTC-4, Matthew Kocher wrote:
Hi All,

Were looking to add a background job / queue system to the Cloud Controller. The current use case for this is managing the complex lifecycle of Services which occur via 3rd party web transactions, which obviously fail occasionally. Further use cases could include staging & starting in the background, scheduled jobs, moving some processes that are a better fit for the background job model out of the event driven code, etc.

How do you envision this complex service lifecycle creation integrating with the current synchronous API for create-service?   In other words, today I can create a service synchronously on the cli using the create-service verb or during push.   What if my service creation does fall into the category of something more complex that requires a sort of mini-workflow or possible retries as you have stated.   Would these services enter into some sort of "pending" state until it completed creation?  Would service binding be deferred until creation could succeed?   Would service binding also have instances where this same queueing system could be utilized?

Brian Martin
This message has been hidden because it was flagged for abuse.
Re: [vcap-dev] Re: Background job system for Cloud Controller Dr Nic Williams 7/29/13 11:14 PM
Thanks Matt for this process & the update.
Dr Nic Williams
Stark & Wayne LLC - the consultancy for Cloud Foundry
+1 415 860 2185
twitter: drnic
This message has been hidden because it was flagged for abuse.
Re: [vcap-dev] Re: Background job system for Cloud Controller 7/30/13 10:51 AM
Great! As I mentioned before, please let me know if I can help in any way.