CloudFoundry for long running non-web apps

[marcos campal]

Hi,

My name is Marcos Campal from Scrapinghub. We are overhauling our cloud platform where we run scrapy based jobs for our customers. Although we already have an in-house PaaS, we would like our next version to be based around open source. We are considering CloudFoundry, but we have some doubts regarding how well our platform fits into CFs model.

The primary use case we are trying to solve is (very simplified):

- A customer deploys an app that is built using buidpacks.

- He then schedules a job by selecting a built app and a set of arguments. We aim for very quick job start times.

- The job starts, executes and ends. Jobs should not be automatically restarted when they finish successfully (and probably even when they fail).

Please note that:

* We do need to run multiple instances of an application simultaneously (but with different job specifications).

* We can't afford to build and deploy an app when the customer schedules a job, it must already be deployed and ready to run.

Do you think CloudFoundry can be an appropriate platform to accommodate this use case?

Thanks in advance, 

Marcos

[Ben Hale]

Marcos,

I apologize for no one getting back to you before now; we really should be a bit better with community outreach.

I'm a little confused about a couple of your requirements.  You state:

- He then schedules a job by selecting a built app and a set of arguments. We aim for very quick job start times.

* We can't afford to build and deploy an app when the customer schedules a job, it must already be deployed and ready to run.

If I read your requirements right, it seems like you want an application that  sits resident waiting to run a job.  Once a user schedules that job, it is quickly executed by the application.  How do you plan on notifying the application that a new job has been scheduled?  Typically we see applications exposing REST-ful endpoints for this functionality (which means that the application is a web application) although this isn't a hard requirement.  For example, if you wanted to notify the application via a messaging system like RabbitMQ, you could start the application without binding a domain (and therefore without exposing an HTTP port).  If either of those are an accurate representation of your use-case, then Cloud Foundry should be able to do the job for you.

-Ben Hale

Cloud Foundry Java Experience