Where do you run your main process?

[John Lavoie]

Hi everyone,

We've been growing unhappy with the design of how we spin up our jobs and are considering making some changes.  I'm curious what everyone else does. The job process is the only single-point-of-failure in our pipeline and we're looking for more robustness as well as more decoupling and dynamic deployment capabilities.

Our app exposes a web service where users can submit requests for processing with various options.  We take those incoming requests and spawn new processes to execute the cascading workflow on the same app server that handles the request.   If there is any hiccup with the job process the entire workflow will fail.  And with some jobs running for multiple hours, restarting, even with checkpoints, is expensive and time consuming.

We are considering moving the the main process that starts our cascading workflow off of the app servers to somewhere else.  Where do you run yours? 

John Lavoie

[Ken Krugler]

Hi John,

We run our workflows on a smaller “controller” in AWS.

We typically wrap jobs with a FlowRunner (class in cascading.utils project) that gives us a way to monitor the workflow execution running in a separate thread.

— Ken

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at https://groups.google.com/group/cascading-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/a0c6e2c0-5cdf-4156-b06d-2c92a2ed1d97%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--------------------------

Ken Krugler

+1 530-210-6378

http://www.scaleunlimited.com

custom big data solutions & training

Hadoop, Cascading, Cassandra & Solr