running multiple required processes in a cloud foundry app container

[James Bayer]

it’s been awhile since we had this thread [1] about running multiple processes as a single app. this use case has come again recently. maybe you want to run an agent along side your app in a separate process. maybe you need redis embedded with your app. there are many reasons you might have a multi-process app.

the way cloud foundry apps work now is that the DEA is watching the Warden container for a single OS process per cloud foundry app, so if a parent process spins off 2 child processes A and B, and child processes A is required for child process B to work properly, then you want the DEA/Warden/Health Manager to restart the entire app in a new app container when any of the required child processes fail, not to wait until all child processes have finished.

david supplied a bash start.sh script [2] that can act as a parent process and watch child processes as well as a more polished solution using forego [3] (a golang version of foreman). forego does a nice job of aggregating the stdout/stderr for the child processes into the parent process stdout/stderr with nice color formatting. you can see this by running forego start in the eg dir example in the forego repo [4]. forego is certainly a nice elegant and simple solution, but you have to either package the 6MB forego binary with your app or have a buildpack that will either download it or compile it during the compile buildpack phase.

as a simple bash alternative to forego, i found that david’s linked example bash start.sh script did not actually exit when a single child processes exited, rather it waited until all child processes exited. so i put together a little bash script called startAndExitOnAnyChildExit.sh [5] that takes a lists of processes to run and exit all including the parent when any child processes exits. it uses SIGTERM first and gives all child processes 2 seconds to exit nicely, then sends SIGKILL for any child processes that didn't exit nicely.

example usage. let’s say you have two processes, a sleep process and a ticker process [6]. the ticker process is configured to ignore TERM signals, so you need to send it a KILL.
in terminal 1:
./startAndExitOnAnyChildExit.sh 'sleep 60' './ticker'
then kill the PID from the sleep in terminal 2, and you should something like this:

$ ./startAndExitOnAnyChildExit.sh 'sleep 60' './ticker'
Process "sleep 60" started with pid 58761
Process "./ticker" started with pid 58762
tick: [] -- FOO:
tick: [] -- FOO:
tick: [] -- FOO:
tick: [] -- FOO:
./startAndExitOnAnyChildExit.sh: line 26: 58761 Terminated: 15          $cmd
child process 58761 exited, killing all children and exiting
sending TERM to 58762
received SIGTERM but i'm ignoring it!
tick: [] -- FOO:
tick: [] -- FOO:
sending KILL to 58762

so you should be able to include this bash process in your cloud foundry app and then make the start command something like:
/app/startAndExitOnAnyChildExit.sh /app/child-process1.sh /app/child-process2.sh

[1] https://groups.google.com/a/cloudfoundry.org/d/msg/vcap-dev/OHbSH6gOAOQ/UU63UD_DHSYJ
[2] https://github.com/cityindex/logsearch-flowdock-bot/blob/1d961eb6829b3d682229a3cfd9b58fa3181a69c9/start.sh
[3] https://github.com/ddollar/forego
[4] https://www.evernote.com/shard/s3/sh/7aa5a17d-b5f9-4063-9199-8ad9a5da5bd1/d0898f2aac5458df61f3b53a4697d3af
[5] https://gist.github.com/jbayer/8368233

[6] https://github.com/ddollar/forego/blob/master/eg/ticker

--

Thank you,

James Bayer

[David Laing]

James,

Cool update!

I shall be rolling that back into my projects!
D

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

[Guillaume Berche]

Any one tried using supervisord [1] ? It seems to have nice subprocess mgt/restarts and log collection [2]. It seems to be 1 MB once installed [3] but requires python interpreter which I assume is much larger.

[1] http://supervisord.org/introduction.html
[2] http://supervisord.org/logging.html#child-process-logs
[3] http://packages.ubuntu.com/fr/lucid/supervisor

Guillaume.

[Glenn Oppegard]

We're successfully using foreman (the precursor to forego) to manage multiple processes (rails, nginx, golang app). I was concerned about multiple processes logging to stdout for loggregator, and that the output would be garbled. However, foreman manages all this and logs look great (also, they're pre-pended by the process name so you can filter which log came from which process).

I also wanted to let people know about a foreman gotcha that we had to figure out: it overwrites the PORT environment variable for each managed process beyond the first one, incrementing it by 100 [1]. As a workaround we're using the VCAP_APP_PORT environment variable instead.

[1] https://github.com/ddollar/foreman/blob/2b97cb458a56a066a28c123e819175dc49a82888/lib/foreman/engine.rb#L264

-Glenn

[Daniel Mikusa]

This seems like a nice workaround, however it’s still a workaround. It would be ideal if build pack authors don't need to worry about this. Starting multiple processes seems like a valid use case to me. With the PHP build pack, I do this to start both a web server and php-fpm. Just wondering if there any plans to allow running multiple processes from a single app where the DEA would handle this logic? If so, is there is a tracker story somewhere that I can watch?

Thanks

Dan

[James Bayer]

there are no current plans for application instances to support the concept of monitoring multiple processes for a single app instance.

it may be something the diego team looks at as they implement the updated architecture.

mark kropf is the PM and you can lobby him for the feature. personally, i don't find the available options too limiting that make this an urgent need as much as some of the other runtime improvements we want to make that hit more surface area. for example app "zero downtime deployments" built in by default instead of having a "blue/green" style deployment between different apps that share the same route. 

[Daniel Mikusa]

Thanks for the info. I was mostly trying to figure out if I should go ahead and use a workaround or wait for a fix. Since it’s not on the road map, I’ll just go with one of the workarounds.

I would still like to see this as a feature of the system, but it’s certainly not urgent. I’d consider it a nice to have.

@Mark - Do you think we could get a story for this, with the understanding that its a low priority?

Dan