[erlang-questions] managing OS processes

[Benoit Chesneau]

I'm trying to find a way to manage some os processes launched from
Erlang. For example I need to relaunch them when the external process
die for an unknown reason or send an HUP signal or just stop the
external process.

For now I'm using erlang ports to open an external process and detect
when it die. I also send signals using the `os:cmd` and and always
send a SIGKILL signal using the `os:cmd`before closing the port. But
is there a better way to do that? Maybe a module somewhere ? I saw
erlexec [1] but it doesn't seem to be maintained since a while.


- benoît


[1] http://code.google.com/p/erlexec/
_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

[Gleb Peregud]

On Sat, May 26, 2012 at 11:06 AM, Benoit Chesneau <bche...@gmail.com> wrote:
> I'm trying to find a way to manage some os processes launched from
> Erlang. For example I need to relaunch them when the external process
> die for an unknown reason or send an HUP signal or just stop the
> external process.

Hello Benoit

Looks like there are two of us, who needs a better mechanism to manage
OS processes from Erlang. I need it for a CI server which I'm working
on, so my requirements are not as high as yours, but still os:cmd
doesn't work for me and ports are missing one feature which I need.

Maybe we could reanimate erlexec together?

Cheers,
Gleb

[Tim Watson]

I'm also working on very similar requirements in
https://github.com/nebularis/systest/blob/master/src/systest_cli.erl
and like you two, I have to rely on a combination of open_port with
its constituent events and os:cmd/1 in order to SIGHUP/SIGKILL which
feels rather 'hackish' to me. I've also noticed that it makes consumer
code rather timing dependent, which is frustrating.

I would be willing to participate in either

(a) improving the built in port functionality or
(b) working on erlexec

Although C I'm fairly comfortable with, but C++ (which erlexec appears
to be written in) isn't really my bag. I do have to ask though, is
erlexec in need of much additional work? I also have 'client' modules
for the OTP slave and ssh modules and don't want to invest a mountain
of time in replacing open_port, but a little TLC to bring erlexec up
to scratch would be time well spent if it would give a more consistent
interface. In particular I have some issues with open_port based code
with now such as

1. cli clients my code launches can be 'detached' (usually erlang
nodes run with -detached or scripts that launch erlang nodes using `sh
-C` or setsid/nohup and the like), which makes the port handling code
both complex and messy.
2. some client code I'm testing will execute another (different)
script/command to shut down the node and the listening port doesn't
always get the exit status properly
3. signal handling is a mess

So erlexec does sound promising, but IMO it would be *better* if the
built in open_port functionality could be extending to deal with these
situations properly.

Cheers,
Tim

[Anthony Molinaro]

I've found a need for this sort of thing as well. +1 for getting something
into OTP itself. I'm sure the real issue will be with Windows, although
I believe there was some code in couchdb for doing some control of windows
processes, and maybe erlexec already supports Windows? Anyway, I can't
say I'd do much other than testing, but just wanted to voice my support.

-Anthony

--
------------------------------------------------------------------------
Anthony Molinaro <anth...@alumni.caltech.edu>

[Tim Watson]

On 26 May 2012 23:00, Anthony Molinaro <anth...@alumni.caltech.edu> wrote:
> I've found a need for this sort of thing as well.  +1 for getting something
> into OTP itself.   I'm sure the real issue will be with Windows, although
> I believe there was some code in couchdb for doing some control of windows
> processes, and maybe erlexec already supports Windows?  Anyway, I can't
> say I'd do much other than testing, but just wanted to voice my support.
>

So I really do prefer the idea of patching the OTP mechanism, but I
think it's worth those of us who're consider it discussing (a) what is
missing that we'd like to have and (b) the general approach, with the
OTP team before doing this. I do not want to spend a month building
some patch that will just be rejected - I'd rather know up front if
it's likely to be accepted and what kind of approach would be
reasonable.

That is the flip side of this - it would be better to have the feature
in OTP proper, but it introduces risk to change the emulator and
having *normal* port process handling erlexec is nice and safe. But
not so clean.

[Zabrane Mickael]

Quick look to "erlexec" code shows that's only support Linux for now (with dependency on libpcap).
Any revival work should extend "erlexec" to all other OSes when Erlang is expected to run (ex. Windows, OSX, FreeBSD ...) and try to minimize external dependencies.

+1

I would be willing to help for a complete rewrite of "erlexec" (as Tim, I'm more C than a C++ guy).

Regards,
Zabrane

[Gleb Peregud]

On Sun, May 27, 2012 at 12:48 AM, Zabrane Mickael <zabr...@gmail.com> wrote:
> Quick look to "erlexec" code shows that's only support Linux for now (with dependency on libpcap).
> Any revival work should extend "erlexec" to all other OSes when Erlang is expected to run (ex. Windows, OSX, FreeBSD ...) and try to minimize external dependencies.
>
> +1
>
> I would be willing to help for a complete rewrite of "erlexec" (as Tim, I'm more C than a C++ guy).

Yes. In a long run improving open_port would be much better.

The questions are - how to make it cross platform? Would OTP team
accept such patches? Would OTP team accept not fully cross platform
patch?

Features which I feel are missing from oper_port:
- receiving stderr and stdout separately
- sending POSIX signals
- setting startup ENV
- setting startup working directory
- setting OS process priority
- monitoring of ports

Anything else?

[Tim Watson]

On 27 May 2012, at 10:44, Gleb Peregud <gleb...@gmail.com> wrote:

> On Sun, May 27, 2012 at 12:48 AM, Zabrane Mickael <zabr...@gmail.com> wrote:
>> Quick look to "erlexec" code shows that's only support Linux for now (with dependency on libpcap).
>> Any revival work should extend "erlexec" to all other OSes when Erlang is expected to run (ex. Windows, OSX, FreeBSD ...) and try to minimize external dependencies.
>>
>> +1
>>
>> I would be willing to help for a complete rewrite of "erlexec" (as Tim, I'm more C than a C++ guy).
>
> Yes. In a long run improving open_port would be much better.
>
> The questions are - how to make it cross platform? Would OTP team
> accept such patches? Would OTP team accept not fully cross platform
> patch?
>
> Features which I feel are missing from oper_port:
> - receiving stderr and stdout separately
> - sending POSIX signals
> - setting startup ENV

You can do this with open port already.

> - setting startup working directory

You can do this with open port already.

> - setting OS process priority
> - monitoring of ports

Yes the messages between the port and the controlling process could be richer and more robust.

[Tim Watson]

Guys - I think there is code in https://github.com/hawk/lux that
addresses some of these issues.....

[Michael Uvarov]

> - sending POSIX signals
Is there an example of this?

[Tim Watson]

No you can't do this with the current open_port implementation - you
can do os:cmd("kill -" ++ Signal ++ " " ++ OsPid) but that's about it.
I think in the code for lux there may be some signal handling, and
there appears to be support for it in erlexec.

[Michael Uvarov]

> os:cmd("kill -" ++ Signal ++ " " ++ OsPid)

It is platform-dependent.
And there is no way to convert Port into OsPid. Only way is to "ask"
the OS program using stdio.
"with the current open_port implementation" it is easy to call PHP or
python to control processes :)

--
Best regards,
Uvarov Michael

[Tim Watson]

On 28 May 2012, at 10:54, Michael Uvarov wrote:

>> os:cmd("kill -" ++ Signal ++ " " ++ OsPid)
> It is platform-dependent.
> And there is no way to convert Port into OsPid. Only way is to "ask"
> the OS program using stdio.
> "with the current open_port implementation" it is easy to call PHP or
> python to control processes :)
>

Yes absolutely - nobody is saying this is correct. We *clearly* need a proper way to handle sending signals. BTW someone mentioned off-list that the name erlexec clashes with a built in executable. Perhaps as well as forking the project, it would be a good idea to rename it.....

[Daniel Goertzen]

There is at least one portable C++ process management library out
there that could be leveraged...

http://www.highscore.de/boost/process/process/introduction.html

... and nifpp might be a good way to talk to it
(https://github.com/goertzenator/nifpp)


This might be a quick and dirty way to get it done, but this probably
not what you want for something that is shooting for inclusion into
OTP. The library source should at least provide some good
implementation guidance.

Dan.

[Tim Watson]

On 28 May 2012, at 15:13, Daniel Goertzen <daniel....@gmail.com> wrote:

> There is at least one portable C++ process management library out
> there that could be leveraged...
>
> http://www.highscore.de/boost/process/process/introduction.html
>
> ... and nifpp might be a good way to talk to it
> (https://github.com/goertzenator/nifpp)
>
>

What I'd really like is native processes, one per externally managed pid, wrapped around a native sub process management api for each supported platform. Monitoring becomes simple that way, as once the native process dies you put the exit code into the 'EXIT' message unless it is zero. Signal handling should be do-able too and I/O will be more erlang-ish.

[Michael Uvarov]

Maybe I will write a wrapper around Boost.process as a driver.

[Richard O'Keefe]

On 29/05/2012, at 4:55 AM, Tim Watson wrote:

>
> What I'd really like is native processes, one per externally managed pid, wrapped around a native sub process management api for each supported platform. Monitoring becomes simple that way, as once the native process dies you put the exit code into the 'EXIT' message unless it is zero. Signal handling should be do-able too and I/O will be more erlang-ish.

+1

[Ciprian Dorin Craciun]

On Sat, May 26, 2012 at 3:12 PM, Gleb Peregud <gleb...@gmail.com> wrote:
> On Sat, May 26, 2012 at 11:06 AM, Benoit Chesneau <bche...@gmail.com> wrote:
>> I'm trying to find a way to manage some os processes launched from
>> Erlang. For example I need to relaunch them when the external process
>> die for an unknown reason or send an HUP signal or just stop the
>> external process.
>
> Hello Benoit
>
> Looks like there are two of us, who needs a better mechanism to manage
> OS processes from Erlang. I need it for a CI server which I'm working
> on, so my requirements are not as high as yours, but still os:cmd
> doesn't work for me and ports are missing one feature which I need.
>
> Maybe we could reanimate erlexec together?
>
> Cheers,
> Gleb

Hello all!

I don't think its the right way (although it seems to be suggested
by the Erlang manual or somewhere else I've read it), but the way I
did it was to write a small C program that is the one "babysitting"
the real process, and a small Erlang wrapper.

The code is here (Apache 2.0 licensed):
https://github.com/cipriancraciun/mosaic-node/tree/development/applications/mosaic-harness/sources
https://github.com/cipriancraciun/mosaic-node/blob/development/applications/mosaic-harness/sources/mosaic_harness_backend.erl
https://github.com/cipriancraciun/mosaic-node/blob/development/applications/mosaic-harness/sources/mosaic_harness.c

The C program allows the following:
* it uses a small protocol based on JSON to communicate with the
Erlang wrapper, using solely `stdin` / `stdout` between itself and
Erlang, and between itself and the "babysitted" program;
* it allows a single process to be executed at a time (controlling
environment, arguments, and working directory); (it could be extended
to multiple process in parallel if wanted;) (but you can run it again
if you want;)
* it allows a signal to be sent;
* it allows "exchange" messages to be sent to and from Erlang and
the process;
* it allows brutal termination; (i.e. SIGKILL;)
* if `stdin` is closed by the Erlang side it kills the process;
* if `stdin` is closed by the process side it informs the Erlang side;

The C part is a self-contained `.c` file (of about 2k lines, of
which 25% are structure definitions), written in an event driven
manner (no threads), statically linkable, depending only on the
`jansson` library (for JSON), using mostly POSIX syscalls.

I've successfully used it for about a year now (in non-production
systems) and it behaves nicely.

If someone finds it useful I could try to extract it in a
standalone project.

Ciprian.

[Robert Raschke]

I have a similar "proxy" to run killable external programs on Windows. It's part of a larger project though, so I'd need to extract it. Would there be interest for me to do that?

The approach is:

Start the proxy program using open_port/2 and send it the command to run, the proxy then uses DuplicateHandle() and CreateProcessW() to kick off the command in a separate OS process but with stdin/stdout/stderr forwarded from the proxy to the new external command, prints the OS process id back to Erlang and waits for the external OS process to finish.

Thus you now have your command running in the backgroud, able to communicate using stdio with the Erlang process that started the proxy.

If you want to stop the command, you invoke the proxy with the process id you got earlier and it kills it using TerminateProcess().

Robby

PS I vaguely remember getting the idea of passing back a process id for later kill purposes from some open source app that uses a similar approach to running killable commands from Erlang. Not that I can remember which one at the mooment, sorry.

[Zabrane Mickael]

> I have a similar "proxy" to run killable external programs on Windows. It's part of a larger project though, so I'd need to extract it. Would there be interest for me to do that?

Yep, could you share that code please?

Regards,
Zabrane

[Tim Watson]

Guys,

This stuff is really good. But fundamentally, we're still in the world where the limitations of open_port are what dominates. Dave Smith's sh modules/functions for both rebar and retest handle this kind of proxy approach just fine, simply by using `sh -C 'echo $$; exec ' ++ Cmd` and using port io. Admittedly it doesn't support signal handling as cleanly.

I hear (on twitter) that there are improvements to port communication coming in R16, and I dearly hope that native processes will be with us by then. At that time, these issues *might* just go away, although they might not. 

The other problem I run into is, what about daemon/detached scripts, things launched using setsid/nohup and/or put straight into the background? And how is your proxy program going to exhibit characteristics any different in this situation, where you cannot 'spawn' a sub-process (or whatever the windows equivalent is). 

Anyway, despite my reticence, I think your programs both sounds pretty useful - although I'm bemused that you'd use JSON serialization instead of simply using the ei interface - and I would also suggest that packaging them as separate libraries would be very useful to the community, giving others the benefit of utilising your work! 

Cheers,

Tim

[Ciprian Dorin Craciun]

On Tue, May 29, 2012 at 3:38 PM, Tim Watson <watson....@gmail.com> wrote:
> Guys,

>
> The other problem I run into is, what about daemon/detached scripts, things
> launched using setsid/nohup and/or put straight into the background?

From my point of view such application -- i.e. that daemonize
themselves -- are broken to begin with and they live you with only
limited options:
* search another solution; (long live the open-source world;) :)
* fix the respective applications and submit patches upstream;
(again long live open-source, but unfortunately not everybody
understands why always daemonizing is a bad approach -- I've
encountered such cases...)
* use OS-specific tricks to force it to stay in foreground (i.e.
PID namespaces in Linux, etc.);

As such I don't see this as being the purpose of such a library.

> And how
> is your proxy program going to exhibit characteristics any different in this
> situation, where you cannot 'spawn' a sub-process (or whatever the windows
> equivalent is).

Again I don't see "not supporting Windows" as a big problem, because:
* most Erlang developers (that would need such process control
features) are educated enough to work in an UNIX compatible
environment;
* most serious Erlang deployments I guess live in an UNIX
compatible environment;

> Anyway, despite my reticence, I think your programs both sounds pretty
> useful - although I'm bemused that you'd use JSON serialization instead of
> simply using the ei interface

Unfortunately the `ei` interface is available (natively) only for
C/C++ and Erlang. But that leaves out things like Python and NodeJS
which are mandatory in my case. And I still find more easily looking
over JSON than over a binary message. (BTW I use the `packet` Erlang
message framing `4 bytes length + payload`.)

> - and I would also suggest that packaging them
> as separate libraries would be very useful to the community, giving others
> the benefit of utilising your work!

:) Yes but most of the time I doubt someone will actually use it.
That's why I've asked if someone wants to use it, I could extract it.


Ciprian.

[Jack Moffitt]

> over JSON than over a binary message. (BTW I use the `packet` Erlang
> message framing `4 bytes length + payload`.)

I have been using {packet, 2}. What happens when (via bad data or
malicious intent) someone sends you 0xffffffff as the message size? Is
there some way I haven't noticed to set a maximum allowed length on
{packet, 4} packets?

jack.

[Ciprian Dorin Craciun]

On Tue, May 29, 2012 at 7:13 PM, Jack Moffitt <ja...@metajack.im> wrote:
>> over JSON than over a binary message. (BTW I use the `packet` Erlang
>> message framing `4 bytes length + payload`.)
>
> I have been using {packet, 2}. What happens when (via bad data or
> malicious intent) someone sends you 0xffffffff as the message size? Is
> there some way I haven't noticed to set a maximum allowed length on
> {packet, 4} packets?

:) This problem hit me a few times before (someone wrote from the
process side to `stdout`).

Interestingly enough (being on x86-32) it tried to allocate 2-4 GB
of data via `malloc`, returned `NULL`, the assertion was broken and
the controlled process was killed in a graceful manner.

And as seen from the code I didn't fix it yet. :)

But indeed such a check should have been made.

Ciprian.

[Garrett Smith]

On Sat, May 26, 2012 at 4:06 AM, Benoit Chesneau <bche...@gmail.com> wrote:
> I'm trying to find a way to manage some os processes launched from
> Erlang. For example I need to relaunch them when the external process
> die for an unknown reason or send an HUP signal or just stop the
> external process.
>
> For now I'm using erlang ports to open an external process and detect
> when it die. I also send signals using the `os:cmd` and and always
> send a SIGKILL signal using the `os:cmd`before closing the port. But
> is there a better way to do that? Maybe a module somewhere ? I saw
> erlexec [1] but it doesn't seem to be maintained since a while.

I think it'd be great to get this into OTP core, but until that
functionality is a) implemented and b) has demonstrated solid
operations across range of platforms, I'm inclined to use something
like runit.

http://smarden.org/runit

Runit will "daemonize" for you (wants child processes to run in the
foreground) and provides a simple interface for sending signals to a
child process. E.g. to send KILL to a process (shell example):

printf k > $SERVICE/supervise/control

The control scheme is described here:

http://smarden.org/runit/runsv.8.html

It's not as tightly integrated as a futuristic open_port might be.
E.g. if you want to detect a pathological case (service flapping, app
errors, etc.) you'd have to poll the environment (track pid changes,
scan logs, etc.) But there are sometimes advantages to keeping OS
process supervision and monitoring at arms length.

Garrett

[Serge Aleynikov]

Take a look at erlexec:

http://code.google.com/p/erlexec/

I believe it does exactly what you need.