[erlang-questions] can a program launched with open_port({spawn, Cmd}, Options) remain running after the port closes?

[Tim Watson]

I'm seeing this happen, and it's most confusing. The program is not launched using nohup/setsid or anything to send it to the background. The arguments passed to open_port/2 are thus:

[{args, Args}, {env, Env}, exit_status, hide, stderr_to_stdout, use_stdio, {line, 16384}]

The script that is being run essentially does this:

# fiddle around with some environment variables, then
exec erl \
-pa ${EBIN_ROOT} \
${START_INSTRUCTION} \
-sname ${NODENAME} \
-boot start_sasl \
${CONFIG_ARG} \
+W w \
${SERVER_ERL_ARGS} \
${LISTEN_ARG} \
"$@"


After some tests are run, the port is closed (using close_port). The connected process for the port is a gen_server, so it's *possible* that this exits without cleanly shutting down, but I'm under the impression (from experimentation) that if the connected process dies, the port is closed anyway. Nevertheless, I'm seeing *stuck nodes* that survive the test run and are locked up because some code on the node (which should be dead after the port goes) is stuck in an io:format/2 call which will never return because stdout is not responding - hardly surprising under the circumstances.

Can someone explain to me what's going on here? Is this an issue with the exec call, and if so what is the problem with it? I've got a minimal example using exec that doesn't behave like this, so I'm a bit bemused. What's really confusing is that the process the port is used to open should presumably be closed regardless of what some sub-process is doing, and I thought that exec returned the stdout of the subprocess anyway!?

[Tim Watson]

On 26 Sep 2012, at 22:54, Tim Watson wrote:
>
> Can someone explain to me what's going on here? Is this an issue with the exec call, and if so what is the problem with it? I've got a minimal example using exec that doesn't behave like this, so I'm a bit bemused. What's really confusing is that the process the port is used to open should presumably be closed regardless of what some sub-process is doing, and I thought that exec returned the stdout of the subprocess anyway!?
>

Oh and I'm under the impression that there's no way for the connected process to block the stdout for the port program right? My reading of the bif is that we end up with erts_open_driver (is that right?) and that io between the port program and emulator will ultimately result in driver_output (or a similar friend) passing messages on to the connected process. Is that correct, and presumably under the smp emulator this is perfectly safe and so on?

[Steve Davis]

I believe I have had a similar issue with attempting to close then unloading a driver. I did not get to the bottom of it. So just left it loaded for that VM.

This issue too was under smp.

/s

[Erik Søe Sørensen]

It is the responsibility of the port program to shut down when stdin reaches EOF.
This behaviour/expectation confused me too at some point, but it probably has its merits: not all software is designed for "let it crash", so simply killing the port program might be too crude in some circumstances.

_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

[Tim Watson]

Hi Erik,

On 27 Sep 2012, at 03:20, Erik Søe Sørensen wrote:

It is the responsibility of the port program to shut down when stdin reaches EOF.
This behaviour/expectation confused me too at some point, but it probably has its merits: not all software is designed for "let it crash", so simply killing the port program might be too crude in some circumstances.

Well that's fine, but it should be documented a bit more clearly IMHO. Also the port program, in this instance, is another beam emulator, so it's clearly *not* going to behave that way! Currently the connected process for the port has 3 shutdown modes:

1. rpc:call(Node, custom_app, stop, [])

2. close_port(Port)

3. open_port({spawn, "kill -9 " ++ OsPid, Opts)

Item 3 is the least preferable as it is not portable (relies on 'kill' being a shell built-in, etc) and requires that the os process id is known. It sounds like (2) is not a workable solution unless the external program is hand written to deal with 'stdin reaches EOF'. This is less than ideal, as it means that open_port/2 is not much use for spawning (and monitoring thereafter) arbitrary programs unless you're willing to live with the fact that you're not able to shut them down by closing the port. 

[Jesper Louis Andersen]

I remember this was something I had going. Let me dig code:

----
#!/bin/sh

(cat && kill 0) | opentracker $*
----

So the trick is that the … && kill 0 part sends a kill signal to all processes in the current process group. This lets you stop another process since cat(1) understands what happens when stdin is closed and handles it accordingly.

I've only used it in tests though so it may need work for real programs

Jesper Louis Andersen
Erlang Solutions Ltd., Copenhagen

[Erik Søe Sørensen]

2012/9/27 Jesper Louis Andersen <jesper.lou...@erlang-solutions.com>

I remember this was something I had going. Let me dig code:

----
#!/bin/sh

(cat && kill 0) | opentracker $*

Nice incantation. I was just about to suggest using a wrapper program, but I probably wouldn't have thought of that variation.
 

----

So the trick is that the … && kill 0 part sends a kill signal to all processes in the current process group. This lets you stop another process since cat(1) understands what happens when stdin is closed and handles it accordingly.

I've only used it in tests though so it may need work for real programs

Such as replacing "&&" with ";" ? :-)
Just in case someone came along and killed the cat. (Can't at the moment imagine other scenarios where cat would exit with non-zero return values, but they probably exist.)

Also, replace $* with "$@" (including the quotes), for proper handling of parameters with spaces in them and such.

/Erik

 

[Erik Søe Sørensen]

2012/9/27 Tim Watson <watson....@gmail.com>

Hi Erik,

On 27 Sep 2012, at 03:20, Erik Søe Sørensen wrote:

It is the responsibility of the port program to shut down when stdin reaches EOF.
This behaviour/expectation confused me too at some point, but it probably has its merits: not all software is designed for "let it crash", so simply killing the port program might be too crude in some circumstances.

Well that's fine, but it should be documented a bit more clearly IMHO.

 

It's not in the API docs for open_port(), but in the section in the manual which definesports (http://www.erlang.org/doc/reference_manual/ports.html#id84207) it does say

The external program resides in another OS process. By default, it should read from standard input (file descriptor 0) and write to standard output (file descriptor 1). The external program should terminate when the port is closed.

 

Also the port program, in this instance, is another beam emulator, so it's clearly *not* going to behave that way!

Why not? - see below.
 

Currently the connected process for the port has 3 shutdown modes:

1. rpc:call(Node, custom_app, stop, [])

2. close_port(Port)

3. open_port({spawn, "kill -9 " ++ OsPid, Opts)

You could add
4. Ensure that the node runs a reading loop which detects EOF - e.g. using file:read_line(standard_io) or file:read(standard_io, SomeBlockSize).

Example for detecting EOF:
erl -noshell -eval 'F=fun(G)->X=file:read_line(standard_io), io:format("~p\n", [X]), X==eof orelse G(G) end, F(F), init:stop().'

You can even do exactly this in practise: put the loop code on the command line.
 

Item 3 is the least preferable as it is not portable (relies on 'kill' being a shell built-in, etc) and requires that the os process id is known. It sounds like (2) is not a workable solution unless the external program is hand written to deal with 'stdin reaches EOF'. This is less than ideal, as it means that open_port/2 is not much use for spawning (and monitoring thereafter) arbitrary programs unless you're willing to live with the fact that you're not able to shut them down by closing the port. 

No, the port mechanism was apparently not meant for arbitrary slave programs. However, a little wrapping can often get you far, as Jesper demonstrated earlier.

[Tim Watson]

Guys,

On 27 Sep 2012, at 13:18, Erik Søe Sørensen wrote:

2012/9/27 Jesper Louis Andersen <jesper.lou...@erlang-solutions.com>

I remember this was something I had going. Let me dig code:

----
#!/bin/sh

(cat && kill 0) | opentracker $*

Nice incantation. I was just about to suggest using a wrapper program, but I probably wouldn't have thought of that variation.
 

----

So the trick is that the … && kill 0 part sends a kill signal to all processes in the current process group. This lets you stop another process since cat(1) understands what happens when stdin is closed and handles it accordingly.

I've only used it in tests though so it may need work for real programs

Such as replacing "&&" with ";" ? :-)
Just in case someone came along and killed the cat. (Can't at the moment imagine other scenarios where cat would exit with non-zero return values, but they probably exist.)

Also, replace $* with "$@" (including the quotes), for proper handling of parameters with spaces in them and such.

Thanks both of you, I guess that I could drop this in in front of the required invocation, so you get the required behaviour. The tricky part here is that the code calling open_port/2 is kind of a generic 'run a script and monitor things' process, so I'll need to experiment with this a bit. Thanks for the suggestions anyway - I'll certainly give this a try.

[Tim Watson]

Hi Erik,

On 27 Sep 2012, at 13:37, Erik Søe Sørensen wrote:

2012/9/27 Tim Watson <watson....@gmail.com>

Hi Erik,

On 27 Sep 2012, at 03:20, Erik Søe Sørensen wrote:

It is the responsibility of the port program to shut down when stdin reaches EOF.
This behaviour/expectation confused me too at some point, but it probably has its merits: not all software is designed for "let it crash", so simply killing the port program might be too crude in some circumstances.

Well that's fine, but it should be documented a bit more clearly IMHO.

 

It's not in the API docs for open_port(), but in the section in the manual which definesports (http://www.erlang.org/doc/reference_manual/ports.html#id84207) it does say

The external program resides in another OS process. By default, it should read from standard input (file descriptor 0) and write to standard output (file descriptor 1). The external program should terminate when the port is closed.

Ah yes, I see that now. I should've probably gone through that more carefully.

 

Also the port program, in this instance, is another beam emulator, so it's clearly *not* going to behave that way!

Why not? - see below.

You could add
4. Ensure that the node runs a reading loop which detects EOF - e.g. using file:read_line(standard_io) or file:read(standard_io, SomeBlockSize).

Example for detecting EOF:
erl -noshell -eval 'F=fun(G)->X=file:read_line(standard_io), io:format("~p\n", [X]), X==eof orelse G(G) end, F(F), init:stop().'

You can even do exactly this in practise: put the loop code on the command line.
 

This is excellent - I can actually provide that as a hook that can be set to run on each remote vm that is launched. Presumably this won't over-stress the io subsystem too much?

Item 3 is the least preferable as it is not portable (relies on 'kill' being a shell built-in, etc) and requires that the os process id is known. It sounds like (2) is not a workable solution unless the external program is hand written to deal with 'stdin reaches EOF'. This is less than ideal, as it means that open_port/2 is not much use for spawning (and monitoring thereafter) arbitrary programs unless you're willing to live with the fact that you're not able to shut them down by closing the port. 

No, the port mechanism was apparently not meant for arbitrary slave programs. However, a little wrapping can often get you far, as Jesper demonstrated earlier.

Yes, I'll probably look at both options to see which fits best. I'm actually finding launching and managing the vm via scripts quite cumbersome compared to using the slave module and a dose of rpc, but the primary goal is testing existing code and that code is normally launched and managed mainly via wrapper scripts, so it's probably the right thing to test them in the same way.

 

Thanks again for your input, and I'll try to ponder the documentation a bit more carefully next time. I'm also more aware now of the constraint and will keep it in mind for the future.

Cheers!

Tim

[fr...@circlewave.net]

On Thu, Sep 27, 2012 at 11:29:03AM +0200, Jesper Louis Andersen wrote:
> I remember this was something I had going. Let me dig code:
>
> ----
> #!/bin/sh
>
> (cat && kill 0) | opentracker $*
> ----
>

> So the trick is that the ? && kill 0 part sends a kill signal to all processes in the current process group. This lets you stop another process since cat(1) understands what happens when stdin is closed and handles it accordingly.

>
> I've only used it in tests though so it may need work for real programs

Haven't been following this thread closely, but it seems erlexec wasn't
mentioned yet:

http://code.google.com/p/erlexec/

BR,
-- Jachym

[Tim Watson]

Guys,

On 27 Sep 2012, at 13:18, Erik Søe Sørensen wrote:

2012/9/27 Jesper Louis Andersen <jesper.lou...@erlang-solutions.com>

I remember this was something I had going. Let me dig code:

----
#!/bin/sh

(cat && kill 0) | opentracker $*

Nice incantation. I was just about to suggest using a wrapper program, but I probably wouldn't have thought of that variation.
 

----

So the trick is that the … && kill 0 part sends a kill signal to all processes in the current process group. This lets you stop another process since cat(1) understands what happens when stdin is closed and handles it accordingly.

I've only used it in tests though so it may need work for real programs

Such as replacing "&&" with ";" ? :-)
Just in case someone came along and killed the cat. (Can't at the moment imagine other scenarios where cat would exit with non-zero return values, but they probably exist.)

Also, replace $* with "$@" (including the quotes), for proper handling of parameters with spaces in them and such.

This works a treat, with some minor modifications. Thanks for pointing it out!

[Tim Watson]

On 27 Sep 2012, at 19:18, fr...@circlewave.net wrote:
> Haven't been following this thread closely, but it seems erlexec wasn't
> mentioned yet:
>
> http://code.google.com/p/erlexec/

Doesn't appear very well maintained. I might pick it apart and rewrite it when native processes get released - it is a cool idea. Anyway, for the time being it's not a great help as I'm writing a framework to help set up and tear down clusters for distributed testing (based largely on common test) so I want to minimise the number of dependencies as much as possible. Besides, I'm more interested in working on the remote/ssh runner+monitor next, rather than spending another age fiddling around with the script runner.

But yes, erlexec does look interesting and has been on my radar since it popped up on this list a few months back, discussing the desire to do proper signal handling (such as sending SIGTERM, SIGHUP, etc to the external program) without resorting to os:cmd("kill -" ++ Code ++ " " ++ OsPid). Again, this is something that linked-in drivers *can* deal with to a limited extent, and therefore I suspect that native processes will make for a very nice foundation on which to build an alternative to open_port/2.

Also, IIRC there was a cool command line testing framework open sourced in the last year which was announced on this list. I can't remember what it was called, but it has a port driver that handles the the external comms and so on - looked very nice, although that's not what I'm trying to do/be as my focus is on getting things in a consistent state and monitoring to make sure they stay that way through the test run, then tearing down nicely before the next phase kicks off.

Cheers,
Tim

[Serge Aleynikov]

Regarding the statement that "erlexec" hasn't been well maintained - if
you submit a bug report, I'll be happy to look into it, but other than
that I believe the project does well what it was designed to do - spawn
OS processes, pass them signals, chain them with links/monitors and link
them to Erlang PIDs, and kill them reliably by signals or custom
commands. Hence I haven't had a need to evolve it beyond where it has
been as of right now. My "wish list" included adding portability so
that it works on Windows as well as it does on Linux, but I never really
had a need for it, and if there's a volunteer for this task, I'd be glad
to accept patches.

Regards,

Serge


On 9/27/2012 3:11 PM, Tim Watson wrote:> On 27 Sep 2012, at 19:18,

fr...@circlewave.net wrote:
>> Haven't been following this thread closely, but it seems erlexec wasn't
>> mentioned yet:
>>
>> http://code.google.com/p/erlexec/
>

[Tim Watson]

Hey Serge,

I wasn't insinuating that erlexec isn't fit for purpose, just that it doesn't seem to be under active development. From the sounds of things, that's because it's relatively stable, which is cool. Thanks for piping up and pointing that out - I will look at it for some other projects I've got brewing.

Cheers
Tim

[Tim Watson]

The project doesn't appear to build cleanly on mac os lion:

t4@iske:erlexec $ ./configure
configure: error: cannot find install-sh, install.sh, or shtool in config "."/config
t4@iske:erlexec $ ./build
Install dir: /usr/local/src/erlang/erlexec/install
aclocal
autoconf -i
configure.ac:19: error: possibly undefined macro: AC_PROG_LIBTOOL
If this token and others are legitimate, please use m4_pattern_allow.
See the Autoconf documentation.
make: *** [configure] Error 1
t4@iske:erlexec $
t4@iske:erlexec $ autoconf -V
autoconf (GNU Autoconf) 2.69
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+/Autoconf: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>, <http://gnu.org/licenses/exceptions.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by David J. MacKenzie and Akim Demaille.
t4@iske:erlexec $


Is there a specific version of autotools required to build and compile erlexec?

[Serge Aleynikov]

Is it possible that you don't have libtool installed (missing
AC_PROG_LIBTOOL)?

I have the following on linux and everything builds fine:
$ autoconf -V
autoconf (GNU Autoconf) 2.68
$ libtool --version
libtool (GNU libtool) 2.4
$ ./build
$ make
$ make install
$ tree install
install
└── exec-1.0
├── ebin
│ ├── exec.app
│ ├── exec_app.beam
│ └── exec.beam
├── include
│ └── exec.hrl
├── priv
│ └── x86_64-unknown-linux-gnu
│ └── bin
│ └── exec-port
└── src
├── exec_app.erl
└── exec.erl

On 9/28/2012 7:45 AM, Tim Watson wrote:> The project doesn't appear to

[Tim Watson]

Well I do, but clearly not the same version as you?

t4@iske:erlexec $ which libtool
/usr/bin/libtool
t4@iske:erlexec $ ls -la /usr/bin/ | grep libtool
-r-xr-xr-x 1 root wheel 149216 22 May 15:30 libtool
lrwxr-xr-x 1 root wheel 7 22 May 15:30 ranlib -> libtool
t4@iske:erlexec $ libtool --version
libtool: unknown option character `-' in: --version
Usage: libtool -static [-] file [...] [-filelist listfile[,dirname]] [-arch_only arch] [-sacLT]
Usage: libtool -dynamic [-] file [...] [-filelist listfile[,dirname]] [-arch_only arch] [-o output] [-install_name name] [-compatibility_version #] [-current_version #] [-seg1addr 0x#] [-segs_read_only_addr 0x#] [-segs_read_write_addr 0x#] [-seg_addr_table <filename>] [-seg_addr_table_filename <file_system_path>] [-all_load] [-noall_load]
t4@iske:erlexec $

Honestly the osx build tool chain drives me nuts sometimes. :/

[Serge Aleynikov]

Unfortunately I don't have OSx to test this. I can't reproduce your
problem on Linux. Perhaps you could check the libtool version this way:

$ grep "^macro_version" /usr/bin/libtool
macro_version=2.4

and if yours is different, try to upgrade it.

In order to simplify the build process I'll try to convert erlexec to
use rebar some time next week, which will eliminate autotools-based
toolchain. Need to figure out with rebar how to test and optionally
enable conditional defines based on presence of special C headers such
as "sys/capability.h" when compiling a port program.

On 9/28/2012 5:16 PM, Tim Watson wrote:> Well I do, but clearly not the

[Tim Watson]

I'm a heavy rebar user and have compiled ports with it before, so give me a shout if you need any assistance. I can also do testing on several unix platforms.

[Serge Aleynikov]

In order to compile a port program using rebar, I need to place the
resulting port program in the architecture-specific folder:
priv/$ARCH/exec-port

where $ARCH is the result of the call:
erlang:system_info(system_architecture)

I was thinking of setting ARCH in the pre-compile hook. However, it
doesn't seem that the "port_specs" config clause allows expansion of
environment variables, so if the "ARCH" variable has a value, the
following won't work:

{port_specs, [{"priv/$ARCH/exec-port", ["c_src/*.cpp"]}]}.

Is there a work-around I could use to accomplish this?

[Richard O'Keefe]

On 29/09/2012, at 9:41 AM, Serge Aleynikov wrote:

> Unfortunately I don't have OSx to test this. I can't reproduce your
> problem on Linux. Perhaps you could check the libtool version this way:
>
> $ grep "^macro_version" /usr/bin/libtool
> macro_version=2.4
>
> and if yours is different, try to upgrade it.

It's not a version issue. The Mac OS X libtool and the Linux libtool
program are not versions of the same program. Gordon Matzigkeit's
Linux libtool is a shell script. Apple's /Developer/usr/bin/libtool
is a Mach-O universal binary. They don't do the same thing. The
Apple is (waves hands vaguely) a replacement for ar+ranlib and also
generates .dylib files. To find the version of Apple's libtool:
m% libtool -V
Apple Computer, Inc. version cctools-800

[Tim Watson]

You can do this by including a rebar.config.script in the project, which is handled with file:script/1 to produce the config. This can read the regular rebar.config (either using file:consult/1 or rebar_config) and modify whatever it likes. An alternative is to write a plugin which hooks into the post_compile:

-module(example).

post_compile(Conf, _) ->
%% move the binar wherever you want
ok.

[Serge Aleynikov]

Thanks for your tip on the dynamic configuration of rebar.

Below please find a version of erlexec that uses rebar:

https://github.com/saleyn/erlexec

Along with its documentation:

http://saleyn.github.com/erlexec

The old repository http://code.google.com/p/erlexec/ is deprecated.

Please let me know if this builds on other OS's besides Linux.

Regards,

Serge

On 9/29/2012 5:23 AM, Tim Watson wrote:

[Tim Watson]

Thanks Serge, I'll take a look and let you know.