[boost] [boost.process] 0.6 Alpha
Klemens Morgenstern
the boost.process 0.6 library is now in alpha, that is: it has some
(compiling) examples, a documentation and tests that pass on windows as
well as on linux.
It basically allows you to manage a process like this:
pstream pipe;
child c("some_prog", "--some-arg", env["PATH"]+="C:/stuff", std_in <
"data.csv", std_out > pipe, std_err > null, start_dir="./bin");
std::string data;
pipe >> data;
c.wait();
cout << c.exit_code() << endl;
You can find the documentation here:
http://klemens-morgenstern.github.io/process/
Get the source code from here:
https://github.com/klemens-morgenstern/boost-process
If you want to use it on windows, you currently would need to clone my
boost.winapi fork also: https://github.com/klemens-morgenstern/winapi
As for the scope of the library, it is meant to implement a small,
portable layer to allow platform-independent process management. Thereby
it's functionality is (unless I missed something) basically what is
possible on posix as well as on windows. There are some platform
extension, but I would not recommend to use them. It is also possible
(though not yet documented) to write extensions, and if they work on
both platforms I will gladly add them.
It is however not the goal of the library, to be the be-all and end-all
of process management; in past disscussions, different ideas where
brought up, like implementing a DSEL for that or only allowing
asynchronous communication. This library will never be that, since it
would be much more fitting to build this on-top of it. E.g. a DSEL could
be written (e.g. boost.shell) using boost.process underneath, so the
library could concentrate only on the DSEL.
It doesn't depend on boost.iostreams anymore, though it still uses
boost.asio, boost.fusion and boost.filesystem.
As for the open issues: some error handling is still lacking (but that'd
be something like being unable to duplicate a handle), which would be
required for proper extension, which is also why it's not documented.
Also there's still an open discussion about the naming of the class and
functions (child, system, spawn), so that may change also. But I
consider this simple enough for a easy refactoring. If anyone has naming
ideas, please tell me. One Idea was 'subprocess' but
'process::subprocess' just looks too strange to me.
I hope you try it out, and give me some feedbacks. Or write angry error
reports, both helps.
Thanks!
Klemens
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Edward Diener
> Hi all,
>
> the boost.process 0.6 library is now in alpha, that is: it has some
> (compiling) examples, a documentation and tests that pass on windows as
> well as on linux.
>
> It basically allows you to manage a process like this:
>
> pstream pipe;
>
> child c("some_prog", "--some-arg", env["PATH"]+="C:/stuff", std_in <
> "data.csv", std_out > pipe, std_err > null, start_dir="./bin");
>
> std::string data;
> pipe >> data;
>
> c.wait();
> cout << c.exit_code() << endl;
>
> You can find the documentation here:
> http://klemens-morgenstern.github.io/process/
> Get the source code from here:
> https://github.com/klemens-morgenstern/boost-process
with no explanation of what it is doing that you put comments in the
code example. This should be true whether the code example is in a
message or in your docs.
Klemens Morgenstern
>> http://klemens-morgenstern.github.io/process/
>> Get the source code from here:
>> https://github.com/klemens-morgenstern/boost-process
>
> I would like to immediately suggest that when you show a code example
> with no explanation of what it is doing that you put comments in the
> code example. This should be true whether the code example is in a
> message or in your docs.
obvious to me. I'll add some more detailed example in the documentation
as soon as I have the time. As for the previous example (since I can't
edit the mail):
pstream pipe; //iostream implementation with a pipe
//start the process
child c(
"some_prog", //program name
"--some-arg", //argument
env["PATH"]+="C:/stuff", //add C:/stuff to the PATH-environment
variable
std_in < "data.csv", //redirect stdin to a file
std_out > pipe, //redirect stdout to created pipe
std_err > null, //redirect stderr to /dev/null
start_dir="./bin");//start in ./bin
std::string data;
pipe >> data; //read a string from the pipe.
c.wait(); //wait for the program to exit
cout << c.exit_code() << endl;//print the return code
Rob Stewart
___
Rob
(Sent from my portable computation engine)
Klaim - Joël Lamotte
wrote:
So far I had time only to go quickly through the documentation and a bit in
the code so here are quick remarks:
1. It is not clear at all what happen when child object is destroyed.
2. The way child process termination is implemented for each platform
should be documented.
3. On most platforms there is a way to send a "quit" command to the process
and a way to kill it by force ("kill -9").
Clarifying what is happening in the default cases and having a way to
terminate one way or the other (or a combination of
both, that is requesting termination but force-kill after a time if it
didn't die)
On windows there is an issue with this because the "termination request"
message can only be received if
you didn't build a console application but a windows application (with a
WinMain) but that only means
that the message could be ignored by the process if it's console
application (there are ways to have both WinMain and
a console window, it's just a bit more clumsy to setup).
In our (Softbank Robotics EU) use of child process tracking we need both
termination request and forced.
One could argue that the termination request do not need to be
implemented by this library but it would be very useful
to have a cross platform function doing this for simple cases.
Joël Lamotte
Klemens Morgenstern
> So far I had time only to go quickly through the documentation and a bit in
> the code so here are quick remarks:
>
> 1. It is not clear at all what happen when child object is destroyed.
process if it wasn't detached or exited before. I added that, will be up
in the next doc build.
> 2. The way child process termination is implemented for each platform
> should be documented.
(also added)
> 3. On most platforms there is a way to send a "quit" command to the process
> and a way to kill it by force ("kill -9").
> Clarifying what is happening in the default cases and having a way to
> terminate one way or the other (or a combination of
> both, that is requesting termination but force-kill after a time if it
> didn't die)
> On windows there is an issue with this because the "termination request"
> message can only be received if
> you didn't build a console application but a windows application (with a
> WinMain) but that only means
> that the message could be ignored by the process if it's console
> application (there are ways to have both WinMain and
> a console window, it's just a bit more clumsy to setup).
>
> In our (Softbank Robotics EU) use of child process tracking we need both
> termination request and forced.
>
> One could argue that the termination request do not need to be
> implemented by this library but it would be very useful
> to have a cross platform function doing this for simple cases.
handle, not the proc handle. That technically means, that you cannot
tell the process to exit, but a part of it. Hence I don't consider this
the same as signaling the process to exit, which means it's not
portable, thus I do argue it shouldn't be part of the library. If I have
a child::request_exit function, it must work on all processes, not just
some of the windows-procs. If there was a portable way to do it, it
would be part of this library.
I think it's easy enough, you can get the pid (or even gid, if you use a
group) and implement that in two lines. On windows you can get the HWND
as an std::intptr_t through a pipe, though that's a bit more code.
Thanks,
Klemens
Klaim - Joël Lamotte
wrote:
is a native windows application (a console application is similar but with
additional
code to instantiate the console window).
All OS we use have a platform-specific way to request exit to a process
(and signals are manageable on all these platforms).
What I am suggesting is to provide a cross-platform function doing the
platform-specific request, which would give a chance to the child process
to
kill itself. One can implement this with their own protocol too, but as
platform-specific protocols are already there, having a ready way to do it
would help.
> thus I do argue it shouldn't be part of the library. If I have a
> child::request_exit function, it must work on all processes, not just some
> of the windows-procs.
some special implementation of windows processes, like console processes.
Even with a native windows process, or a linux process, you still have to
do some platform-specific message processing in the child process code
to manage this request, which is fine to me (and could also be generalized).
> If there was a portable way to do it, it would be part of this library.
>
provide example if you want? It's clumsy but it shows how we do it. (see
below)
> I think it's easy enough, you can get the pid (or even gid, if you use a
> group) and implement that in two lines. On windows you can get the HWND as
> an std::intptr_t through a pipe, though that's a bit more code.
>
>
this library, although it's not a show stopper to me.
Klemens Morgenstern
differently on different platforms. But maybe it is possible to create
some handling
for the current process in the this_process namespace. E.g. a callback
`this_process::on_exit_request(...)` or a bit
`this_process::exit_requested()` so you can
also skip the platform-specifics here.
>> If there was a portable way to do it, it would be part of this library.
>>
> I don't see any way in which it is not possible to make it portable. I can
> provide example if you want? It's clumsy but it shows how we do it. (see
> below)
handling, regarding the this_process namespace.
>> I think it's easy enough, you can get the pid (or even gid, if you use a
>> group) and implement that in two lines. On windows you can get the HWND as
>> an std::intptr_t through a pipe, though that's a bit more code.
>>
>>
> Exactly. Which is why I think such function would be a good candidate for
> this library, although it's not a show stopper to me.
adding it.
Klaim - Joël Lamotte
wrote:
OK let me gather our open-source and closed source code in one example and
I'll send it to you privately.
Our solution is not perfect but it seems to work well.
Joël Lamotte
Rob Stewart
>On 6 June 2016 at 12:00, Klemens Morgenstern
><klemens.m...@gmx.net>
>wrote:
>
>>> should be documented.
>>>
>> Makes sense. It's TerminateProcess on windows and "kill -9" on posix.
>> (also added)
That's too harsh as a default. Default signal handling in Posix systems means that sending SIGTERM first would signal a graceful exit. That doesn't mean the process will exit successfully or that it won't ignore the signal, so after waiting for a limited time (user specified with a default), you would send SIGKILL.
(You could actually send SIGTERM, SIGINT, and SIGQUIT, one after the other to increase the chance that the process recognizes the need to exit.)
On Windows, you can send WM_CLOSE. The process may respond within the allotted time and it may not (it certainly won't if it's an ordinary console app). TerminateProcess() is the final step if the process handle isn't signaled within the timeout period.
Thus, terminating a process behaves similarly on both platforms: try a nice signal, wait, then terminate it forcefully if need be.
___
Rob
(Sent from my portable computation engine)
_______________________________________________
Klemens Morgenstern
Am 07.06.2016 um 10:37 schrieb Rob Stewart:
> On June 6, 2016 6:49:44 AM EDT, "Klaim - Joël Lamotte" <mjk...@gmail.com> wrote:
>> On 6 June 2016 at 12:00, Klemens Morgenstern
>> <klemens.m...@gmx.net>
>> wrote:
>>
>>> 2. The way child process termination is implemented for each platform
>>>> should be documented.
>>>>
>>> Makes sense. It's TerminateProcess on windows and "kill -9" on posix.
>>> (also added)
> That's too harsh as a default. Default signal handling in Posix systems means that sending SIGTERM first would signal a graceful exit. That doesn't mean the process will exit successfully or that it won't ignore the signal, so after waiting for a limited time (user specified with a default), you would send SIGKILL.
>
> (You could actually send SIGTERM, SIGINT, and SIGQUIT, one after the other to increase the chance that the process recognizes the need to exit.)
>
> On Windows, you can send WM_CLOSE. The process may respond within the allotted time and it may not (it certainly won't if it's an ordinary console app). TerminateProcess() is the final step if the process handle isn't signaled within the timeout period.
>
> Thus, terminating a process behaves similarly on both platforms: try a nice signal, wait, then terminate it forcefully if need be.
I actually thought the same thing, but it is an issue of security.
Consider this:
ipstream is;
child c("prog", std_in < is);
is << generate_input() << endl;
Not if generate_input throws, I need to terminate the child, elsewise I
get a deadlock (since "prog" waits for input). If you do not want this,
you can detach it or join it. In that it is similar to std::thread,
though it doesn't terminate the current process. Now using a timeout
would be possible, but I really don't like to set an arbitrary value here.
Regarding the WM_CLOSE version: I currently don't think that is a really
portable solution, since you signal the HWND not the Process, which
means that console programs will cause problems here. Joël Lamotte
recommended a similar solution and will send me some examples, so a
child::request_exit() function might be added. Now if that happens, I
still will not change the behaviour in child, but you might be able to
do this then:
soft_exit se (child("thingy"), milliseconds(100));
It really depends on how this can be achieved on windows; I didn't look
into that, because it's not part of the Process handling in the WinAPI,
but maybe there's a workaround.
Brendon Costa
interested in the process library.
I have also written a C++ process (and signals) library supporting
windows/OSX/Linux/Android (for an employer though so I can't propose it for
boost). It actually suprised me how many platform differences existed that
needed to be worked around in writing this library compared to others I
have written.
I also did something similar to Klemens if what I undertood is correct, I
did a fallback scenario of send TERM if not closed within time period send
a KILL. I didn't really like it though and wonder if the std::thread
destructor behavior is better to std::terminate if the child is still
running but not detached and treat it more like a bug to destroy an active
process instance than something to rely on to close it.
Not having a SIGTERM equivilant for windows processes bothered me as
SIGKILL is just too nasty as a default as mentioned. So I looked into this
quite thoroughly. For GUI processes you can easily send WM_CLOSE message to
it to simulate something akin to a SIGTERM. I think I used a WM_CLOSE from
memory. I know there was a choice between QUIT and CLOSE, I think CLOSE was
more suitable but I cant recall the exact reason now and may be wrong.
For console processes I only found bad options. I found something that
worked quite well but ended up dropping it in the end as it was not
required and was super hacky IMO. I only ever wanted to SIGTERM my own
processes in the end so just used a non generic solution of a named pipe to
listen for signals and raise() to local process.
The GenerateConsoleCtrlEvent() comes close to doing what we want but its
got the worst behaviour in how to choose who gets the signal delivered and
is not sufficient (even when creating processes to detach/attach to other
consoles to generate the events). In the end I followed the basic idea
proposed at:
http://www.latenighthacking.com/projects/2003/sendSignal/
With extra ideas proposed in comment (20110401) and a few extra changes of
my own. I got this working well in all scenarios I could think of to test
on win32 and 64 up to Win7 (I think I went down to XP but cant recall for
sure).
The idea is basically to use the mechanism that GenerateConsoleCtrlEvent()
uses to deliver the signal, but you choose the process where to deliver it
yourself instead of using the stupid rules that GenerateConsoleCtrlEvent()
follows.
The idea is to create a new "helper" process, have it register a signal
handler, raise a signal to itself and in the handler walk back the stack to
find the entry point where the signal was generated. It turns out that the
function where the signal was generated (I think it was
kernel32!CtrlRoutine) this function has the correct prototype for
CeateRemoteThread and is in the kernel32.dll so is in the same location in
all processes on the system and is how GenerateConsoleCtrlEvent() is
implemented.
So you basically then find the process you care about, create a remote
thread in it to call the specified function and presto you end up with a
SIGTERM that can be handled by a normal signal() handler in a console
process.
I hope my research was useful, but this approach was not really acceptable
to me and I dont think would be to the boost community. Maybe another idea
might arise from it though, in which case I would love to hear about it.
On 7 June 2016 at 18:59, Klemens Morgenstern <klemens.m...@gmx.net>
wrote:
Klemens Morgenstern
> I am vaguely following this thread as I don't have lots of time but am
> interested in the process library.
>
> I have also written a C++ process (and signals) library supporting
> windows/OSX/Linux/Android (for an employer though so I can't propose it for
> boost). It actually suprised me how many platform differences existed that
> needed to be worked around in writing this library compared to others I
> have written.
>
> I also did something similar to Klemens if what I undertood is correct, I
> did a fallback scenario of send TERM if not closed within time period send
> a KILL. I didn't really like it though and wonder if the std::thread
> destructor behavior is better to std::terminate if the child is still
> running but not detached and treat it more like a bug to destroy an active
> process instance than something to rely on to close it.
>
explicitly call wait or join, or wait_for/wait_until if you want a timeout.
arbitrary values. But that would break the console and might not even work.
> The idea is to create a new "helper" process, have it register a signal
> handler, raise a signal to itself and in the handler walk back the stack to
> find the entry point where the signal was generated. It turns out that the
> function where the signal was generated (I think it was
> kernel32!CtrlRoutine) this function has the correct prototype for
> CeateRemoteThread and is in the kernel32.dll so is in the same location in
> all processes on the system and is how GenerateConsoleCtrlEvent() is
> implemented.
>
> So you basically then find the process you care about, create a remote
> thread in it to call the specified function and presto you end up with a
> SIGTERM that can be handled by a normal signal() handler in a console
> process.
>
> I hope my research was useful, but this approach was not really acceptable
> to me and I dont think would be to the boost community. Maybe another idea
> might arise from it though, in which case I would love to hear about it.
>
>
right, that's a bit too much hacking for a stable library.
Rob Stewart
>Am 07.06.2016 um 10:37 schrieb Rob Stewart:
>> On June 6, 2016 6:49:44 AM EDT, "Klaim - Joël Lamotte"
><mjk...@gmail.com> wrote:
>>> On 6 June 2016 at 12:00, Klemens Morgenstern <klemens.m...@gmx.net>
>>> wrote:
>>>
>>>> It's TerminateProcess on windows and "kill -9" on posix.
>
I see your point. You could provide a default timeout for that case and include a function permitting the user to override that default.
>Regarding the WM_CLOSE version: I currently don't think that is a
>really
>portable solution, since you signal the HWND not the Process, which
>means that console programs will cause problems here.
I mentioned that. Sending signals on posix systems may not terminate a process either. That's why you fall back on SIGKILL. On Windows, you try the WM_CLOSE approach and fall back on TerminateProcess().
>Joël Lamotte
>recommended a similar solution and will send me some examples, so a
>child::request_exit() function might be added.
The remote thread technique sounds interesting.
>Now if that happens, I
>still will not change the behaviour in child, but you might be able to
>do this then:
>
>soft_exit se (child("thingy"), milliseconds(100));
Why not c.terminate(milliseconds(100)) and c.kill()?
Klemens Morgenstern
Sry, but no. I don't see the point of timeouts, especially default ones.
In this case it is quite clear, that there's no point in continuing, so
why put in timeout?
>
>> Regarding the WM_CLOSE version: I currently don't think that is a
>> really
>> portable solution, since you signal the HWND not the Process, which
>> means that console programs will cause problems here.
>
> I mentioned that. Sending signals on posix systems may not terminate a process either. That's why you fall back on SIGKILL. On Windows, you try the WM_CLOSE approach and fall back on TerminateProcess().
Difference is: every posix program can catch SIGTERM, not every windows
program will get WM_CLOSE - console applications will never receive
that. It depends on the compile options.
Now for this to be part of the process core library (i.e. included in
boost/process.hpp) it would need to work like this both platforms:
//father
child c(...);
c.request_exit();
//child
this_process::on_exit_request(std::function<bool()> func);
That's not possible, so it won't be in the library. It might be added to
the platform-extensions though, i.e. you could have something like the
following functions if you include boost/process/posix.hpp or
boost/process/windows.hpp (needs to be distinguished by #ifdef)
posix::send_terminate(child &);
windows::send_wm_close(child&);
windows::send_console_kill(child&);
I have no problem having a platform-specific function there, but if I
have a function in the multi-platform part of boost.process it has to
behave the same everywhere.
>
>> Joël Lamotte
>> recommended a similar solution and will send me some examples, so a
>> child::request_exit() function might be added.
>
> The remote thread technique sounds interesting.
>
>> Now if that happens, I
>> still will not change the behaviour in child, but you might be able to
>> do this then:
>>
>> soft_exit se (child("thingy"), milliseconds(100));
>
> Why not c.terminate(milliseconds(100)) and c.kill()?
>
Yeah would be possible, too. Though as written above, there won't be a
terminate-request member-function of child. But what you could do is this.
some_magic_terminate_request(c); //your SIGTERM impl.
if (!c.wait_for(milliseconds(100))
c.terminate();
Rob Stewart
>Am 08.06.2016 um 04:06 schrieb Rob Stewart:
>
>>> Consider this:
>>>
>>> ipstream is;
>>> child c("prog", std_in < is);
>>> is << generate_input() << endl;
>>>
>>> Not if generate_input throws, I need to terminate the child,
>elsewise I
>>> get a deadlock (since "prog" waits for input). If you do not want
>this,
>>> you can detach it or join it. In that it is similar to std::thread,
>>> though it doesn't terminate the current process. Now using a timeout
>>> would be possible, but I really don't like to set an arbitrary value
>>> here.
>>
>> I see your point. You could provide a default timeout for that case
>and include a function permitting the user to override that default.
>
>Sry, but no. I don't see the point of timeouts, especially default
>ones.
>In this case it is quite clear, that there's no point in continuing, so
>why put in timeout?
Your approach gives the child no chance to shut down gracefully. It cannot close database connections, remove temporary files, clean up content in shared memory, whatever.
>> Sending signals on posix systems may not terminate
>a process either. That's why you fall back on SIGKILL. On Windows, you
>try the WM_CLOSE approach and fall back on TerminateProcess().
>
>Difference is: every posix program can catch SIGTERM, not every windows
>program will get WM_CLOSE - console applications will never receive
>that. It depends on the compile options.
Some console apps can be written to process such messages, but most won't, of course. What's more, if an app doesn't handle the message, you're no worse off than when you just call TerminateProcess().
>Now for this to be part of the process core library (i.e. included in
>boost/process.hpp) it would need to work like this both platforms:
>
>//father
>child c(...);
>c.request_exit();
>
>//child
>this_process::on_exit_request(std::function<bool()> func);
>
>That's not possible, so it won't be in the library. It might be added
>to
>the platform-extensions though, i.e. you could have something like the
>following functions if you include boost/process/posix.hpp or
>boost/process/windows.hpp (needs to be distinguished by #ifdef)
>
>posix::send_terminate(child &);
>windows::send_wm_close(child&);
>windows::send_console_kill(child&);
>
>I have no problem having a platform-specific function there, but if I
>have a function in the multi-platform part of boost.process it has to
>behave the same everywhere.
I see that you only want to send the nice termination signal if the child can install a handler in a common way. I was missing that side of the equation. The Windows issue is whether it's possible to determine, at compile time or runtime, whether an association has a GUI message loop or not. If so, you'd have to install a message loop handler for WM_CLOSE, and if not, you'd have to use SetConsoleCtrlHandler() to install a callback.
I don't know much about Windows message handler loops, but you could install a message hook if there isn't a better way to integrate into the app's own message loop.
>>> Joël Lamotte
>>> recommended a similar solution and will send me some examples, so a
>>> child::request_exit() function might be added.
>>
>> The remote thread technique sounds interesting.
>>
>>> Now if that happens, I
>>> still will not change the behaviour in child, but you might be able
>to
>>> do this then:
>>>
>>> soft_exit se (child("thingy"), milliseconds(100));
>>
>> Why not c.terminate(milliseconds(100)) and c.kill()?
>
>Yeah would be possible, too. Though as written above, there won't be a
>terminate-request member-function of child. But what you could do is
>this.
>
>some_magic_terminate_request(c); //your SIGTERM impl.
>
>if (!c.wait_for(milliseconds(100))
> c.terminate();
I'm confused why you think some_magic_terminate_request() should be a free function, while terminate() is a member function.
Klemens Morgenstern
> On June 8, 2016 3:05:37 AM EDT, Klemens Morgenstern <klemens.m...@gmx.net> wrote:
>> Am 08.06.2016 um 04:06 schrieb Rob Stewart:
>>
>>>> Consider this:
>>>>
>>>> ipstream is;
>>>> child c("prog", std_in < is);
>>>> is << generate_input() << endl;
>>>>
>>>> Not if generate_input throws, I need to terminate the child,
>> elsewise I
>>>> get a deadlock (since "prog" waits for input). If you do not want
>> this,
>>>> you can detach it or join it. In that it is similar to std::thread,
>>>> though it doesn't terminate the current process. Now using a timeout
>>>> would be possible, but I really don't like to set an arbitrary value
>>>> here.
>>>
>>> I see your point. You could provide a default timeout for that case
>> and include a function permitting the user to override that default.
>>
>> Sry, but no. I don't see the point of timeouts, especially default
>> ones.
>> In this case it is quite clear, that there's no point in continuing, so
>> why put in timeout?
>
> Your approach gives the child no chance to shut down gracefully. It cannot close database connections, remove temporary files, clean up content in shared memory, whatever.
>
A timeout doesn't make sense if you have no standard way to signal the
child to exit. If you implement your version of that, you can use
chlid::wait_for and then terminate. The child class is meant to be
joined before destructing; the terminate on destruction is for exceptions.
That, AND it ought to be the default on the system for exit requests.
> I don't know much about Windows message handler loops, but you could install a message hook if there isn't a better way to integrate into the app's own message loop.
>
It doesn't work really, especially since SetConsoleCtrlHandler does only
allow two values (Ctrl+C & Ctrl+Break), which are both not clearly
termination requests, but more of a terminate command I think. Also to
do that, I would need to put all those child processes on a new process
group, which would mean, that Ctrl+C will NOT be transmitted to them.
That's also a dealbreaker.
>>>> Joël Lamotte
>>>> recommended a similar solution and will send me some examples, so a
>>>> child::request_exit() function might be added.
>>>
>>> The remote thread technique sounds interesting.
>>>
>>>> Now if that happens, I
>>>> still will not change the behaviour in child, but you might be able
>> to
>>>> do this then:
>>>>
>>>> soft_exit se (child("thingy"), milliseconds(100));
>>>
>>> Why not c.terminate(milliseconds(100)) and c.kill()?
>>
>> Yeah would be possible, too. Though as written above, there won't be a
>> terminate-request member-function of child. But what you could do is
>> this.
>>
>> some_magic_terminate_request(c); //your SIGTERM impl.
>>
>> if (!c.wait_for(milliseconds(100))
>> c.terminate();
>
> I'm confused why you think some_magic_terminate_request() should be a free function, while terminate() is a member function.
>
Because I would be fine to have it as an extension in the posix- or
windows-namespace, but not in the core-library. Hence it won't be a
method of child.
Oswin Krause
> That's not possible, so it won't be in the library. It might be added
> to the platform-extensions though, i.e. you could have something like
> the following functions if you include boost/process/posix.hpp or
> boost/process/windows.hpp (needs to be distinguished by #ifdef)
this was that the library needs a way to kill a child for example to
prevent a deadlock when a pipe breaks, etc. What would in this case be
the default behaviour on linux? I would think that a solution would be
perfectly fine if it behaved as expected on each operating system, even
if the behaviour is different for the different operating systems.
Klemens Morgenstern
> Hi,
>
>> That's not possible, so it won't be in the library. It might be added
>> to the platform-extensions though, i.e. you could have something like
>> the following functions if you include boost/process/posix.hpp or
>> boost/process/windows.hpp (needs to be distinguished by #ifdef)
> I only partially followed the discussion. I think the starting point of
> this was that the library needs a way to kill a child for example to
> prevent a deadlock when a pipe breaks, etc. What would in this case be
> the default behaviour on linux? I would think that a solution would be
> perfectly fine if it behaved as expected on each operating system, even
> if the behaviour is different for the different operating systems.
>
default on constructor call of the child-handle. And that's terminate,
i.e. kill, without any timeout.
The question is if we can have a SIGTERM equivalent on windows, and it
doesn't seem like it. There is a WM_QUIT message, but that's not send to
the process, but its HWNDs, rendering it useless, if you have a console app.
Gavin Lambert
> It doesn't work really, especially since SetConsoleCtrlHandler does only
> allow two values (Ctrl+C & Ctrl+Break), which are both not clearly
> termination requests, but more of a terminate command I think. Also to
> do that, I would need to put all those child processes on a new process
> group, which would mean, that Ctrl+C will NOT be transmitted to them.
> That's also a dealbreaker.
So if you want to send one of those two signals then
GenerateConsoleCtrlEvent is probably the right way to go. (You might
need to use CreateRemoteThread to make the child raise it itself rather
than doing it in the parent process though, which requires different
versions for 32-bit vs. 64-bit processes.)
If you want to raise SIGTERM, this is possible but you'd have to use
CreateRemoteThread to call raise(SIGTERM) directly in the child process.
Where this gets a little tricky is that you'd need to call the right
version of "raise", which differs depending on the CRT the child process
uses (static or dynamic, different compiler versions, etc) -- and the
required code differs between 32-bit and 64-bit processes too.
And of course both of the above are dependent on the console app
developer actually intercepting and processing them as a graceful
shutdown -- otherwise it's little different from a TerminateProcess.
Services are really the only kind of executable that have a defined way
to externally start/stop them programmatically. Other process types
have interactions that *can* do so but they're intended for user action
rather than programmatic action. (Even WM_CLOSE for GUI apps isn't
always the right answer, depending on app design.)
Typically graceful programmatic shutdown is only possible when both
parent and child are made by the same developers so that they can do
something custom to request it.
Klemens Morgenstern
> On 8/06/2016 21:19, Klemens Morgenstern wrote:
>> It doesn't work really, especially since SetConsoleCtrlHandler does only
>> allow two values (Ctrl+C & Ctrl+Break), which are both not clearly
>> termination requests, but more of a terminate command I think. Also to
>> do that, I would need to put all those child processes on a new process
>> group, which would mean, that Ctrl+C will NOT be transmitted to them.
>> That's also a dealbreaker.
>
> Ctrl+C is equivalent to SIGINT and Ctrl+Break is equivalent to
> SIGBREAK. So if you want to send one of those two signals then
> GenerateConsoleCtrlEvent is probably the right way to go. (You might
> need to use CreateRemoteThread to make the child raise it itself
> rather than doing it in the parent process though, which requires
> different versions for 32-bit vs. 64-bit processes.)
>
> If you want to raise SIGTERM, this is possible but you'd have to use
> CreateRemoteThread to call raise(SIGTERM) directly in the child
> process. Where this gets a little tricky is that you'd need to call
> the right version of "raise", which differs depending on the CRT the
> child process uses (static or dynamic, different compiler versions,
> etc) -- and the required code differs between 32-bit and 64-bit
> processes too.
to get it, and then look for the entry point, so I can pass it to
CreateRemoveThread, right? Do you have an example on how to do this? I
would need to be able to this for ANY process having the raise function.
> And of course both of the above are dependent on the console app
> developer actually intercepting and processing them as a graceful
> shutdown -- otherwise it's little different from a TerminateProcess.
>
> Services are really the only kind of executable that have a defined
> way to externally start/stop them programmatically. Other process
> types have interactions that *can* do so but they're intended for user
> action rather than programmatic action. (Even WM_CLOSE for GUI apps
> isn't always the right answer, depending on app design.)
>
> Typically graceful programmatic shutdown is only possible when both
> parent and child are made by the same developers so that they can do
> something custom to request it.
>
Gavin Lambert
SIGINT/SIGBREAK (it does SIGBREAK) at
https://github.com/walware/statet/blob/master/de.walware.statet.r.console.core/cppSendSignal/sendsignal.cpp
(link sharing for the lulz), but I don't think this is ASLR safe as it
assumes that the address of the kernel32 console control handler is the
same in both processes. The version that uses AttachConsole is probably
safer instead.
(Not that SIGINT/SIGBREAK would be a good idea for Boost.Process anyway,
since many console apps -- eg. cmd itself -- handle that as an
interactive command and might display an "are you sure" prompt or
similar. Which is also just like WM_CLOSE, incidentally.)
Other signals are more problematic, because signals aren't actually a
thing on Windows, and so the others are just emulated in the CRT. (And
processes not raising them themselves probably aren't expecting them
anyway.)
>> Typically graceful programmatic shutdown is only possible when both
>> parent and child are made by the same developers so that they can do
>> something custom to request it.
>
> Sad, but true. That's why it probably won't be part of boost.process.
you could probably do it (eg. use a pipe or other IPC mechanism to let
the parent trigger the child calling raise(SIGTERM) on itself.) I'm not
convinced it'd be worth the effort though.
Klemens Morgenstern
"The Windows API has done more to retard skill development than anything
since COBOL maintenance." (Larry O'Brien)
>
> (Not that SIGINT/SIGBREAK would be a good idea for Boost.Process anyway,
> since many console apps -- eg. cmd itself -- handle that as an
> interactive command and might display an "are you sure" prompt or
> similar. Which is also just like WM_CLOSE, incidentally.)
>
> Other signals are more problematic, because signals aren't actually a
> thing on Windows, and so the others are just emulated in the CRT. (And
> processes not raising them themselves probably aren't expecting them
> anyway.)
candidate to emulate signals if we need a custom receiver on the
child-side.
>>> Typically graceful programmatic shutdown is only possible when both
>>> parent and child are made by the same developers so that they can do
>>> something custom to request it.
>>
>> Sad, but true. That's why it probably won't be part of boost.process.
>
> Now, if you can guarantee that both parent and child use Boost.Process,
> you could probably do it (eg. use a pipe or other IPC mechanism to let
> the parent trigger the child calling raise(SIGTERM) on itself.) I'm not
> convinced it'd be worth the effort though.
>
However: I've decided to definitely not add this until boost.process is
reviewed - which will still take a lot of work.
We have not discussed four approaches to this problem and no one cuts it
in my opinion. If one shall be added, it must be tested before. So I'd
accept a pull request if it is either marked experimental or a
platform-extension. There will however not be a standard way to do this
until the first release of the library. I don't think having no SIGTERM
will be a deal-breaker and if we have an accepted boost.process it will
be much easier to experiment with that stuff.
I will however summarize the ideas and reopen the discussion when I
think boost.process is ready for experimental features.
Damien Buhl
We have been using the Boost.Process state from Boris Schäling for more
than 3 year in productive use on linux embedded devices, and one thing
that we had to change is to use vfork instead of the fork function.
The problem with the fork + execve is that it produce a RAM copy of the
process page, while vfork+execve just don't. And when the parent process
launching child is pretty big, imagine a jvm using some jni library
using Boost.Process... it duplicates the whole Jvm im RAM to then
deallocate all at the moment of execve. vfork doesn't have this issue on
linux.
Without the use of vfork, you end up in situations where you get the
following error: `boost::process::detail::posix_start: fork(2) failed:
Cannot allocate memory`
I think changing from fork to vfork is not much and brings alot of
advantages, but one must be aware that at_fork handler won't be called.
But this is not important as fork is used to do execve afterwards in
Boost.Process.
Cheers,
--
Damien Buhl
alias daminetreg
Klemens Morgenstern
> Hi Klemens,
>
> We have been using the Boost.Process state from Boris Schäling for more
> than 3 year in productive use on linux embedded devices, and one thing
> that we had to change is to use vfork instead of the fork function.
>
> The problem with the fork + execve is that it produce a RAM copy of the
> process page, while vfork+execve just don't. And when the parent process
> launching child is pretty big, imagine a jvm using some jni library
> using Boost.Process... it duplicates the whole Jvm im RAM to then
> deallocate all at the moment of execve. vfork doesn't have this issue on
> linux.
>
> Without the use of vfork, you end up in situations where you get the
> following error: `boost::process::detail::posix_start: fork(2) failed:
> Cannot allocate memory`
>
> I think changing from fork to vfork is not much and brings alot of
> advantages, but one must be aware that at_fork handler won't be called.
> But this is not important as fork is used to do execve afterwards in
> Boost.Process.
I appreciate your problem, but I am not sure it's that easy. There's on
major problem: vfork is removed from the posix standard as of
POSIX.1-2008. And I'm trying to conform to posix, not linux.
So I guess, I wouldn´t use vfork as default, but it might be possible to
add a property, which will cause the library to use that. I.e. you write:
boost::process::child c("java.exe", "overhead.jar",
boost::process::posix::use_vfork);
But I'd need to be able to check if vfork is available, so I can disable
the property if not.
Would that be sufficient for your problem?
Btw.: if you have any particular experience, that might help me improve
things, please share it. I'm always interested in that; the current
state is just based on how I'd use it.
Thanks,
Klemens
Gavin Lambert
> We have been using the Boost.Process state from Boris Schäling for more
> than 3 year in productive use on linux embedded devices, and one thing
> that we had to change is to use vfork instead of the fork function.
>
> The problem with the fork + execve is that it produce a RAM copy of the
> process page, while vfork+execve just don't. And when the parent process
> launching child is pretty big, imagine a jvm using some jni library
> using Boost.Process... it duplicates the whole Jvm im RAM to then
> deallocate all at the moment of execve. vfork doesn't have this issue on
> linux.
MMU architectures shouldn't have this issue, as fork does a
shared-memory-copy-on-write mapping so that the pages aren't actually
duplicated unless written to, and the subsequent exec* unmaps the pages
so this never happens (other than a bit of stack).
NOMMU architectures don't support those kinds of mappings, so have to be
rewritten to use vfork instead (which is generally unsafe unless
immediately followed by exec*, unless you know what you're doing).
Although the last time that I played with NOMMU, fork used to just fail;
perhaps it's been changed to make copies of all the pages instead? If
so, that would indeed be problematic for large parent processes.
Damien Buhl
boost::process::posix::use_vfork(_if_possible) then yes for me all will
be fine.
But on the other hand from a library design point of view, shouldn't the
library have the best smart default in terms of performance and overhead
on a given platform ? Instead of having a flag telling : please do it
the same but efficiently ? Because on linux vfork is nothing but
obsoleted and for a scenario of using execve looks better to me.
> On 17/06/2016 01:21, Gavin Lambert wrote:
>>
>> Is this specifically for NOMMU linux?
>>
>> MMU architectures shouldn't have this issue, as fork does a
>> shared-memory-copy-on-write mapping so that the pages aren't actually
>> duplicated unless written to, and the subsequent exec* unmaps the
>> pages so this never happens (other than a bit of stack).
>>
>> NOMMU architectures don't support those kinds of mappings, so have to
>> be rewritten to use vfork instead (which is generally unsafe unless
>> immediately followed by exec*, unless you know what you're doing).
>>
>> Although the last time that I played with NOMMU, fork used to just
>> fail; perhaps it's been changed to make copies of all the pages
>> instead? If so, that would indeed be problematic for large parent
>> processes.
It's with an MMU but the parent process on this product is a
memory-hungry-monster-jvm and the fork fails.
I can be wrong, and it looks like you know more from what's happening
there, but as long as I can remember the issue arose due to virtual
memory commit, because fork on linux even though is optimized to not
copy the whole process pages but to do copy-on-write, still needs to
commit virtual memory, and overcommitment is not always allowed, or by
default is heuristically allowed.
Quoting man proc, the default is 0:
> /proc/sys/vm/overcommit_memory
> This file contains the kernel virtual memory accounting
> mode. Values are:
>
> 0: heuristic overcommit (this is the default)
> 1: always overcommit, never check
> 2: always check, never overcommit
So it might fail, and it happens to fail often on a 80% RSS usage of the
device RAM from the parent process which will do the fork. While vfork
doesn't need to commit vm memory, or at least not so much as the parent
process, but just what's needed for pid_t and a call to execve I find it
a better alternative when someone does only do a fork followed by execve.
But there might be considerations I don't understand, I never worked on
the linux kernel process creation code, and so I'm just sharing from a
dumb user point-of-view.
Perhaps we could just remove the jvm, but as I proposed this to the java
team they didn't invited me for the coffee anymore. :p
Klemens Morgenstern
ought to go with. Please note, that I'm trying to provide two platforms:
Posix & Windows. Not Linux & Windows. Thereby I want to provide the most
common way for both platforms; and though I really appreciate your
scenario, I would not consider it the common way.
I think you underestimate the dangers of vfork; because you know, two
processes sharing memory (including the stack!) can be rather fun.
There's a reason it was removed and so it will be optional in
boost.process.
It would be a platform extension, so no, this would NOT compile with
windows. I thought about turning the platform extensions into NOPs ,
but that's just too weird - especially since things like signal(SIGCHLD,
...) are also provided. There you should rather use the preprocessor and
put an #ifdef there - then it's obvious what you're doing. Also I'd need
a #define to know whether vfork is available, that I would probably
provide. So you could then write something like that:
child c("ls"
#if defined(BOOST_POSIX_HAS_VFORK)
, posix::use_vfork
#endif
);
I guess I can check that via 'CLONE_VFORK'.
>>> Is this specifically for NOMMU linux?
>>>
>>> MMU architectures shouldn't have this issue, as fork does a
>>> shared-memory-copy-on-write mapping so that the pages aren't actually
>>> duplicated unless written to, and the subsequent exec* unmaps the
>>> pages so this never happens (other than a bit of stack).
>>>
>>> NOMMU architectures don't support those kinds of mappings, so have to
>>> be rewritten to use vfork instead (which is generally unsafe unless
>>> immediately followed by exec*, unless you know what you're doing).
>>>
>>> Although the last time that I played with NOMMU, fork used to just
>>> fail; perhaps it's been changed to make copies of all the pages
>>> instead? If so, that would indeed be problematic for large parent
>>> processes.
> Hi Gavin ;)
>
> It's with an MMU but the parent process on this product is a
> memory-hungry-monster-jvm and the fork fails.
instead of - you know - java.io.process. But I guess you want
performance and your java-developers want a job, so JNI is the way to go?
> I can be wrong, and it looks like you know more from what's happening
> there, but as long as I can remember the issue arose due to virtual
> memory commit, because fork on linux even though is optimized to not
> copy the whole process pages but to do copy-on-write, still needs to
> commit virtual memory, and overcommitment is not always allowed, or by
> default is heuristically allowed.
process and there would be no way of getting to know it.
Damien Buhl
> [snip]
use #ifdef at all, that's why I was asking. But that's all the
difficulty of Boost.Process : it's pretty hard to abstract something
which is really different in the little detail.
Just for reference, this is what posix_spawn does in the glibc, which is
roughly equivalent to what Boost.Process provides via the child class,
they are also feared about using vfork, so you are surely right to avoid
it as much as possible :
In glibc /sysdeps/posix/spawni.c
> /* Generate the new process. */
> if ((flags & POSIX_SPAWN_USEVFORK) != 0
> /* If no major work is done, allow using vfork. Note that we
> might perform the path searching. But this would be done by
> a call to execvp(), too, and such a call must be OK according
> to POSIX. */
> || ((flags & (POSIX_SPAWN_SETSIGMASK | POSIX_SPAWN_SETSIGDEF
> | POSIX_SPAWN_SETSCHEDPARAM | POSIX_SPAWN_SETSCHEDULER
> | POSIX_SPAWN_SETPGROUP | POSIX_SPAWN_RESETIDS)) == 0
> && file_actions == NULL))
> new_pid = __vfork ();
> else
> new_pid = __fork ();
>
The rationale of posix_spawn here is a good addition to what we discuss
I think :
http://pubs.opengroup.org/onlinepubs/009695399/functions/posix_spawn.html#tag_03_427_08
And as your goal is Posix + Windows, I understand you don't want to deal
with specialities of linux / solaris / ... so I think you're right it's
safer the default to be fork.
>> [snip]
>> It's with an MMU but the parent process on this product is a
>> memory-hungry-monster-jvm and the fork fails.
> It seems a bit strange to me, that you would use boost.process here,
> instead of - you know - java.io.process. But I guess you want
> performance and your java-developers want a job, so JNI is the way to go?
>
library we had for long, where we added a JNI interface for it, which is
launching at one point processes that are not developed internally and
that we cannot implement as library because we don't have the source code.
Additionally I believe that in the version of jvm implementation used
(oracle - arm - early 1.7) I think java.io.process does fork+exec, which
has been changed after because of similar issues:
http://bugs.java.com/view_bug.do?bug_id=6868160
Thanks for dealing with my wishes to vfork :)
Klemens Morgenstern
> I think :
> http://pubs.opengroup.org/onlinepubs/009695399/functions/posix_spawn.html#tag_03_427_08
>
> And as your goal is Posix + Windows, I understand you don't want to deal
> with specialities of linux / solaris / ... so I think you're right it's
> safer the default to be fork.
>
>>> [snip]
>>>
>>> It's with an MMU but the parent process on this product is a
>>> memory-hungry-monster-jvm and the fork fails.
>> It seems a bit strange to me, that you would use boost.process here,
>> instead of - you know - java.io.process. But I guess you want
>> performance and your java-developers want a job, so JNI is the way to go?
>>
> It's off-topic but to explain: The fact is that they just reuse a C++
> library we had for long, where we added a JNI interface for it, which is
> launching at one point processes that are not developed internally and
> that we cannot implement as library because we don't have the source code.
>
> Additionally I believe that in the version of jvm implementation used
> (oracle - arm - early 1.7) I think java.io.process does fork+exec, which
> has been changed after because of similar issues:
> http://bugs.java.com/view_bug.do?bug_id=6868160
>
> Thanks for dealing with my wishes to vfork :)
>
- it's not documented though. If you pass posix::use_vfork to the
function it will work.
BUT: I just removed the test for that, because it ran locally (on my
debian VM) but travis didn't like it very much - for some unknown
reason. Maybe you have some experience and might be able to find a fix.
I'd be really happy to have a test for this - I'd just need travis to work.
Gavin Lambert
>> But on the other hand from a library design point of view, shouldn't the
>> library have the best smart default in terms of performance and overhead
>> on a given platform ? Instead of having a flag telling : please do it
>> the same but efficiently ? Because on linux vfork is nothing but
>> obsoleted and for a scenario of using execve looks better to me.
>
> Again: obsolete and now removed in the posix-standard and that's what I
> ought to go with. Please note, that I'm trying to provide two platforms:
> Posix & Windows. Not Linux & Windows. Thereby I want to provide the most
> common way for both platforms; and though I really appreciate your
> scenario, I would not consider it the common way.
fork+exec, for that matter) was deprecated in favour of posix_spawn.
Which appears to be less well documented, but does exist in some
environments at least.
Damien Buhl
>
> Alright, the feature is in now, you find that in
> boost/process/posix.hpp - it's not documented though. If you pass
> posix::use_vfork to the function it will work.
>
> BUT: I just removed the test for that, because it ran locally (on my
> debian VM) but travis didn't like it very much - for some unknown
> reason. Maybe you have some experience and might be able to find a
> fix. I'd be really happy to have a test for this - I'd just need
> travis to work.
>
travis-ci.
Damien Buhl
> On 17/06/2016 21:45, Klemens Morgenstern wrote:
>>> But on the other hand from a library design point of view, shouldn't
>>> the
>>> library have the best smart default in terms of performance and
>>> overhead
>>> on a given platform ? Instead of having a flag telling : please do it
>>> the same but efficiently ? Because on linux vfork is nothing but
>>> obsoleted and for a scenario of using execve looks better to me.
>>
>> Again: obsolete and now removed in the posix-standard and that's what I
>> ought to go with. Please note, that I'm trying to provide two platforms:
>> Posix & Windows. Not Linux & Windows. Thereby I want to provide the most
>> common way for both platforms; and though I really appreciate your
>> scenario, I would not consider it the common way.
>
> Possibly of interest is that according to the notes vfork+exec (and
> fork+exec, for that matter) was deprecated in favour of posix_spawn.
> Which appears to be less well documented, but does exist in some
> environments at least.
>
another posix_spawn, brought to another level though) :
> I think :
> http://pubs.opengroup.org/onlinepubs/009695399/functions/posix_spawn.html#tag_03_427_08
> /* Generate the new process. */
> if ((flags & POSIX_SPAWN_USEVFORK) != 0
> /* If no major work is done, allow using vfork. Note that we
> might perform the path searching. But this would be done by
> a call to execvp(), too, and such a call must be OK according
> to POSIX. */
> || ((flags & (POSIX_SPAWN_SETSIGMASK | POSIX_SPAWN_SETSIGDEF
> | POSIX_SPAWN_SETSCHEDPARAM | POSIX_SPAWN_SETSCHEDULER
> | POSIX_SPAWN_SETPGROUP | POSIX_SPAWN_RESETIDS)) == 0
> && file_actions == NULL))
> new_pid = __vfork ();
> else
> new_pid = __fork ();
>