waitpid and reading from pipe - sequence?

[Ian Pilcher]

I am reading the output of an external command with the usual sequence
of pipe[2], fork, dup2, execv, etc. The parent process currently
reads the child's output from the pipe and then calls waitpid. Pretty
straightforward.

I'm trying to refactor the code into more digestible chunks, and I'd
like to separate the "lower level" process management stuff into its
own function. This would mean that the parent process would call
waitpid before reading the child's output from the pipe.

This seems to work fine in my initial smoke testing (on Fedora 19), but
I have a slightly uncomfortable feeling that the fate of any "in flight"
data in the pipe may not be guaranteed in this situation. Thus far I
haven't been able to come up with search terms that shed any light on
the question.

Am I just being paranoid (or am I not being paranoid enough)?

Thanks!

--
========================================================================
Ian Pilcher arequ...@gmail.com
Sometimes there's nothing left to do but crash and burn...or die trying.
========================================================================

[Nobody]

On Thu, 12 Sep 2013 15:17:30 -0500, Ian Pilcher wrote:

> I am reading the output of an external command with the usual sequence
> of pipe[2], fork, dup2, execv, etc. The parent process currently
> reads the child's output from the pipe and then calls waitpid. Pretty
> straightforward.
>
> I'm trying to refactor the code into more digestible chunks, and I'd
> like to separate the "lower level" process management stuff into its
> own function. This would mean that the parent process would call
> waitpid before reading the child's output from the pipe.
>
> This seems to work fine in my initial smoke testing (on Fedora 19), but
> I have a slightly uncomfortable feeling that the fate of any "in flight"
> data in the pipe may not be guaranteed in this situation. Thus far I
> haven't been able to come up with search terms that shed any light on
> the question.
>
> Am I just being paranoid (or am I not being paranoid enough)?

The latter. You typically shouldn't call waitpid() until the child has
closed its end of the pipe; at least, not without the WNOHANG flag.

You'll get away with it provided that that child doesn't write more than
pathconf(_PC_PIPE_BUF) bytes to the pipe. If it tries to write any more
than that, you'll get a deadlock: the child will block waiting for the
reader (i.e. the parent) to consume some data to make space in the pipe,
which won't happen because the parent will be blocked on waitpid().

[Jorgen Grahn]

On Thu, 2013-09-12, Ian Pilcher wrote:
> I am reading the output of an external command with the usual sequence
> of pipe[2], fork, dup2, execv, etc. The parent process currently
> reads the child's output from the pipe and then calls waitpid. Pretty
> straightforward.
>
> I'm trying to refactor the code into more digestible chunks, and I'd
> like to separate the "lower level" process management stuff into its
> own function.

In my limited experience, it's better to keep the "lower levels"
as classic, non-generalized Unix, and design your application-specific
stuff with nice and tidy interfaces suitable for use with the Unix
model.

I've tried in the past to e.g. write general, reusable select-
driven TCP socket server frameworks. It never turned out well.

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .

[Ian Pilcher]

On 09/12/2013 03:39 PM, Nobody wrote:
>
> The latter. You typically shouldn't call waitpid() until the child has
> closed its end of the pipe; at least, not without the WNOHANG flag.
>
> You'll get away with it provided that that child doesn't write more than
> pathconf(_PC_PIPE_BUF) bytes to the pipe. If it tries to write any more
> than that, you'll get a deadlock: the child will block waiting for the
> reader (i.e. the parent) to consume some data to make space in the pipe,
> which won't happen because the parent will be blocked on waitpid().
>

That makes sense. Glad I asked.

I have to admit that I'm still tempted to "get away with it". It makes
the code so much cleaner, and _PC_PIPE_BUF is 4096 on Linux, which is
way more than I need.

I guess the main danger of that approach would be if the external
command "goes nuts" and spews more than 4096 bytes of output. Of
course, I'm already going to hang if the external command never exits.

Aargh! Now I'm not so glad I asked.

[Rainer Weikusat]

It is also pretty useless: 'Generalized frameworks' which enable
'saving' some odd 400 lines of code are not worth the effort of learning
how to use them.

[Rainer Weikusat]

Ian Pilcher <arequ...@gmail.com> writes:
> On 09/12/2013 03:39 PM, Nobody wrote:
>>
>> The latter. You typically shouldn't call waitpid() until the child has
>> closed its end of the pipe; at least, not without the WNOHANG flag.
>>
>> You'll get away with it provided that that child doesn't write more than
>> pathconf(_PC_PIPE_BUF) bytes to the pipe. If it tries to write any more
>> than that, you'll get a deadlock: the child will block waiting for the
>> reader (i.e. the parent) to consume some data to make space in the pipe,
>> which won't happen because the parent will be blocked on waitpid().
>>
>
> That makes sense. Glad I asked.
>
> I have to admit that I'm still tempted to "get away with it". It makes
> the code so much cleaner, and _PC_PIPE_BUF is 4096 on Linux, which is
> way more than I need.

You could move the waitpid into a SIGCHLD handler.

[Jorgen Grahn]

Yes, that was the other part I realized. People need to be intimate
with the real Unix interfaces anyway, and hiding them hinders that.

[Ian Pilcher]

On 09/13/2013 06:26 AM, Rainer Weikusat wrote:
>
> You could move the waitpid into a SIGCHLD handler.
>

I did think of that. This is a multi-threaded program, though, so doing
anything with signals is painful.

I ultimately decided to split my "lower level" function in two -- one
function to spawn the external command and return the read end of the
pipe and a second function to clean everything up.

Thanks!

[Ian Pilcher]

On 09/12/2013 04:55 PM, Jorgen Grahn wrote:
> I've tried in the past to e.g. write general, reusable select-
> driven TCP socket server frameworks. It never turned out well.

Who said anything about a reusable framework? I was just splitting my
code up into more manageable/readable functions.

[Rainer Weikusat]

Ian Pilcher <arequ...@gmail.com> writes:
> On 09/13/2013 06:26 AM, Rainer Weikusat wrote:
>>
>> You could move the waitpid into a SIGCHLD handler.
>>
>
> I did think of that. This is a multi-threaded program, though, so doing
> anything with signals is painful.

If you don't want the exit status of the process, that would be just a
matter of having a SIGCHLD handler run

while (waitpid(-1, NULL, WNOHANG) > 0);

waitpid is async-signal safe so there won't be 'bad interactions' with
other threads of the program and it really wouldn't matter which thread
executed it.

[Jorgen Grahn]

On Thu, 2013-09-19, Ian Pilcher wrote:
> On 09/12/2013 04:55 PM, Jorgen Grahn wrote:
>> I've tried in the past to e.g. write general, reusable select-
>> driven TCP socket server frameworks. It never turned out well.
>
> Who said anything about a reusable framework?

I did, and it was just an illustration. My point was more general:
"don't fight the APIs".

> I was just splitting my
> code up into more manageable/readable functions.

[Ian Pilcher]

On 09/19/2013 11:00 AM, Rainer Weikusat wrote:

> Ian Pilcher <arequ...@gmail.com> writes:
>
> If you don't want the exit status of the process, that would be just a
> matter of having a SIGCHLD handler run
>
> while (waitpid(-1, NULL, WNOHANG) > 0);
>

Unfortunately, I do care about the exit status.