Another GAWK issue (interaction between background process and cmd|getline)

[Kenny McCormack]

The problem: Using the "fork" extension lib, you can create a "background
process" in your GAWK script. If you then do a "cmd|getline" operation on
a "bad" pipeline twice, while the "background process" is still running,
the second one (that is, the second "cmd|getline") hangs until the
background process exits.

This is all rather obscure, so let's start with first principles:

$ gawk4 'BEGIN { cmd="echo one;echo two";cmd|getline;print;cmd|getline;print}'
one
two
$

All well and good.
Next, we introduce a command that generates no output on the standard output:

$ gawk4 'BEGIN { cmd="ls ksdjfhkdsjhfk";cmd|getline;print;cmd|getline;print}'
ls: cannot access ksdjfhkdsjhfk: No such file or directory


$

Again, all well and good. Now, we bring in the fork() extension lib (part
of the GAWK distribution, presumably vetted, working code). I made no
changes in this lib; it is compiled automatically as part of the GAWK build
process. We use the "xmessage" program as our "background process".

$ gawk4 -l .libs/fork 'BEGIN { if (!fork()) {system("xmessage \047This is a test\047");exit} print "In the parent...";cmd="echo one;echo two";print "First getline...";cmd|getline;print "Second getline...";cmd|getline;print "Done"}'
In the parent...
First getline...
Second getline...
Done
$

Still, well and good. Now, we do it with the failing "ls" command:

$ gawk4 -l .libs/fork 'BEGIN { if (!fork()) {system("xmessage \047This is a test\047");exit} print "In the parent...";cmd="ls ksjdfhkjdhf";print "First getline...";cmd|getline;print "Second getline...";cmd|getline;print "Done"}'
In the parent...
First getline...
ls: cannot access ksjdfhkjdhf: No such file or directory
Second getline...
{ Here's the problem }
Done
$

At the point marked "Here's the problem", the GAWK program hangs until I
click the "okay" button on the "xmessage" window. Once I do that, the GAWK
programs continues as normal.

The workaround is to check the result of the first getline, and only
proceed if you got something, like this:

...
cmd | getline var1
if (var1)
cmd | getline var2
...

But obviously, that's not desirable. Nor should it be necessary...

Note: I found this problem in my own code, in a real world use case, using
my own extension lib to generate the background process. Having found the
problem there, I retro-fitted it back to the distro-supplied "fork"
extension lib.

--
Here's a simple test for Fox viewers:

1) Sit back, close your eyes, and think (Yes, I know that's hard for you).
2) Think about and imagine all of your ridiculous fantasies about Barack Obama.
3) Now, imagine that he is white. Cogitate on how absurd your fantasies
seem now.

See? That wasn't hard, was it?

[Andrew Schorr]

On Tuesday, April 7, 2015 at 8:01:44 AM UTC-4, Kenny McCormack wrote:
> $ gawk4 -l .libs/fork 'BEGIN { if (!fork()) {system("xmessage \047This is a test\047");exit} print "In the parent...";cmd="ls ksjdfhkjdhf";print "First getline...";cmd|getline;print "Second getline...";cmd|getline;print "Done"}'
> In the parent...
> First getline...
> ls: cannot access ksjdfhkjdhf: No such file or directory
> Second getline...
> { Here's the problem }
> Done
> $
>
> At the point marked "Here's the problem", the GAWK program hangs until I
> click the "okay" button on the "xmessage" window. Once I do that, the GAWK
> programs continues as normal.

I am unable to duplicate this problem on my linux system. What platform are you using? It sounds like the parent gawk is waiting to reap the child process before it exits. For example, when happens when you try this?

bash-4.2$ time gawk -l fork -l time 'BEGIN {if (fork()) { print "Parent is exiting immediately"; exit 0} print "Child will pause for 1 second"; sleep(1); print "child exiting now"}'
Parent is exiting immediately
Child will pause for 1 second

real 0m0.002s
user 0m0.000s
sys 0m0.001s
bash-4.2$ child exiting now

Regards,
Andy

[Kenny McCormack]

In article <99d9ab5d-6ae2-45dc...@googlegroups.com>,

Andrew Schorr <asc...@telemetry-investments.com> wrote:
>On Tuesday, April 7, 2015 at 8:01:44 AM UTC-4, Kenny McCormack wrote:
>> $ gawk4 -l .libs/fork 'BEGIN { if (!fork()) {system("xmessage \047This is a
>test\047");exit} print "In the parent...";cmd="ls ksjdfhkjdhf";print "First
>getline...";cmd|getline;print "Second getline...";cmd|getline;print "Done"}'
>> In the parent...
>> First getline...
>> ls: cannot access ksjdfhkjdhf: No such file or directory
>> Second getline...
>> { Here's the problem }
>> Done
>> $
>>
>> At the point marked "Here's the problem", the GAWK program hangs until I
>> click the "okay" button on the "xmessage" window. Once I do that, the GAWK
>> programs continues as normal.
>
>I am unable to duplicate this problem on my linux system.

Interesting. I'm definitely not making it up. I've verified it on 3
platforms (Linux x64, Linux ARM-32, and Mac OSX-64). The command given above,
copied exactly, does the same thing on all 3 platforms. I'm guessing that
you didn't test exactly that - maybe you don't have xmessage on your
systems?

Note: It could be specific to xmessage - I haven't tested with any other
process. It could also be specific to the released version of GAWK 4.1.1.
Are you, by any chance, using a later (development) version of GAWK?

>What platform are you
>using? It sounds like the parent gawk is waiting to reap the child process
>before it exits.

I'm pretty sure that it has something to do with child reaping.

For what it is worth, I've never liked the the definition of the basic Unix
"wait" call. The way that it waits for (essentially) "anything" to happen,
makes it seem to me to be too wide a net to be useful. I know that there
are various other wait*() functions available these days (waitpid, wait3,
wait4, etc), but I wouldn't be surprised to find that some of the
underlying calls that GAWK ends up using are, at the last step, using plain
old wait().

And that that is causing confusion when there is more than one "other
process" running...

>For example, when happens when you try this?
>bash-4.2$ time gawk -l fork -l time 'BEGIN {if (fork()) { print "Parent is
>exiting immediately"; exit 0} print "Child will pause for 1 second"; sleep(1);
>print "child exiting now"}'
>Parent is exiting immediately
>Child will pause for 1 second
>
>real 0m0.002s
>user 0m0.000s
>sys 0m0.001s
>bash-4.2$ child exiting now

I haven't tried this, because I can't see what it would be trying to
demonstrate.

Could you say a little bit more about what you're going for here?

P.S. My first impression was (like yours) that it had to do with child
reaping (i.e., wait*()), but it might instead have to do with open files.
I know that when you have multiple processes running, it is very important
to close any unused file descriptors, or else things can hang in mysterious
ways. Maybe xmessage is using a pipe or something and something is not
being closed... So, not clear that this is necessarily a GAWK problem, per
se, but still, I find it all interesting...

--

There are many self-professed Christians who seem to think that because
they believe in Jesus' sacrifice they can reject Jesus' teachings about
how we should treat others. In this country, they show that they reject
Jesus' teachings by voting for Republicans.

[Andrew Schorr]

On Wednesday, April 8, 2015 at 6:44:11 AM UTC-4, Kenny McCormack wrote:
> Interesting. I'm definitely not making it up. I've verified it on 3
> platforms (Linux x64, Linux ARM-32, and Mac OSX-64). The command given above,
> copied exactly, does the same thing on all 3 platforms. I'm guessing that
> you didn't test exactly that - maybe you don't have xmessage on your
> systems?

I do have xmessage, and I ran your test exactly. There is no problem on my system.

> >For example, when happens when you try this?
> >bash-4.2$ time gawk -l fork -l time 'BEGIN {if (fork()) { print "Parent is
> >exiting immediately"; exit 0} print "Child will pause for 1 second"; sleep(1);
> >print "child exiting now"}'
> >Parent is exiting immediately
> >Child will pause for 1 second
> >
> >real 0m0.002s
> >user 0m0.000s
> >sys 0m0.001s
> >bash-4.2$ child exiting now
>
> I haven't tried this, because I can't see what it would be trying to
> demonstrate.

It is trying to isolate the problem. I can't help you if you won't run the test.

> Could you say a little bit more about what you're going for here?

Removing the getline confusion to create a simpler example that proves this is a child reaping problem.

Regards,
Andy