osx - awk: i/o error occurred while closing /dev/stdin

[brad dunbar]

When running awk within tup, I get the following error message:

awk: i/o error occurred while closing /dev/stdin

It doesn't seem to be causing any issues and the command runs manually without complaint.  However, I'm worried it's causing issues I'm just not aware of yet.

It seems to be specific to osx because I've run the some code under linux without the warning.  Has anyone seen this issue before?

I can faithfully reproduce this with a simplistic Tupfile of the form

: in |> awk '/foo/{print}' %f > %o |> out

which produces

brad:~/dev/tuptest$ tup upd
[ tup ] Scanning filesystem...0.001s
[ tup ] No tup.config changes.
[ tup ] Parsing Tupfiles...
[    0/1    ] .
[    1/1    ]
[ tup ] No files to delete.
[ tup ] Executing Commands...
[    0/1    ] awk '/foo/{print}' in > out
awk: i/o error occurred while closing /dev/stdin
 input record number 5, file in
 source line number 1
[    1/1    ]

[ tup ] Updated. 

 Thanks!

[Anatol Pomozov]

Hi

On Sat, Jul 16, 2011 at 8:44 AM, brad dunbar <dunb...@gmail.com> wrote:

When running awk within tup, I get the following error message:

awk: i/o error occurred while closing /dev/stdin

I believe the warning happens because of this piece of code: https://github.com/gittup/tup/blob/master/src/tup/server/fuse_server.c#L265

And I think the reason is the difference in awk implementations (BSD vs GNU). According to information I found the BSD awk expects that all standard streams a open at the end of awk execution.

[1] http://www.zsh.org/mla/workers/2000/msg02112.html

[2] http://mail-index.netbsd.org/netbsd-users/2009/06/15/msg003894.html

So to make awk silent you need to provide some STDIN for it, add "< /dev/null"

: in |> awk '/foo/{print}' %f < /dev/null > %o |> out

 

It doesn't seem to be causing any issues and the command runs manually without complaint.  However, I'm worried it's causing issues I'm just not aware of yet.

It seems to be specific to osx because I've run the some code under linux without the warning.  Has anyone seen this issue before?

I can faithfully reproduce this with a simplistic Tupfile of the form

: in |> awk '/foo/{print}' %f > %o |> out

which produces

brad:~/dev/tuptest$ tup upd
[ tup ] Scanning filesystem...0.001s
[ tup ] No tup.config changes.
[ tup ] Parsing Tupfiles...
[    0/1    ] .
[    1/1    ]
[ tup ] No files to delete.
[ tup ] Executing Commands...
[    0/1    ] awk '/foo/{print}' in > out
awk: i/o error occurred while closing /dev/stdin
 input record number 5, file in
 source line number 1
[    1/1    ]

[ tup ] Updated. 

 Thanks!

--
tup-users mailing list
email: tup-...@googlegroups.com
unsubscribe: tup-users+...@googlegroups.com
options: http://groups.google.com/group/tup-users?hl=en

[Mike Shal]

On Sat, Jul 16, 2011 at 12:45 PM, Anatol Pomozov
<anatol....@gmail.com> wrote:
> Hi
>
> On Sat, Jul 16, 2011 at 8:44 AM, brad dunbar <dunb...@gmail.com> wrote:
>>
>> When running awk within tup, I get the following error message:
>>
>> awk: i/o error occurred while closing /dev/stdin
>
> I believe the warning happens because of this piece of
> code: https://github.com/gittup/tup/blob/master/src/tup/server/fuse_server.c#L265

Yeah, that would be the only reason I can think of that would cause
this error message.

> And I think the reason is the difference in awk implementations (BSD vs
> GNU). According to information I found the BSD awk expects that all standard
> streams a open at the end of awk execution.
> [1] http://www.zsh.org/mla/workers/2000/msg02112.html
> [2] http://mail-index.netbsd.org/netbsd-users/2009/06/15/msg003894.html
>
> So to make awk silent you need to provide some STDIN for it, add "<
> /dev/null"
> : in |> awk '/foo/{print}' %f < /dev/null > %o |> out

Does this work-around work for you Brad? Tup needs to close stdin to
make sure that no sub-process inadvertently hangs the build. Or maybe
tup could automatically dup it do /dev/null so that programs have a
valid stdin, but reading from it won't hang the build?

Does this patch help?

diff --git a/src/tup/server/fuse_server.c b/src/tup/server/fuse_server.c
index a9270ad..69ebb2c 100644
--- a/src/tup/server/fuse_server.c
+++ b/src/tup/server/fuse_server.c
@@ -251,18 +251,28 @@ int server_exec(struct server *s, int
vardict_fd, int dfd, const char *cmd,
goto err_rm_group;
}
if(pid == 0) {
+ int nullfd;
tup_lock_close();

if(virt_tup_chdir(dtent, s) < 0) {
exit(1);
}
server_setenv(vardict_fd);
- /* Close down stdin - it can't reliably be used during the
- * build (for example, when building in parallel, multiple
- * programs would have to fight over who gets it, which is just
- * nonsensical).
+ /* Use /dev/null for stdin, since stdin can't reliably be used
+ * during the build (for example, when building in parallel,
+ * multiple programs would have to fight over who gets it,
+ * which is just nonsensical).
*/
- close(0);
+ nullfd = open("/dev/null", O_RDONLY);
+ if(nullfd < 0) {
+ perror("/dev/null");
+ exit(1);
+ }
+ if(dup2(nullfd, 0) < 0) {
+ perror("dup2");
+ exit(1);
+ }
+ close(nullfd);

execl("/bin/sh", "/bin/sh", "-e", "-c", cmd, NULL);
perror("execl");

-Mike

[Anatol Pomozov]

Would it be better to open a nullfd only once in _init() and just reuse/dup2 here?

 

+               if(nullfd < 0) {
+                       perror("/dev/null");
+                       exit(1);
+               }
+               if(dup2(nullfd, 0) < 0) {

0 -> STDIN_FILENO 

[brad dunbar]

Thanks for the quick response!

The work-around above does in fact work for me (at least it silences awk).  I'll try the patch above shorty for a more permanent solution.

Thanks!
-brad

--

[Anatol Pomozov]

On Sat, Jul 16, 2011 at 10:12 AM, Mike Shal <mar...@gmail.com> wrote:

On Sat, Jul 16, 2011 at 12:45 PM, Anatol Pomozov
<anatol....@gmail.com> wrote:
> Hi
>
> On Sat, Jul 16, 2011 at 8:44 AM, brad dunbar <dunb...@gmail.com> wrote:
>>
>> When running awk within tup, I get the following error message:
>>
>> awk: i/o error occurred while closing /dev/stdin
>
> I believe the warning happens because of this piece of
> code: https://github.com/gittup/tup/blob/master/src/tup/server/fuse_server.c#L265

Yeah, that would be the only reason I can think of that would cause
this error message.

> And I think the reason is the difference in awk implementations (BSD vs
> GNU). According to information I found the BSD awk expects that all standard
> streams a open at the end of awk execution.
> [1] http://www.zsh.org/mla/workers/2000/msg02112.html
> [2] http://mail-index.netbsd.org/netbsd-users/2009/06/15/msg003894.html
>
> So to make awk silent you need to provide some STDIN for it, add "<
> /dev/null"
> : in |> awk '/foo/{print}' %f < /dev/null > %o |> out

Does this work-around work for you Brad? Tup needs to close stdin to
make sure that no sub-process inadvertently hangs the build. Or maybe
tup could automatically dup it do /dev/null so that programs have a
valid stdin, but reading from it won't hang the build?

Does this patch help?

Yep it fixes the issue on osx 10.6.8

 

[Mike Shal]

On Mon, Jul 18, 2011 at 8:51 PM, Anatol Pomozov
<anatol....@gmail.com> wrote:
>
>
> On Sat, Jul 16, 2011 at 10:12 AM, Mike Shal <mar...@gmail.com> wrote:
>> Does this patch help?
>
> Yep it fixes the issue on osx 10.6.8

Ok, I put a similar fix into master. It just opens /dev/null once when
the fuse server starts, as you suggested.

Thanks,
-Mike