Exec() hangs if 4k written to StdErr, even with StdErr.ReadAll()
John Frensen
a child app. I've found that Exec() hangs if the child
writes more than about 4 kilobytes to StdErr, even if the main
app is doing the standard StdErr.ReadAll() to empty the
file stream.
Below are a trivial main app (consumer.js) and child app
(producer.js) that demonstrate the problem. I can understand
the producer stalling when the StdErr buffer fills, but it
should resume when it has available space again. I don't
see why the producer goes into a coma.
If you comment out the write to StdErr and uncomment the
write to StdOut, everything works fine. If you reduce the
number of chars written to StdErr to 3500, e.g., everything works
fine.
producer.js:
-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-
var strData = "123456789012345678901234567890";
for (var nChars = 0; nChars < 10000; nChars += strData.length + 2) {
//WScript.StdOut.Writeline(strData);
WScript.StdErr.Writeline(strData);
}
-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-
consumer.js, launched with "cscript.exe consumer.js":
-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-
WScript.StdOut.Writeline("This is WSH version " + WScript.Version);
var wshShell = WScript.CreateObject("WScript.Shell");
var strStdout = "";
var strStderr = "";
var execObj = wshShell.Exec("cscript.exe //nologo producer.js");
while (execObj.Status == 0) {
while (!execObj.Stdout.AtEndOfStream)
WScript.StdOut.Writeline(execObj.Stdout.ReadAll());
while (!execObj.Stderr.AtEndOfStream)
WScript.StdErr.Writeline(execObj.Stderr.ReadAll());
WScript.Sleep(100);
}
WScript.StdOut.Writeline("status=" + execObj.Status);
-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-
Tom Lavedas
(if there are newline characters in the output stream) or Read(nByteCount),
where nByteCount is less than the buffer's size (~4kB) to empty the StdErr
buffer. You just can't safely use the ReadAll method, because it is waiting
for StdErr to be released by the called process - which becomes blocked
before the ReadAll can get access to empty it.
Tom Lavedas
===========
John Frensen
> Yes, this is a known issue. The solution is to use a loop around
> a Readline (if there are newline characters in the output stream)
> or Read(nByteCount), where nByteCount is less than the buffer's size
> (~4kB) to empty the StdErr buffer. You just can't safely use the ReadAll
> method, because it is waiting for StdErr to be released by the called
> process - which becomes blocked before the ReadAll can get access to
> empty it.
The more I look into this, the worse it looks -- MS really messed this
up badly. The problem is that 1) if the child app fills the stdout or
stderr buffer, it will hang forever; and 2) the AtEndOfStream property
of stdout and stderr will hang the main app until the child writes data
to the textstream or the child app exits. So the following "solution"
is unsafe, because the main app can't predict whether the child will
write to stdout or stderr at all. If it tests execObj.Stdout.AtEndOfStream
or execObj.Stderr.AtEndOfStream, and the child hasn't written any data yet,
the main app will hang:
while (execObj.Status == 0) {
while (!execObj.Stdout.AtEndOfStream)
WScript.StdOut.Write(execObj.Stdout.ReadAll());
while (!execObj.Stderr.AtEndOfStream)
WScript.StdErr.Write(execObj.Stderr.ReadAll());
WScript.Sleep(100);
}
The following trivial example shows that AtEndOfStream blocks the
main app if no data has been written to the textstream:
child app "bar.vbs", just sleeps for 10 seconds:
-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-
WScript.Sleep(10000)
-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-
main app "foo.vbs", launched with "cscript.exe foo.vbs":
-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-
Set wshShell = WScript.CreateObject("WScript.Shell")
Set execObj = wshShell.Exec("cscript.exe //nologo bar.vbs")
Set stderr = WScript.StdErr
stderr.WriteLine "TypeName execObj=" & TypeName(execObj)
stderr.WriteLine "TypeName execObj.StdErr=" & TypeName(execObj.StdErr)
stderr.WriteLine "execObj.StdErr.AtEndOfStream=" & execObj.StdErr.AtEndOfStream
stderr.WriteLine "TypeName execObj=" & TypeName(execObj)
stderr.WriteLine "TypeName execObj.StdOut=" & TypeName(execObj.StdOut)
stderr.WriteLine "execObj.StdOut.AtEndOfStream=" & execObj.StdOut.AtEndOfStream
-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-X-
Michael Harris (MVP)
> this
> up badly. The problem is that 1) if the child app fills the stdout or
> stderr buffer, it will hang forever; and 2) the AtEndOfStream property
> of stdout and stderr will hang the main app until the child writes
> data
> to the textstream or the child app exits. So the following "solution"
> is unsafe, because the main app can't predict whether the child will
> write to stdout or stderr at all. If it tests
> execObj.Stdout.AtEndOfStream
> or execObj.Stderr.AtEndOfStream, and the child hasn't written any
> data yet, the main app will hang:
Here's a vbscript example of servicing and capturing both stdout and stderr
separately in a way where neither block and hang the script.
(Change the dir to a bogus command name like zdir to get some stderr
captured)...
sCmd = "%comspec% /c dir *.*"
set shell = createobject("wscript.shell")
Set sdStdOut = CreateObject("Scripting.Dictionary")
Set sdStdErr = CreateObject("Scripting.Dictionary")
set wsx = shell.exec(sCmd)
set wsxOut = wsx.stdout
set wsxErr = wsx.stderr
do: wscript.sleep 10
do until wsxOut.atendofstream
sdStdOut(sdStdOut.count) = wsxOut.readline
loop
do until wsxErr.atendofstream
sdStdErr(sdStdErr.count) = wsxErr.readline
loop
loop until wsx.status <> 0 _
and wsxOut.atendofstream _
and wsxErr.atendofstream
arOutLines = sdStdOut.items()
arErrLines = sdStdErr.items()
wscript.echo string(80,"=")
wscript.echo "stdout lines:", sdStdOut.count
wscript.echo string(80,"=")
wscript.echo join(arOutLines,vbcrlf)
wscript.echo string(80,"=")
wscript.echo "stderr lines:", sdStdErr.count
wscript.echo string(80,"=")
wscript.echo join(arErrLines,vbcrlf)
wscript.echo string(80,"=")
--
Michael Harris
Microsoft.MVP.Scripting
Sammamish WA US
John Frensen
> Here's a vbscript example of servicing and capturing both stdout and stderr
> separately in a way where neither block and hang the script.
>
> (Change the dir to a bogus command name like zdir to get some stderr
> captured)...
>
> sCmd = "%comspec% /c dir *.*"
>
Michael --
Your script does not work if the stderr buffer fills before data
is written to stdout. Try your main script with the child script
"producer.js" below, which writes 10k to stderr or stdout.
Your script works fine if the child writes data to stdout. But
if the child writes to stderr, then the call to wsx.stdout.AtEndOfStream
property blocks the main app until the child writes data to stdout
or the child app exits. In the meantime, the child has filled
the stderr buffer and is hung. So both the main and child apps
are hung.
(See my previous post for a trivial demonstration that the
AtEndOfStream property blocks until data is written to the stream.)
Michael Harris (MVP)
> is written to stdout. Try your main script with the child script
> "producer.js" below, which writes 10k to stderr or stdout.
>
> Your script works fine if the child writes data to stdout. But
> if the child writes to stderr, then the call to
> wsx.stdout.AtEndOfStream property blocks the main app until the child
> writes data to stdout
> or the child app exits. In the meantime, the child has filled
> the stderr buffer and is hung. So both the main and child apps
> are hung.
So, you have a real (or contrived) case where an Exec'd console process may
write exclusively to either stderr or stdout but you are unable to predict
which will be written to first, if at all.
In that case the only solution that I know of is to contruct the command
line executed such that stderr output is redirected to and combined with
stdout. That means explicit execution of %compsec% and the 2>&1 (if I
remember the syntax right) redirection. The problem lays in the WSH support
for the stdout/stderr streams being implelemnted as TextStream objects, over
which you have little control.
John Frensen
> So, you have a real (or contrived) case where an Exec'd console
> process may write exclusively to either stderr or stdout but you
> are unable to predict which will be written to first, if at all.
Exactly. In my case, my WSH application was mysteriously hanging,
and I eventually came up with the simple scripts that I posted here
that recreate the problem with just a few lines.
To summarize this, there are basic flaws in the way that WSH handles
stdout and stderr: 1) if the child app fills the stdout or stderr buffer,
it will hang forever; and 2) the AtEndOfStream property of stdout and
stderr will hang the main app until the child writes data to the
textstream or the child app exits.
A program that correctly uses the stdout/stderr model will write
"expected" information to stdout, and only write "error" information
to stderr. The main app can't predict if the child app will write
data to stdout, stderr, both, or neither. It can't safely use the
AtEndOfStream property to check for data, since it will hang if no
data has been written. But if it doesn't use AtEndOfStream and some
flavor of textstream Read() to empty the stdout and stderr buffers,
then the child may hang.
The most serious bug here is in the AtEndOfStream property,
which blocks until some data has been written to the stream.
This is clearly incorrect. A program uses the AtEndOfStream
property before issuing a "read", because it does not want to
block if there is nothing to read. If no data has ever been
written to the stream, it is pretty obvious that there is nothing to
read at the moment. In this case, the AtEndOfStream property must
not block, and it must return a value of "true".
Michael Harris (MVP)
> which blocks until some data has been written to the stream.
> This is clearly incorrect. A program uses the AtEndOfStream
> property before issuing a "read", because it does not want to
> block if there is nothing to read. If no data has ever been
> written to the stream, it is pretty obvious that there is nothing to
> read at the moment. In this case, the AtEndOfStream property must
> not block, and it must return a value of "true".
You won't get any argument from me on how it *should* behave ;-)...
But don't hold your breath. WSH and the related COM based scripting
engines/runtime are in 'sustained engineering' mode meaning only serious
bugs (like security exposures or data corruption issues) are considered for
bug fixes. No future releases are planned.
kmashint
The best way I found to avoid blocking is to redirect StdOut or StdErr
to a file, or if you don't need them separate then you can redirect
StdErr to StdOut using the POSIX standard 2>&1 (redirect handle 2,
stderr, to the address of handle 1, stdout), e.g.
dir /b 2>&1
--
kmashint
------------------------------------------------------------------------
Posted via http://www.codecomments.com
------------------------------------------------------------------------
asdf
this stuff is a mystery for scripting b3eginners, and
advanced and professionals alike, whose time is of the essence.
Why not a sample instead of swaping and absorbing
genetic knowledege bits among a few.
"kmashint" <kmashin...@mail.codecomments.com> wrote in message
news:kmashin...@mail.codecomments.com...
