using wshell.exec causes lockup while reading the output - what to do ?

[R.Wieser]

Hello all

I'm running a console-based windows program, and need to capture its output.
When I try to use the "exec" method than, nomatter what I try, something
locks up.

My code is basically this:

Set oExec = WshShell.Exec(sCmd)
Do While oExec.Status = 0
WScript.Sleep 100
wscript.echo oExec.stdout.readall
Loop

As the program also outputs on stderr I've changed the above reading of
stdout with

if not oExec.stdout.atendofstream then wscript.echo oExec.stdout.read(1000)
if not oExec.stderr.atendofstream then wscript.echo oExec.stderr.read(1000)

But no matter what I've tried (small buffers, large buffers), at some point
in the output it locks up. :-(

Apart from switching to the "run" method*, what can I do ?

*I've got the "run" method working. But the output capturing needs the use
of "cmd /e ", which in turn causes my screen to go black a few times
(mutiple calls) with no (progress) output on it. Not nice.

Regards,
Rudy Wieser

[Mayayana]

"R.Wieser" <add...@not.available> wrote

| I'm running a console-based windows program, and need to capture its
output.
| When I try to use the "exec" method than, nomatter what I try, something
| locks up.
|
| My code is basically this:
|
| Set oExec = WshShell.Exec(sCmd)
| Do While oExec.Status = 0
| WScript.Sleep 100
| wscript.echo oExec.stdout.readall
| Loop
|
| As the program also outputs on stderr I've changed the above reading of
| stdout with
|
| if not oExec.stdout.atendofstream then wscript.echo
oExec.stdout.read(1000)
| if not oExec.stderr.atendofstream then wscript.echo
oExec.stderr.read(1000)
|
| But no matter what I've tried (small buffers, large buffers), at some
point
| in the output it locks up. :-(

I'm not sure I have an answer, but this is a function
I've also had trouble with. I wrote an HTA to wrap
youtube-dl and would like to get it to print an update
to the HTA window as it progresses. But no matter what
I do, I can't get it to update the HTA UI.

However, I've never seen it lock up. For what it's worth,
this is my current code:

'......
Set ObjExec = SH.Exec(s1)
s1 = ""
While (ObjExec.Status = 0)
SleepEx 100
s1 = s1 & ObjExec.StdOut.ReadAll
s2 = s2 & ObjExec.StdErr.ReadAll
TReport.value = s1 & vbCrLf & s2
DoEvents
Wend

s1 = s1 & ObjExec.StdOut.ReadAll
s2 = s2 & ObjExec.StdErr.ReadAll
TReport.value = s1 & vbCrLf & s2

Set ObjExec = Nothing

'......

Sub SleepEx(Lms)
Dim Ret
Ret = SH2.Run("sleeper.vbs " & Lms, , True)
End Sub

Sub DoEvents()
SH.run "%comspec% /c exit", 0, True
End Sub

'....

Sleeper.vbs is just this:

Dim Arg
on error resume next
Arg = WScript.Arguments(0)
wscript.sleep Arg

SleepEx and DoEvents were added to try to make
the HTA update during the process, but it doesn't
work. Only after the whole thing is finished does
the command window disappear and the total output
text appears in the window. Would such pauses help
you? I don't know. Maybe it depends on the executable?

I have seen a lockup occasionally with youtube-dl.
I've assumed it's a problem with the download itself.
Maybe it got interrupted and youtube-dl couldn't
continue. I don't know. When I see that downloading
has stopped I just run the command again and it picks
up where it left off. But that problem is intermittent.

[JJ]

When any handle is not a file, TextStream object can't really detect the
presence of data in that handle. Thus, AtEndOfStream() or AtEndOfLine()
won't return unless there's data in the handle. This also applies to all
Read and Skip methods, because they seem to use the same function to detect
presence of data in the handle.

[R.Wieser]

JJ,

> When any handle is not a file, TextStream object can't really detect
> the presence of data in that handle. Thus, AtEndOfStream() or
> AtEndOfLine() won't return unless there's data in the handle.

That should not be a problem. As long as the program that is exec-ed runs
it should be generating output (in a seperate thread I presume), and the
buffer should eventually gain some new contents. But no, I can see the
output stops somewhere rather early, nowhere near to the end.

Regardless, do you have any suggestions to how to solve it/what to try ?
Or should I conclude that trying to grab output from an "exec" method is
undependable, and thus simply a waste of time ?

Regards,
Rudy Wieser

[JJ]

On Wed, 17 Jun 2020 18:51:41 +0200, R.Wieser wrote:
>
> That should not be a problem. As long as the program that is exec-ed runs
> it should be generating output (in a seperate thread I presume), and the
> buffer should eventually gain some new contents. But no, I can see the
> output stops somewhere rather early, nowhere near to the end.

It doesn't matter whether the program output/error is written by multithread
or not, because the problem lies within the TextStream object's methods.

> Regardless, do you have any suggestions to how to solve it/what to try ?

When the input is a character device, instead of a file, don't use counted
read operation unless you know exactly when the program stops writing data
into a standard handle. That would produce a deadlock unless the program
generate at least the same length of data being requested to read.

In your code, you have these lines:

if not oExec.stdout.atendofstream then wscript.echo oExec.stdout.read(1000)
if not oExec.stderr.atendofstream then wscript.echo oExec.stderr.read(1000)

If the program writes some lines totalling 500 bytes into standard output,
the script will be stuck within `oExec.stdout.read(1000)` because it's
expecting 500 more bytes of data.

It'be better to read the program output line by line like below.

do while not oExec.stdout.atendofstream
wscript.echo oExec.stdout.readline
loop

> Or should I conclude that trying to grab output from an "exec" method is
> undependable, and thus simply a waste of time ?

What's wrong with that? I use it myself to parse various programs' output,
as well as input commands into the executed programs.

[R.Wieser]

JJ,

> If the program writes some lines totalling 500 bytes into standard output,
> the script will be stuck within `oExec.stdout.read(1000)` because it's
> expecting 500 more bytes of data.
>
> It'be better to read the program output line by line like below.
>
> do while not oExec.stdout.atendofstream
> wscript.echo oExec.stdout.readline
> loop

Well, that doesn't quite work either: While the programs stdout does (in
this case!) emit lines, the stderr emits a long stream which never
terminates. Trying to read a line there directly blocks the whole thing.
And yes, I did precede it with an "atendofstream" check. :-)

>> Or should I conclude that trying to grab output from an "exec" method is
>> undependable, and thus simply a waste of time ?
>
> What's wrong with that?

Whut ? You're not serious, are you ?

Regards,
Rudy Wieser

[JJ]

On Thu, 18 Jun 2020 17:26:48 +0200, R.Wieser wrote:
>
> ...the stderr emits a long stream which never terminates.

That's a very important information which should have been mentioned in the
initial post. If a stream never terminates - assuming that it never emit an
EOL (CRLF/LF), ReadLine() will never return. So counted read is the only way
to read the stream.

However, you will need to know when the stream data ends. You need to make
sure the code is not reading more bytes than what's available. Otherwise,
it'll get stuck.

If the end of stream data is unknown and is unterminated, then I suggest
redirecting the standard error to a file instead, using CMD. Exec() and
TextStream in this case, can't do the job. AtEndOfStream() will be useless
because it will be stuck if a stream is not a file and has no data.

>>> Or should I conclude that trying to grab output from an "exec" method is
>>> undependable, and thus simply a waste of time ?
>>
>> What's wrong with that?
>
> Whut ? You're not serious, are you ?

Sorry. That wasn't said correctly. What I meant is that I use TextStream
too. It's undependable, yes sometimes. But it's not a waste of time, because
it's all we got in WSH. That being said, it can be a waste of time,
depending of the situation due to TextStream limitations.

[R.Wieser]

JJ,

>> ...the stderr emits a long stream which never terminates.
>
> That's a very important information which should have been
> mentioned in the initial post.

If I would have known it was important I probably would have.

> So counted read is the only way to read the stream.
>
> However, you will need to know when the stream data ends.
> You need to make sure the code is not reading more bytes
> than what's available.

Sounds resonable. So, any idea how I can ask a stream how many
bytes/characters it has currently buffered ? :-)

> If the end of stream data is unknown and is unterminated,
> then I suggest redirecting the standard error to a file instead,
> using CMD.

I already got that to work, but it has got its own drawbacks. Hence my
attempt to use "exec".

Regards,
Rudy Wieser

P.s.
Besides having to use output chunked into lines, according to the below link
those lines may not be longer than the internal (pipe)buffer size - which is
just a measily 4K (no idea if that is in bytes or chars).

https://groups.google.com/d/topic/microsoft.public.scripting.wsh/kIvQsqxSkSk

[JJ]

On Fri, 19 Jun 2020 19:08:39 +0200, R.Wieser wrote:
>
> Sounds resonable. So, any idea how I can ask a stream how many
> bytes/characters it has currently buffered ? :-)

That's the main problem I've been trying to tell you. TextStream simply
isn't capable enough when the stream is a standard handle which is a
character device. It at least need a help from an outside tool/COM.

> I already got that to work, but it has got its own drawbacks. Hence my
> attempt to use "exec".

I'm guessing that you also need to inspect the outputted data right away,
instead of waiting until the program has terminated?

> P.s.
> Besides having to use output chunked into lines, according to the below link
> those lines may not be longer than the internal (pipe)buffer size - which is
> just a measily 4K (no idea if that is in bytes or chars).
>
> https://groups.google.com/d/topic/microsoft.public.scripting.wsh/kIvQsqxSkSk

There's that limit, yes. But it's not what causing TextStream to get stuck.

[R.Wieser]

JJ,

> That's the main problem I've been trying to tell you. TextStream
> simply isn't capable enough when the stream is a standard handle
> which is a character device

So, I /can't/ check if its empty, I /can't/ ask how full it is, and its
"read data" methods are all blocking ones.

The only way "exec" with output capturing can work is either with having
just a single stream (on which the "readall" method would probably work), or
when having two streams that /both/ emit lines of data (so that the
"readline" method can be used), with both lines being shorter than a buffers
worth.

That strongly limits the "exec" methods use cases.

Wait ... Nope, having both emit lines will still cause deadlocks. Just
imagine one stream getting shorter lines than the other. As you can't empty
a stream (trying to do so will cause a blocking situation) and you can't
skip reading a stream when it doesn't have any data (checking for that also
blocks), the buffer getting the longer lines will eventually fill up and
block.

I really start to hate MS. :-(

> I'm guessing that you also need to inspect the outputted data
> right away, instead of waiting until the program has terminated?

The stderr emits a progress counter for the screen (output count and a CR,
the new value overwriting the last), stdout is supposed to be written to a
file.

> There's that limit, yes. But it's not what causing TextStream to
> get stuck.

I think I understood the bit about blocking reads. The buffersize problem
goes, when using the "readline" method, ontop of that. Or did I mis
something ?

Regards,
Rudy Wieser

[JJ]

On Sat, 20 Jun 2020 18:56:24 +0200, R.Wieser wrote:
>
> I really start to hate MS. :-(

Can't live with it, and but can't throw it away either, right?

> The stderr emits a progress counter for the screen (output count and a CR,
> the new value overwriting the last), stdout is supposed to be written to a
> file.

If you need to parse the counter, you can use counted read one character at
a time until CR is recevied, then parse the collected counter characters.
Repeat that to process the next counter. That should take care of the
counter output processing without getting a deadlock.

Now the problem is knowing when the couter stops being updated. So, what
data is emitted to the standard error the program has completed its
processing? i.e. the data emitted after the last counter. Whatever that is,
it should be treated as a signal to stop the loop for processing the counter
output. If no more data is emitted to the standard error and the programs
simply ends, then there's no need to ready anything any more. Just exit the
loop.

> I think I understood the bit about blocking reads. The buffersize problem
> goes, when using the "readline" method, ontop of that. Or did I mis
> something ?

Buffer size doesn't matter much. ReadLine() can still work properly without
getting stuck when the source's line (with EOL) has a length which is larger
than the buffer size. e.g.

set fs = createobject("scripting.filesystemobject")
set f = fs.createtextfile("temp.txt", true)
f.writeline string(10000, "A")
f.close
set ws = createobject("wscript.shell")
set xc = ws.exec("cmd.exe /c type temp.txt")
wscript.echo xc.stdout.readline

If AtEndOfLine()/ReadLine()/SkipLine() is stuck, it means that either the
source's data has no EOL, or has no data at all.

[R.Wieser]

JJ,

> If you need to parse the counter, you can use counted read
> one character at a time until CR is recevied, then parse the
> collected counter characters.

Yes, that could work as long as I have a single stream. Than again, in that
case I could just use blocking reads of just 10 chars or so (so the output
of it to screen updates regulary) and be done with it.

The whole problem is that I have /two/ streams I need to read. With, in
this case, the stdout not outputting in the same frequency as stderr.

IOW: I can wait for and than read exactly one counter-outputs worth of
characters, but than what? There is nothing I can do with the other stream
that won't block if there is no(t enough) output available yet. And that
ofcourse brings us back to square one.

> Buffer size doesn't matter much. ReadLine() can still work
> properly without getting stuck when the source's line (with
> EOL) has a length which is larger than the buffer size. e.g.

Argh. Yes, ofcourse. Part of it can already be loaded into the string,
freeing up the stream buffer to receive more of the line.

Regards,
Rudy Wieser

[JJ]

On Sun, 21 Jun 2020 08:53:00 +0200, R.Wieser wrote:
>
> The whole problem is that I have /two/ streams I need to read. With, in
> this case, the stdout not outputting in the same frequency as stderr.
>
> IOW: I can wait for and than read exactly one counter-outputs worth of
> characters, but than what? There is nothing I can do with the other stream
> that won't block if there is no(t enough) output available yet. And that
> ofcourse brings us back to square one.

I've been in that shoes before. i.e. two character-typed streams need to be
read, but which one is emitted first and their data length, are unknown and
can not be detected or predicted. Reading any of the stream may lead to a
wrong choice and resulting a deadlock. My conclusion was that, it's an
impossible job for TextStream. I ended up creating a custom native program
to handle the program outputs.

[R.Wieser]

JJ,

> My conclusion was that, it's an impossible job
> for TextStream.

To be honest, I could not care less what method "exec" would use to buffer
the data. :-) As long as it allows me do stupidly basic things like a "is
the buffer empty", or the even better "how much data is available for
reading" check. And perhaps a "read as much as you can, but return
immediatily" method.

>I ended up creating a custom native program
> to handle the program outputs.

Yeah, I was already thinking of that too*, but first wanted to make sure I
had not overlooked some dumb thing.

*and wrap it into an ActiveX component.

Thanks for the responses.

Regards,
Rudy Wieser