The Tcl notifier and Console on Windows
Alex
I'm struggling (outside Tcl) to make event-driven I/O work with the
console on Windows.
I known Tcl does it right: [fileevent stdin readable] works.
However when looking at the code in tclWinChan.c and tclWinConsole.c, I
see a fairly intricate setup: namely a ConsoleReaderThread which does
WaitForSingleObject on the console, and then signals an event object,
which is the one waited for by the main notifier thread.
Can a kind soul explain this extra layer ? Why not wait for the console
handle directly from the main notifier thread ?
It turns out that doing it directly in a test program subtly fails: the
first character of every line typed on the console is garbled... So
doing it like Tcl does is a Good Thing. But why ???
Many many thanks in advance for any amount of insight on this !
-Alex
David Gravereaux
> Hi,
>
> I'm struggling (outside Tcl) to make event-driven I/O work with the
> console on Windows.
>
> I known Tcl does it right: [fileevent stdin readable] works.
> However when looking at the code in tclWinChan.c and tclWinConsole.c, I
> see a fairly intricate setup: namely a ConsoleReaderThread which does
> WaitForSingleObject on the console, and then signals an event object,
> which is the one waited for by the main notifier thread.
>
> Can a kind soul explain this extra layer ? Why not wait for the console
> handle directly from the main notifier thread ?
Tcl's channel system is 2 part. First you have the generic layer and
the notifier generic side, then the OS specific side.
Yes, Tcl_WaitForEvent could wait on the console handle directly, but
there isn't a cross-platform facility for such behavior.
Instead, the specifics are internal and when a channel becomes
alertable, the OS specific code of the channel driver calls
Tcl_AlertNotifier. When waken, all channel sources are checked, and if
any have inserted work into the event loop, the work is performed, repeat.
I'm sure that stuff is all clear, but the channel driver for the console
probably looks odd. It does to me, too.
<history_lesson>
Once upon a time, Tcl was ported to windows by Scott Stanton in 1996
while working for the Tcl group at Sun (before Scriptics was formed) and
it was good.
</history_lesson>
Not all handles that windows uses were waitable types at the time
(Win3.11) and this is why the extra two threads are used for reading and
writing to the console handle.
Nowadays, considering Win98 as minimum, I think that method could be
rewritten to use overlapped-IO instead.
> It turns out that doing it directly in a test program subtly fails: the
> first character of every line typed on the console is garbled... So
> doing it like Tcl does is a Good Thing. But why ???
I've used console handles directly in a WaitFor... followed by a
ReadConsoleInputW and can't say I had the same behavior.
Jeff Hobbs
[Win I/O discussion]
> Nowadays, considering Win98 as minimum, I think that method could be
> rewritten to use overlapped-IO instead.
There has been some discussion about making 8.5 ignore any compatibility
with Win9x whatsoever. Maybe it will work, but no effort to guarantee
that will be made. This hasn't been made official, but there hasn't
been much arguing against the idea.
Remember that June 30, 2006 (next month) is the official end of life for
all Win9x-based systems.
Jeff
Alex
David Gravereaux wrote:
>
> Not all handles that windows uses were waitable types at the time
> (Win3.11) and this is why the extra two threads are used for reading and
> writing to the console handle.
>
> Nowadays, considering Win98 as minimum, I think that method could be
> rewritten to use overlapped-IO instead.
Thanks David, it makes more sense with an explanation.
Perhaps the best thing to do would be not to change the implementation,
but just add a comment in the code explaining just that ;-)
> > It turns out that doing it directly in a test program subtly fails: the
> > first character of every line typed on the console is garbled... So
> > doing it like Tcl does is a Good Thing. But why ???
>
> I've used console handles directly in a WaitFor... followed by a
> ReadConsoleInputW and can't say I had the same behavior.
Thanks for your sample. I see you're addressing the console at the
level of "rich events" (including mouse, keyups and downs etc). I was
trying with a generic ReadFile on the console handle instead:
HANDLE hin=GetStdHandle(STD_INPUT_HANDLE);
for(;;) {
WaitForSingleObject(hin,INFINITE);
if (!ReadFile(hin,buf,1,&bytes_read,NULL)) break;
fprintf(stderr,"read %d bytes %c\n",(int) bytes_read,*((char *)
buf));
}
When I compile and run the above, and type:
- nothing happens for normal chars
- when I hit Enter, the input line is flushed (canonical mode)
and the bytes arrive as expected
- BUT the first one is replaced by a 'i' or a 't' (randomly).
- if I pipe something before the program (so that its stdin is
no longer the console), the bytes are all OK.
Funny, isn't it ?
(Sorry for this digression on comp.lang.tcl, but knowing the complete
truth about the dos and donts of Windows could still be interesting for
those who will undertake the "Win9x-free" rewrite...)
-Alex
PS: I also tried opening CONIN$ instead of using GetStdHandle, and I
also tried ReadConsole instead of ReadFile ... to no avail.
David Gravereaux
> HANDLE hin=GetStdHandle(STD_INPUT_HANDLE);
> for(;;) {
> WaitForSingleObject(hin,INFINITE);
> if (!ReadFile(hin,buf,1,&bytes_read,NULL)) break;
> fprintf(stderr,"read %d bytes %c\n",(int) bytes_read,*((char *)
> buf));
> }
>
> When I compile and run the above, and type:
> - nothing happens for normal chars
> - when I hit Enter, the input line is flushed (canonical mode)
> and the bytes arrive as expected
> - BUT the first one is replaced by a 'i' or a 't' (randomly).
> - if I pipe something before the program (so that its stdin is
> no longer the console), the bytes are all OK.
>
> Funny, isn't it ?
Yup, it is funny. Apparently the WaitFor.. is too sensitive and
responds to all events (like mouse, too) and ends up blocking on
ReadFile waiting for just characters instead which seems to be corrupted
by the WaitFor..
Take out the WaitFor.. and the corruption disappears. So in this case,
ReadFile is expected to be used in blocking mode.
For doing overlapped-I/O, you'd use ReadFileEx and either use APC or a
CompletionRoutine to allow the filled buffers to come back. It gets a
bit complicated, but has great performance (or low overhead, whichever
your perspective).
David Gravereaux
I've been teasing myself with taking this on for sometime now... I'd
better just jump up to the plate and do it to stop the voices in my head :)
MH
So, on July 1, 2006 (Canada Day!), all Win98 installations magically
disappear?
Seriously - I just recently install Win98 in a VmWare session. With VmWare
being free, and soon, Xen and others supporting virtualization, I think
there'll be a whole new round of life for '98.
btw, does anyone know if they'll "cut off" Windows Update for Win98, or just
stop fixing it?
I have older games I want to run, etc. And I *HAVE* legit Win98 licenses.
And that's all I need to watch MLB TV, etc...
MH
Alex
David Gravereaux wrote:
>
> Yup, it is funny. Apparently the WaitFor.. is too sensitive and
> responds to all events (like mouse, too) and ends up blocking on
> ReadFile waiting for just characters instead which seems to be corrupted
> by the WaitFor..
> Take out the WaitFor.. and the corruption disappears. So in this case,
> ReadFile is expected to be used in blocking mode.
Wow. I'm having a hard time silencing my anti-Windows rant here...
Do they call THAT an OS ? ^H^H^H^H^H^H^H^H^H^H sorry :-)
> For doing overlapped-I/O, you'd use ReadFileEx and either use APC or a
> CompletionRoutine to allow the filled buffers to come back. It gets a
> bit complicated, but has great performance (or low overhead, whichever
> your perspective).
Imagine we want to reimplement [vwait] with no extra threads, and we
have a fileevent readable on stdin. So the thing to wait for is
GetStdHandle(STD_INPUT_HANDLE), or an event object bound to it through
an OVERLAPPED structure.
But how do we alter this existing handle to be in overlapped mode,
since it is already opened (it may be a pipe, file, or console), in
non-overlapped mode ???
-Alex
David Gravereaux
> David Gravereaux wrote:
>> Yup, it is funny. Apparently the WaitFor.. is too sensitive and
>> responds to all events (like mouse, too) and ends up blocking on
>> ReadFile waiting for just characters instead which seems to be corrupted
>> by the WaitFor..
>> Take out the WaitFor.. and the corruption disappears. So in this case,
>> ReadFile is expected to be used in blocking mode.
>
> Wow. I'm having a hard time silencing my anti-Windows rant here...
> Do they call THAT an OS ? ^H^H^H^H^H^H^H^H^H^H sorry :-)
So let me guess... So when you are waiting on a handle that can be
triggered for numerous types of events and you're just looking for a
subset of those types with no way of setting a mask (like
WSAEventSelect), its the fault of the OS?
>> For doing overlapped-I/O, you'd use ReadFileEx and either use APC or a
>> CompletionRoutine to allow the filled buffers to come back. It gets a
>> bit complicated, but has great performance (or low overhead, whichever
>> your perspective).
>
> Imagine we want to reimplement [vwait] with no extra threads, and we
> have a fileevent readable on stdin. So the thing to wait for is
> GetStdHandle(STD_INPUT_HANDLE), or an event object bound to it through
> an OVERLAPPED structure.
Well, no actually. All the generic aspects of the notifier/event loop
are just fine the way they are. The part that could change is only
tclWinConsole.c
Yes, you could event trigger on an overlapped ReadFileEx coming back or
instead have a single thread dedicated to servicing a completion port.
I prefer the later myself.
The notifier (the main thread) itself does wait in an APC compliant
manner and *could* be used to service an APC completion routine, but
there could be a starvation issue when you need to emulate blocking in
the Tcl_DriverOutputProc. I guess as long as the block is APC happy, it
could work _without_ the use of additional service threads. APC
compliant waits are functions such as WaitForMultipleObjectsEx that set
bAlertable to TRUE.
Hmm.. Now you got me thinking about the details :)
> But how do we alter this existing handle to be in overlapped mode,
> since it is already opened (it may be a pipe, file, or console), in
> non-overlapped mode ???
Change the opening code in tclWinChan.c:TclpOpenFileChannel prior to
creating the channel for it with TclWinOpenConsoleChannel(). There is
an example of that already for serial channels.
All that's really important is that when the channel becomes serviceable
and we have an async path for it to let us know, that we call
Tcl_AlertNotifier so the notifier wait ends and the open channel list is
queried for new jobs. Oh.. and blocking emulation is working, too.
So in summary, I think ReadFileEx using a completion routine operated in
overlapped mode would be an excellent way to do asynchronous/alertable
reads from a console for just characters.
David Gravereaux
> So in summary, I think ReadFileEx using a completion routine operated in
> overlapped mode would be an excellent way to do asynchronous/alertable
> reads from a console for just characters.
Wait a sec.. I've been through this before...
On NT systems, completion ports with a service thread for the port where
each channel type has it's own port and just one service thread. I've
had great success with this before.
On 9x systems, event notification is probably the best fallback. Note
the 64 handle limit per waiting service thread.
I've been down this road before.. APC (completion routines) are just
too strange for debugging purposes. I personally never found my magic
using Asychronous Procedure Call and CompletionRoutines.
Of course.. Maybe there is good magic to be found using APC..
Alex
> > since it is already opened (it may be a pipe, file, or console), in
> > non-overlapped mode ???
>
> Change the opening code in tclWinChan.c:TclpOpenFileChannel prior to
> creating the channel for it with TclWinOpenConsoleChannel(). There is
> an example of that already for serial channels.
Again, I'm generalizing a bit: I'm not restricting myself to the
console case.
And also I'm not in a position to *open* the handle, it is given to me
as hStdin by the CreateProcess from which I was born. In this specific
case, is there a way to do overlapped io on this handle (or one of its
clones) ?
(I know it's not the way to go for the Tcl notifier, but I'm still
interested in the answer because I have a situation where I cannot
afford an extra thread)
-Alex
David Gravereaux
Try it! Try calling ReadFileEx on it with a CompletionRoutine and see
what it does. You might want to go with APC so don't need a service
thread. Use WaitForMultipleObjectsEx and set bAlertable to TRUE.
If I had the time right now, I'd code up an example for you, but I got
stuff to do.. Maybe I'll try it later tonight.
Alex
> > as hStdin by the CreateProcess from which I was born. In this specific
> > case, is there a way to do overlapped io on this handle (or one of its
> > clones) ?
>
> what it does.
The documentation for ReadFileEx explicitly states:
"This file handle must have been created with the
FILE_FLAG_OVERLAPPED"
while I'm inheriting a non-overlapped hStdin. How can I get out of this
trap ?
-Alex
David Gravereaux
Ack! ReadFileEx keeps complaining.. CreateFile help states that
dwFlagsAndAttributes is ignored for consoles, so I guess that means
FILE_FLAG_OVERLAPPED can't be set even if we reopened..
You might want to use the "rich events" with ReadConsoleInput() and pop
for KEY_EVENT with a keydown and discard the rest (?)
This is sucky.
Jeff Hobbs
> In article <4470F27...@activestate.com>,
> Jeff Hobbs <je...@activestate.com> wrote:
>> David Gravereaux wrote:
>> [Win I/O discussion]
>>> Nowadays, considering Win98 as minimum, I think that method could be
>>> rewritten to use overlapped-IO instead.
>> There has been some discussion about making 8.5 ignore any compatibility
>> with Win9x whatsoever. Maybe it will work, but no effort to guarantee
>> that will be made. This hasn't been made official, but there hasn't
>> been much arguing against the idea.
>>
>> Remember that June 30, 2006 (next month) is the official end of life for
>> all Win9x-based systems.
>
> disappear?
>
> Seriously - I just recently install Win98 in a VmWare session. With VmWare
> being free, and soon, Xen and others supporting virtualization, I think
> there'll be a whole new round of life for '98.
>
> btw, does anyone know if they'll "cut off" Windows Update for Win98, or just
> stop fixing it?
>
> I have older games I want to run, etc. And I *HAVE* legit Win98 licenses.
> And that's all I need to watch MLB TV, etc...
And for you there will always be Tcl/Tk 8.4.
I also have a Win98 Official CD, but I haven't run it in years. We only
have it at work for testing purposes, and hope to soon retire even those
VM installs. IIUC, there will be no more updates for Win98 systems, for
any severity of issue.
Jeff
Alex
>
> Ack! ReadFileEx keeps complaining.. CreateFile help states that
> dwFlagsAndAttributes is ignored for consoles, so I guess that means
> FILE_FLAG_OVERLAPPED can't be set even if we reopened..
> You might want to use the "rich events" with ReadConsoleInput() and pop
> for KEY_EVENT with a keydown and discard the rest (?)
Yes, this would take care of the console case.
But if stdin is a pipe, we're stuck, right ?
> This is sucky.
Indeed. But not anybody is allowed to say so :-}
-Alex
MH
That's understandable. And, using VMWARE, I can just "revert" if I get hit
by something (unlikely - I'm NOT using Win98 for it's amazing networking!).
I'm just wondering if they'll keep "Windows Update" on for Win98 for a few
years, so I can still install it and get SOME patches?
For instance, I just bought Omar Sharif Bridge for $4. I believe it runs
in Win98. Why would I clutter up my Linux session with a Win2K instance?
Gotta be heavier than Win98..
Then again, I rarely run Tcl under Win98 - it's usually under Linux.
Just don't go adding Vista-only stuff for the next 10 years.. :-)
MH
David Gravereaux
> David Gravereaux wrote:
>> Ack! ReadFileEx keeps complaining.. CreateFile help states that
>> dwFlagsAndAttributes is ignored for consoles, so I guess that means
>> FILE_FLAG_OVERLAPPED can't be set even if we reopened..
>> You might want to use the "rich events" with ReadConsoleInput() and pop
>> for KEY_EVENT with a keydown and discard the rest (?)
>
> Yes, this would take care of the console case.
> But if stdin is a pipe, we're stuck, right ?
If you must have the same code manage both types, yes you're stuck. Or
detect the difference with GetConsoleMode(). If the function fails, the
handle is for a pipe.
>> This is sucky.
>
> Indeed. But not anybody is allowed to say so :-}
haha. I'm saying the situation and you think I mean the OS.
Still there's a few more ways to do I/O. The APC one seems to be a NOP.
And IOCP won't work either as it looks like consoles can't use the
overlapped/async methods. I know you mentioned you don't want to use
threads, but the synchronous ReadFile() in a worker thread does work and
for either handle type with the same code, too. And when ReadFile()
does return, just SetEvent() for an event you'll be waiting on in the
main thread. Yes, it won't be the console handle itself, and yes you'll
need a worker thread.
Or there's waiting on the handle itself and using ReadConsoleInput and
discarding non-keypresses, which doesn't need a worker thread.
The following test code I played with sure looks right to me for doing
APC with a CompletionRoutine, but:
1) doesn't work on a console handle from apparently its lack of
ability to do overlapped.
2) pipes should work, but don't recurse. Only one read operation
completes. The second does get posted, but doesn't come back. I know
not why.
If this was a socket, I'd be pulling bytes in a furry.
David Gravereaux
member of the OVERLAPPED structure and initiating an overlapped ReadFile
operation.
It works with pipes just fine.. doesn't appear to want to work with a
console handle, though.
> dir /? |testcon3
^- was a good test case
David Gravereaux
synchronous ReadFile/ReadConsole calls in worker threads. It idles in
an alertable state with WaitForMultipleObjectsEx. async is done with
consoles using the console handle itself as the alert with async reads
performed by ReadConsoleInput and discarding non-keypress events. Pipes
use overlapped ReadFile calls with event notification.
Alex
However, there's still one thing I can't grasp:
You seem to be saying the pipe case is solved with overlapped io (which
means waiting on the handle of the event object in the OVERLAPPED
structure, I know).
However, I'm still stuck with a non-overlapped pipe passed as stdin !
In your code you mention ReOpenFile, which solves the problem for
WS2003. But on XP, do you confirmed I'm doomed ?
To summarise, the following situation is not implementable in windows
XP:
- user types "a.exe | b.exe" in a cmd.exe session
- b.exe is a single-threaded application that does a parallel wait on
its standard input (the pipe from, a.exe) and some other event source
(like a WaveIn event object).
Do you agree ?
-Alex
David Gravereaux
wrote:
>Do you agree ?
It works for me on XP. Anonymous pipes are really two nameless named
pipes already connected on the inside, which are overlap capable
(unless you're on 98).
Whether the pipe was actually created with FLAG_FILE_OVERLAPPED on the
parent side prior to being inherited, I can't say. How the kernel I/O
manager behaves across processes, I can't say either. I just know I
saw it working. MSDN docs have too many mistakes to be taken too word
for word in my experience. I believe what I try.
PS. M$ Windows may or may not suck, but given a large enough hammer it
can be pounded into submission.
Michael A. Cleverly
> PS. M$ Windows may or may not suck, but given a large enough hammer it
> can be pounded into submission.
QOTW?
David Gravereaux
wrote:
>However, I'm still stuck with a non-overlapped pipe passed as stdin !
>In your code you mention ReOpenFile, which solves the problem for
>WS2003. But on XP, do you confirmed I'm doomed ?
I think we might be doomed async operation on an anonymous pipe:
"
Asynchronous (overlapped) read and write operations are not supported
by anonymous pipes. This means that you cannot use the ReadFileEx and
WriteFileEx functions with anonymous pipes. In addition, the
lpOverlapped parameter of ReadFile and WriteFile is ignored when these
functions are used with anonymous pipes.
"
I missed this test case:
> netstat -p udp -n 1 |testcon4
The simple dump pass to the next process was working, but it doesn't
alert when readable.. hmm..
I need a bigger hammer, or more nails to choose from.
I guess if overlapped just won't work (even when I saw it working for
dump on start), then you'll have to do synchrous read and writes with
a worker thread.. Oh well.
Uwe Klein
> PS. M$ Windows may or may not suck, but given a large enough hammer it
> can be pounded into submission.
Yeah, right. But doesn't the applied trauma force it into disability
and care for life?
uwe
Alex
> alert when readable.. hmm..
Interesting: in my case it wakes up immediately, though there is
nothing yet on the pipe.
> I need a bigger hammer, or more nails to choose from.
> I guess if overlapped just won't work (even when I saw it working for
> dump on start), then you'll have to do synchrous read and writes with
> a worker thread.. Oh well.
At least this is a clear statement.
For those who still think that the "situation sucks, not the OS", here
are a few relaxing unix lines of code, doing exactly the thing we've
just discovered Windows XP cannot do:
/* 0 can be pipe/socket/pty/file/whatever */
FD_SET(0,&rs);
audio=open("/dev/dsp",O_RDONLY);
FD_SET(audio,&rs);
select(audio+1,&rs,NULL,NULL,NULL,NULL);
Happy hammering to y'all :-)
-Alex
David Gravereaux
Here's a bug report that makes me laugh (but in a sad way):
https://sourceforge.net/tracker/index.php?func=detail&aid=819667&group_id=10894&atid=310894
If KeySpan wrote their USB driver correctly, it would be reported it
as a serial device.
Its a sad day when Tcl gets patches for buggy third-party stuff.
David Gravereaux
wrote:
>Happy hammering to y'all :-)
Don't be a weenie.. Just put some code around the synchronous
ReadFile and be done with it.
Uwe Klein
> On Fri, 26 May 2006 09:35:05 +0200, Uwe Klein
> <uwe_klein_...@t-online.de> wrote:
>
>
>>David Gravereaux wrote:
>>
>>
>>>PS. M$ Windows may or may not suck, but given a large enough hammer it
>>>can be pounded into submission.
>>
>>Yeah, right. But doesn't the applied trauma force it into disability
>>and care for life?
>>
>>
>>uwe
>
>
> Here's a bug report that makes me laugh (but in a sad way):
> https://sourceforge.net/tracker/index.php?func=detail&aid=819667&group_id=10894&atid=310894
But it is not as sad as it could be.
KeySpan are willing to fix their driver!
The thing i do not understand here is
: why has nobody else noticed and requested this fixed ?
>
> If KeySpan wrote their USB driver correctly, it would be reported it
> as a serial device.
>
> Its a sad day when Tcl gets patches for buggy third-party stuff.
I've written/build lots of stuff that integrated some device or other.
And all had their little snitches that where squalling for attention.
Start with the Atari ST expansion slot that had an unpleasant glitch
in the busrequest handshaking..
the Signetics 68070 had interrupt status bits in a low byte but you
had to write the highbyte to clear them ( a bug not a feature )
uwe
Alex
> Don't be a weenie.. Just put some code around the synchronous
> ReadFile and be done with it.
Oh yeah, of course. Just like Thomas Edison's buddies saying:
"Thomas, don't be a weenie. Just light up a candle and be done with
it."
David Gravereaux
wrote:
Q: What do you call two arguing Frenchmen?
A: Entertainment.
Here's the pipe reader code. Don't call any libc routines from the reader
thread when/if you modify it.
PS. For some reason pipe handles on NT really are overlapable.. try this:
netstat -p tcp -n 1 |testcon4
netstat -p tcp -n 1 |testpipe1
They both work just fine. Now go fly solo.. I'm done here.
David Gravereaux
wrote:
>PS. For some reason pipe handles on NT really are overlapable..
Umm.. correction, no they aren't overlapable after I looked more deeply..
The code from testcon4.c happens to block on what should have been an
overlapped call instead of blocking in the WaitFor..
So it's just like what MSDN says. anon pipes just can't do asynchnous.
Alex
> On Sun, 28 May 2006 19:34:03 -0700, David Gravereaux <davy...@pobox.com>
> wrote:
>
> >PS. For some reason pipe handles on NT really are overlapable..
>
> Umm.. correction, no they aren't overlapable after I looked more deeply..
> The code from testcon4.c happens to block on what should have been an
> overlapped call instead of blocking in the WaitFor..
Yes, that's what I meant by "in my case it wakes up immediately, though
there is nothing yet on the pipe", a few messages up this thread :-)
Oh, by the way, thanks for the very concise and honest account you gave
of all this on comp.os.ms-windows.programmer.win32 under the title
"Why can't MS get I/O operations consistent over different handle
types?"