EOF from Stdin on Windows

[mattn]

Hi.

It seems that windows console can't handle ^D that mean EOF on windows.

Is this expected?

[Rob Pike]

Isn't ^Z EOF on WIndows?

-rob

[mattn]

Ah, sorry I had typo in e-mail. But ^Z doesn't return io.EOF.

[Andrew Gerrand]

What if you pipe a file into a go program? Does it get EOF then?

--
 
---
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[mattn]

When typed ^Z on follow's code, Read is stopped and err become nil. err shouldn't be nil?

package main

import (

"os"

)

func main() {

var b [10]byte

n, err := os.Stdin.Read(b[:])

println(n, err)

[Andrew Gerrand]

Can you try

echo foo > bar
go run main.go < bar

Or just

echo foo | go run main.go

Where main.go is your example programme.

[mattn]

I got

```

6 (0x0,0x0)

```

xxd bar is:

0000000: 666f 6f20 0d0a

[brainman]

What are you trying to do?

Alex

[Rob Pike]

That's what happens on Unix too. You get a zero-byte errorless read.
If you keep reading, though, you'll see EOF.

-rob

[Rob Pike]

s/zero/n/ if there are n bytes. On the terminal, though, you'll see 0, EOF.

Readers are not required to signal EOF when they hit the end, they are
only required to do so AFTER the end of the input.

-rob

[mattn]

Then, how can I get the reader is at EOF or not?

[Rob Pike]

Look for a zero-length read.

-rob

[mattn]

I see. I's useful to do long poll. :)

Thanks.

[mattn]

Hmm, how should this work? This get ^Z as a character, not stop to read.

package main

import (

"io"

"os"

)

func main() {

n, err := io.Copy(os.Stdout, os.Stdin)

println(n, err)

[brainman]

On Tuesday, 27 August 2013 14:07:10 UTC+10, mattn wrote:
> ... This get ^Z as a character, not stop to read.

Why do you think it should work any different? Do you have any links to substantiate your assumption?

Aex

[mattn]

Most of console application on windows which use msvcrt runtime libraries stop reading with CTRL-Z. For example, feof set EOF to the file handle.

while(feof(f)) {

[brainman]

On Tuesday, 27 August 2013 14:42:00 UTC+10, mattn wrote:

Most of console application on windows which use msvcrt runtime libraries stop reading with CTRL-Z. ...

We use ReadConsole API, whatever it does. We use ReadConsole because it allows non-ascii characters to be entered.

Alex

[mattn]

I'm not talking about replace of ReadConsole. I hope to keep compatibilities for unix like (but it's ^Z not ^D) console application. :)

How about this?

https://codereview.appspot.com/13265043

[Rob Pike]

I'm not sure that's quite right. Shouldn't it consume the Ctl-Z when
you see EOF? Otherwise you'll get EOF forever.

Of course, Windows might behave that way, but I don't think so. Unix
doesn't, not that thats relevant.

-rob

[brainman]

On Tuesday, 27 August 2013 15:10:15 UTC+10, mattn wrote:

How about this?

https://codereview.appspot.com/13265043

I am not sure. Convince me we should do it.

Alex

[brainman]

What is not quite right? Your message is out of context.

Alex

[Rob Pike]

The code is wrong, I think.

[brainman]

On Tuesday, 27 August 2013 15:55:14 UTC+10, Rob Pike wrote:

The code is wrong, I think. 

Oh, yeh. That is possible. We don't have any tests to test this code, because the code only runs on *real* console (when you type into it, not redirect from another program). So I would like to have a good reason before we start fiddling with it.

Alex 

[mattn]

I want your agreements that we'll change current behavior. I you are ok, I'll add tests that using AllocConsole and WriteConsoleInput.

[brainman]

On Tuesday, 27 August 2013 16:05:30 UTC+10, mattn wrote:

I want your agreements that we'll change current behavior. ...

Why should we do it?

Alex

[honey...@gmail.com]

As far as I understand, when ReadFile() of Windows receives ^Z at BOL, it returns as zero bytes read. This is the expected/desired behavior of signaling EOF.

However, ReadConsole() doesn't handle ^Z specially; It's just treated as a ordinary character. Therefore, when you  enter ^Z at BOL followed by a newline, ReadConsole() returns as 3 bytes has been read (that is, ^Z + CR + LF). This cannot be seen as EOF at all.

If Golang doen't treat this, there's no way to signal EOF from interactive user input to Console.

Note that ReadFile() treats ^Z specially only at BOL. For instance, if you enter foo^Z, you will  get foo^Z.

It's not the same as EOF char of Unix tty anyway, that just has an effect of immediately sending queued inputs to process that has already bean read.

When windows console is working on ordinary cooked mode, just entering ^Z doesn't break reading. To send inputs to process, user still has to enter newline after ^Z anyway.

As far as I understand, ReadFile() specially treats ^Z after receiving inputs from Console.

[mattn]

For example, If I want to write cat program using io.Copy, we must handle keyboard event to handle EOF. Currently, readConsole doesn't return error even though I type any keys. Most of windows console application handle CTRL-Z as sending EOF.

package main

import (

"io"

"os"

)

func main() {

io.Copy(os.Stdout, os.Stdin)

[Dave Cheney]

So windows terminal does not actually close the input stream when you
use ^Z, but sends the 0x1a char down the stream and the application at
the other end of it interprets it as EOF ? Surely this can't be true.

[mattn]

I updated patch.

https://codereview.appspot.com/13265043/patch/14001/15001

Just only for windows console, we can't read CTRL-Z. CTRL-Z is handled as EOF.

When os.Stdin is a file redirected, it doesn't be handled.

[mattn]

And I noticed wired behavior of ReadFile.

#include <windows.h>

#include <stdio.h>

int

main(int argc, char* argv[]) {

char buf[256] = {0};

DWORD nr = 0;

memset(buf, 0, sizeof(buf));

BOOL r = ReadFile((HANDLE)_get_osfhandle(0), buf, sizeof(buf), &nr, NULL);

printf("%d, %d\n", r, nr);

printf("%d\n", buf[0]);

printf("%d\n", buf[1]);

printf("%d\n", buf[2]);

printf("%d\n", buf[3]);

return 0;

}

When type ^Z at the first, buf[0] become 0x1a and nr become 0.

When type 12^Z, buf become 0x49, 0x50, 0x1a, 0x13...

This behavior doesn't work on redirected files.

So it seems that ReadFile/ReadConsole have 

[mattn]

Ooops sorry.

> So it seems that ReadFile/ReadConsole have ...

So it seems that ReadFile/ReadConsole have especially behavior for ^Z\r\n

[brainman]

On Tuesday, 27 August 2013 16:17:23 UTC+10, mattn wrote:

..., If I want to write cat program using io.Copy, we must handle keyboard event to handle EOF. ...

You can just use whatever Windows API you want instead. For example http://play.golang.org/p/oLfB05kt7C

Alex 

[brainman]

On Tuesday, 27 August 2013 16:23:53 UTC+10, Dave Cheney wrote:

So windows terminal does not actually close the input stream when you
use ^Z, but sends the 0x1a char down the stream and the application at
the other end of it interprets it as EOF ? Surely this can't be true.

There are many ways to skin the cat. Some APIs handle ^Z, others don't.

Alex 

[mattn]

When type ^Z on console, ReadFile get buf=0x1A,0x0D,0xA and nr=0, But ReadConsole get buf=0x1A,0x0D,0x0A and nr=3. If we used ReadFile instead of ReadConsole, it will work fine. However we are using ReadConsole because we want to handle the leading bytes of multi-byte. I guess it's a reason for go's matter, not a reason of use case, So I think we have better to:

* Use ReadFile instead of ReadConsole

or 

* Handle ^Z

[brainman]

On Tuesday, 27 August 2013 17:31:33 UTC+10, mattn wrote:

...

Why should we implement ^Z handling?

Alex 

[nu774]

When you call ReadFile() on ordinary line input mode and user enters ^Z followed by a newline, ReadFile() will set lpNumberOfBytesRead to zero, which means no more byte has been read, and usually be interpreted as EOF (This is the same as read() returning zero on Unix). If ReadFile() is indirectly called via stdio of C, eof flag is set in this case.

On the other hand, in the same condition ReadConsole() will return as 3 chars (^Z + CR + LF) has been read, which cannot usually be interpreted as a signal of EOF.

Without handling of ^Z, there will no general way of signaling EOF from interactive user input on Windows console. That's the problem.

Note that semantics of ^Z on Windows is different from EOF char of Unix tty, that just sends queued input immediately to the process.

On Windows, ^Z doesn't break the blocking read on cooked mode, and user still has to enter newline to send input. And only ^Z at BOL is treated as a signal of EOF by ReadFile().

It's different anyway, but I believe "^Z at BOL as EOF" semantics is quite common on Windows (it's used by ReadFile(), and indirectly by stdio, iostream or something), and better to be followed.

[mattn]

Before CL 7312053, it worked fine with ^Z. But after the changes, the behavior is changed. So I want to get back the older behavior.

[mattn]

For example, we can get back to use RaedFile. This CL decode mutli-byte with using MultiByteToWideChar.

https://codereview.appspot.com/13246045

[brainman]

On Tuesday, 27 August 2013 18:40:41 UTC+10, mattn wrote:

Before CL 7312053, it worked fine with ^Z. But after the changes, the behavior is changed. So I want to get back the older behavior.

Every CL that changes Go source files changes "behavior". That is not a good reason to change. The "behavior" is not documented anywhere.

^Z is not managed by API call that we use ReadConsole. If we want something like that programmed into our stdlib, then we should explain why we doing it and document it. I need help with that.

Alex 

[mattn]

This is a problem which should be default for golang.

* ^Z is handled

  ^Z is handled in default. you can interrupt reading console input with ^Z like generally windows console application.

  If you want to handle ^Z, you must implement your stdin which use ReadConsole. 

* ^Z is not handled

  ^Z isn't handled in default. you can't interrupt reading console input with any keys. This is go's original behavior. If you want to handle ^Z, you must implete your stdin which use ReadFile.

I'm thinking the first have advantage than second.

[mattn]

Alex, why you don't want change?

[mattn]

When reading pipe, readConsole isn't called. So:

C:\> type file_contains_ctrl_z.txt | ./my-go-app

 This works well. For exmaple

C:\> type con

foo

^Z

This send EOF, but

C:\> my-go-app

foo

^Z

^Z

This doesn't send EOF.

On Tuesday, August 27, 2013 3:23:53 PM UTC+9, Dave Cheney wrote:

[brainman]

On Wednesday, 28 August 2013 17:21:20 UTC+10, mattn wrote:

Alex, why you don't want change?

Like I said earlier, I want to see some official documentation about how it is supposed to work.

Alternatively if Go team wants to come up with the way to do it, I'll be happy with that too.

Alex 

[mattn]

http://msdn.microsoft.com/en-us/library/system.console.readline.aspx

> If the Ctrl+Z character is pressed when the method is reading input from the console, the method returns null. This enables the user to prevent further keyboard input when the ReadLine method is called in a loop. The following example illustrates this scenario.

http://en.wikipedia.org/wiki/End-of-file

> In Microsoft's DOS and Windows (and in CP/M and many DEC operating systems), reading from the terminal will never produce an EOF. Instead, programs recognize that the source is a terminal (or other "character device") and interpret a given reserved character or sequence as an end-of-file indicator; most commonly this is an ASCII Control-Z, code 26. Some MS-DOS programs, including parts of the Microsoft MS-DOS shell (COMMAND.COM) and operating-system utility programs (such as EDLIN), treat a Control-Z in a text file as marking the end of meaningful data, and/or append a Control-Z to the end when writing a text file. This was done for two reasons:

[brainman]

> On Wednesday, 28 August 2013 17:50:17 UTC+10, mattn wrote:

> http://msdn.microsoft.com/en-us/library/system.console.readline.aspx

> ...

> http://en.wikipedia.org/wiki/End-of-file

> ...

I take it these two are supposed to explain what happens and why in

os: Handle CTRL-Z on windows console

https://codereview.appspot.com/13265043

but I don't see it where they do explain your approach. Also, since we are using win32 API, I would expect to see some references to win32 API, not .Net lib or wikipedia.

Looking at the other CL you have posted here

os,syscall: use ReadFile/MultiByteToWideChar to read from console

https://codereview.appspot.com/13246045/

I don't think it will work. You assume that every program uses same code page, which isn't true.

Alex

[mattn]

> On Thursday, August 29, 2013 10:18:42 AM UTC+9, brainman wrote:

> > On Wednesday, 28 August 2013 17:50:17 UTC+10, mattn wrote:

> > http://msdn.microsoft.com/en-us/library/system.console.readline.aspx

> > ...

> > http://en.wikipedia.org/wiki/End-of-file

> > ...

> 

> I take it these two are supposed to explain what happens and why in

> 

> os: Handle CTRL-Z on windows console

> https://codereview.appspot.com/13265043

> 

> but I don't see it where they do explain your approach. Also, since we are using win32 API, I would expect to see some references to win32 API, not .Net lib or wikipedia.

I say in earlier, Before CL 7312053, It had a behavior like written in Wikipedia(CTRL-Z interrupt stdin). But After CL 7312053, it isn't. Maybe ReadConsole API is designed to be possible to handle CTRL-Z for makeing the console application which need to handle CTRL-Z (Because the name of API contains Console).

> Looking at the other CL you have posted here

> 

> os,syscall: use ReadFile/MultiByteToWideChar to read from console

> https://codereview.appspot.com/13246045/

> 

> I don't think it will work. You assume that every program uses same code page, which isn't true.

It use GetACP() at once for talking my approach. If you allow this approach, I'll change this patch to get CodePage in everytime.

Thanks.

[brainman]

On Thursday, 29 August 2013 14:02:46 UTC+10, mattn wrote:

... If you allow this approach, I'll change this patch to get CodePage in everytime.

I have no idea what we should do here. Perhaps others understand it better. Sorry.

Alex 

[mattn]

Anyone, please give us your opinions?

[Niklas Schnelle]

What exactly is the use case here? Does anyone seriously use that broken abomination of a console on Windows? 

When usability is as bad as I remember the Windows console (don't get this wrong I'm an avid shell user on *nix) the only reason I can think

of why something like ctrl^z ability would matter is to keep a Go program designed for *nix functional on Windows you never know what people are forced to do...

[mattn]

> What exactly is the use case here? Does anyone seriously use that broken abomination of a console on Windows? 

At least, I use ^Z many times, And I see the tweet "Go's console program on windows doesn't handle ^Z" in recently.

> When usability is as bad as I remember the Windows console (don't get this wrong I'm an avid shell user on *nix) the only reason I can think

> of why something like ctrl^z ability would matter is to keep a Go program designed for *nix functional on Windows you never know what people are forced to do...

I'm thinking the console program (irrespective of go) should provide the way to interrupt stdin.

[mattn]

I'm looking http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/dos_diffs.mspx

> Then type the text you want sorted, pressing ENTER at the end of each line. When you have finished typing text, press CTRL+Z, and then press ENTER. The sort command displays the text you typed, sorted alphabetically.

http://support.microsoft.com/kb/25996/en-us

http://support.microsoft.com/kb/68423/en-us

http://support.microsoft.com/kb/68787/en-us

I'm wondering why go doesn't provide the way to interrupt stdin in default.