clarification on io.Reader EOF semantics

[Dan Kortschak]

I would like some clarification on io.Reader semantics wrt EOF.

I have an io.Reader implementation that will return a io.EOF in some
cases where the []byte parameter is zero length. It's not clear to me
whether this contravenes the contract in the io.Reader documentation -
it is behaviour that is helpful for my use case, so I would like to keep
it, but I'd be keen to hear views on this behaviour.

thanks

[Benjamin Measures]

It sounds like you want to return an EOF for something that's not an EOF.

[Dan Kortschak]

On Mon, 2015-03-16 at 21:05 -0700, Benjamin Measures wrote:
> It sounds like you want to return an EOF for something that's not an
> EOF.

Mmmm. Maybe. If you are at the EOF and you do a read should that return
an EOF?

[Benjamin Measures]

> If you are at the EOF and you do a read should that return an EOF?

How did you get to that EOF?

If a non-zero read, then _that_ read and subsequent reads should return EOF (non-zero or otherwise).

The stream begins with EOF (ie. no bytes to read), then IMO any read should return EOF (non-zero or otherwise).

[Benjamin Measures]

> The stream begins with EOF

*If the stream begins with EOF....

[peterGo]

kortschak,

Here's a program that I wrote some time ago to illustrate my understanding of the io.Reader rules.

readfile.go:

package main

import (
    "bufio"
    "fmt"
    "io"
    "os"
)

func main() {
    f, err := os.Open("readfile.go")
    if err != nil {
        fmt.Println(err)
        return
    }
    defer f.Close()
    r := bufio.NewReader(f)

    fileLen := int64(0)
    buf := make([]byte, 0, 4*1024)
    for {
        n, err := r.Read(buf[:cap(buf)])
        buf = buf[:n]
        if n == 0 {
            if err == nil {
                continue
            }
            if err == io.EOF {
                break
            }
            fmt.Println(err)
            return
        }

        // Do something with buf
        fileLen += int64(len(buf))

        if err != nil && err != io.EOF {
            fmt.Println(err)
            return
        }
    }
    fmt.Println("file length:", fileLen, "bytes")
}

Playground: http://play.golang.org/p/8cPImoflen

I'll interpret your io.Reader case, len(p) == 0 returning n == 0 and err == io.EOF, as a valid EOF.

For type io.Reader (http://golang.org/pkg/io/#Reader):

"a Reader returning a non-zero number of bytes at the end of the input stream may return either err == EOF or err == nil. The next Read should return 0, EOF regardless."

If the next read has len(p) == 0 then you should return 0, EOF regardless.

Peter

[Dan Kortschak]

On Mon, 2015-03-16 at 21:37 -0700, Benjamin Measures wrote:
> > If you are at the EOF and you do a read should that return an EOF?
>
> How did you get to that EOF?

Reading some bytes.

> If a non-zero read, then _that_ read and subsequent reads should
> return EOF (non-zero or otherwise).

Not according to the io.Reader docs.

```
An instance of this general case is that a Reader returning a non-zero

number of bytes at the end of the input stream may return either err ==
EOF or err == nil. The next Read should return 0, EOF regardless.

```

>
> The stream begins with EOF (ie. no bytes to read), then IMO any read
> should return EOF (non-zero or otherwise).

OK, that's what I'm doing.

The reason this came up was that when I did an io.ReadFull on the
reader, I get an unwanted outcome - since len(p) == n, it does not
attempt the read loop, so it doesn't return the EOF. This is an issue
because offsets into the reader (important) are not bijective with
offsets into the backing stream and the way that is currently resolved
is more efficient for my purposes than it would be other ways.

I tried doing a similar thing with an easily accessible io.Reader
(strings.Reader) and it does not give an EOF in this manner:

http://play.golang.org/p/JwhBtBPbcM

[Dan Kortschak]

On Mon, 2015-03-16 at 21:46 -0700, peterGo wrote:
> If the next read has len(p) == 0 then you should return 0, EOF
> regardless.
>

Yes, I think that is the key. Thanks.

[Dan Kortschak]

The corollary of this is that I think strings.Reader does not do what
the docs say.

Before I file an issue, I'd like to see whether other people think this
is a bug or not - I think it is borderline.

[minux]

it's fine to return 0, nil if len(buf) == 0. Don't return io.EOF until the reader

has really reached the end of stream.

See http://golang.org/pkg/io/#Reader:

"Implementations of Read are discouraged from returning a zero byte count

with a nil error, except when len(p) == 0. Callers should treat a return of 0 and

nil as indicating that nothing happened; in particular it does not indicate EOF."

[Dan Kortschak]

On Tue, 2015-03-17 at 00:58 -0400, minux wrote:
> it's fine to return 0, nil if len(buf) == 0. Don't return io.EOF until
> the reader has really reached the end of stream.

But, that is the heart of the issue. I think that it is actually the end
of the stream:

http://play.golang.org/p/b7X-Te_ltg // with relevant text from io.Reader

> See http://golang.org/pkg/io/#Reader:
> "Implementations of Read are discouraged from returning a zero byte
> count with a nil error, except when len(p) == 0. Callers should treat
> a return of 0 and nil as indicating that nothing happened; in
> particular it does not indicate EOF."
>

I don't think that language is relevant here - that is talking about
general case, not the case where there is actually an EOF.

[minux]

On Tue, Mar 17, 2015 at 1:03 AM, Dan Kortschak <dan.ko...@adelaide.edu.au> wrote:

On Tue, 2015-03-17 at 00:58 -0400, minux wrote:
> it's fine to return 0, nil if len(buf) == 0. Don't return io.EOF until
> the reader has really reached the end of stream.

But, that is the heart of the issue. I think that it is actually the end
of the stream:

http://play.golang.org/p/b7X-Te_ltg // with relevant text from io.Reader

If it's a real EOF and len(p) == 0, then returning 0, EOF and 0, nil both seem fine to me.

However, 0, EOF is slightly more preferable.

[Dan Kortschak]

On Tue, 2015-03-17 at 01:08 -0400, minux wrote:
> > http://play.golang.org/p/b7X-Te_ltg

> If it's a real EOF and len(p) == 0,

It clearly is in the linked case.

> then returning 0, EOF and 0, nil both seem fine to me.

> However, 0, EOF is slightly more preferable.

From my reading of the documentation on io.Reader only the latter is
described - "The next Read should return 0, EOF regardless." Either the
error is in the docs (they are too restrictive), or in the
implementation of {bytes,strings}.Reader (it is not tight enough).

Is either of these worth an issue? If so which one.

[minux]

The corner cases of io.Reader are really annoying. File an issue on io.Reader

please. I think the behavior of returning 0, nil is reasonable if len(p) == 0 and

even if we fixed the strings package, there might be other cases elsewhere

that we can't fix.

Fixing the docs does make it clear that you can't use Read(zeroSlice) to test

for EOF though.

[Dan Kortschak]

On Tue, 2015-03-17 at 01:26 -0400, minux wrote:
> The corner cases of io.Reader are really annoying. File an issue on
> io.Reader please.

https://github.com/golang/go/issues/10182

> I think the behavior of returning 0, nil is reasonable if len(p) == 0
> and even if we fixed the strings package, there might be other cases
> elsewhere that we can't fix.

I expect that it would break existing code too.

> Fixing the docs does make it clear that you can't use Read(zeroSlice)
> to test for EOF though.

Thanks.

[peterGo]

Dan,

I don't agree.

Your example: http://play.golang.org/p/b7X-Te_ltg

The first read reads the text (len(p) == len(text)), it does not encounter EOF.

The second read reads zero bytes (len(p) == 0), it does not encounter EOF.

"Implementations of Read are discouraged from returning a zero byte
count with a nil error, except when len(p) == 0"

You can only encounter EOF if you are reading non-zero bytes (len(p) >
0) or it's the next read after err == io.EOF.

Peter

[Dan Kortschak]

On Tue, 2015-03-17 at 01:44 -0400, peterGo wrote:
> I don't agree.
>
> Your example: http://play.golang.org/p/b7X-Te_ltg
>
> The first read reads the text (len(p) == len(text)), it does not
> encounter EOF.

I guess this depends on how you define an EOF; the documentation for
io.Reader clarifies this:

"An instance of this general case is that a Reader returning a non-zero

number of bytes at the end of the input stream may return either err ==
EOF or err == nil."

The first read has returned a non-zero number of bytes, and that
sequence of bytes was from the end of the input stream.

> The second read reads zero bytes (len(p) == 0), it does not encounter
> EOF.

This depends on the answer to the previous question. If the previous did
(and I think this is the case based on my argument above), this falls
into the category of "next Read should return 0, EOF regardless."

> "Implementations of Read are discouraged from returning a zero byte
> count with a nil error, except when len(p) == 0"

This says what should happen when len(p) != 0, it makes no claim about
what should happen when len(p) == 0 which remains open.

> You can only encounter EOF if you are reading non-zero bytes (len(p) >
> 0)

Yes. The first read.

> or it's the next read after err == io.EOF.

Yes. The second read.

[Dan Kortschak]

On 17/03/2015, at 4:44 PM, "Dan Kortschak" <dan.ko...@adelaide.edu.au> wrote:

>
>> "Implementations of Read are discouraged from returning a zero byte
>> count with a nil error, except when len(p) == 0"
>
> This says what should happen when len(p) != 0, it makes no claim about
> what should happen when len(p) == 0 which remains open.

Also, it doesn't preclude returning a zero byte count with a non-nil error (other rules permitting) under either len(p) state.

[peterGo]

Dan,

First, let's consider the meaning of "the end of the input stream."

To illustrate, lets interleave reads and writes to a file or buffer:

http://play.golang.org/p/ZGWDk8JOKr

If we read ahead to determine the end of the input stream, this
wouldn't work. We don't encounter the end of the input stream until we
go past it.

Peter

On 3/17/15, Dan Kortschak <dan.ko...@adelaide.edu.au> wrote:

[Dan Kortschak]

That's not true. We may not know that we have reached the end of some generic stream until we have attempted to read one byte beyond it. This is true. However, there are stream where it is feasible to know exactly when we will reach the end of the stream (for a file for example, we could stat the file), then we know exactly where the end is before we try to go past it.

Do you drive of a bridge that is out rather than stop before the edge? Only if you can't see it.

[atd...@gmail.com]

In general, "reading 0 byte" is not doing anything. So returning (0, nil) makes sense.

In your case, if you consider that EOF is an error that is thrown to warn you that you are trying to read "past" the file length, well, it is not the case if you are trying to read "0 byte" from a 0 length file.

(0, nil) still makes sense here, imho.

[Benjamin Measures]

On Tuesday, 17 March 2015 04:47:39 UTC, kortschak wrote:

On Mon, 2015-03-16 at 21:37 -0700, Benjamin Measures wrote: 

> The stream begins with EOF (ie. no bytes to read), then IMO any read
> should return EOF (non-zero or otherwise).

[...]

I tried doing a similar thing with an easily accessible io.Reader
(strings.Reader) and it does not give an EOF in this manner:

http://play.golang.org/p/JwhBtBPbcM

After seeing your example, I now disagree with my opinion.

If reading len(p) bytes doesn't give EOF, then why should there be a special case where len(p)==0?

http://play.golang.org/p/4oH_a954qA

Perhaps then io.EOF should be thought of as a value to be read -- that is, like an out-of-channel null byte AKA end-of-stream terminator.

[Dan Kortschak]

On 17/03/2015, at 9:47 PM, "atd...@gmail.com" <atd...@gmail.com> wrote:

> In general, "reading 0 byte" is not doing anything. So returning (0, nil) makes sense.
> In your case, if you consider that EOF is an error that is thrown to warn you that you are trying to read "past" the file length,

This seems to be a common interpretation. Though one I think is incorrect.

Why must it read *past* the end of the file? That is certainly not how it is described in the io.Reader documentation (talks about bytes at the end of a stream), nor in the io.EOF documentation (talks about no more data being available).

[Dan Kortschak]

On 17/03/2015, at 9:54 PM, "Benjamin Measures" <saint....@gmail.com> wrote:

> After seeing your example, I now disagree with my opinion.
>
> If reading len(p) bytes doesn't give EOF, then why should there be a special case where len(p)==0?
> http://play.golang.org/p/4oH_a954qA

I think it's a mistake to decide generalities from a single implementation.

It needs to be determined from the documentation. Minux's proposal covers the cases that exist now, clearly and just tightly enough.

> Perhaps then io.EOF should be thought of as a value to be read -- that is, like an out-of-channel null byte AKA end-of-stream terminator.

Absolutely.

[atd...@gmail.com]

I think that EOF was meant to warn of an error (well, it is an error value).

If you read the exact amount of bytes from a  buffer, there shouldn't be an error.

I think that it is the reasoning behind EOF. Typically in streaming situations where you don't know where the end of the file is.

If you are using structured data, EOF is not necessary ( ErrUnexpectedEOF is to be used in case you try to "overread"). 

[Dan Kortschak]

But io.Reader docs say you can.

[atd...@gmail.com]

Yes, but for n > 0 number of bytes read.

If n == 0 and EOF was not returned on the previous read, I reckon it should return (0,nil).

Otherwise, you should indeed see (0, EOF) being returned.

[atd...@gmail.com]

Quick correction,

 

If n == 0 and EOF was not returned on the previous read, I reckon it should return (0,nil)  when len(p) == 0

in accordance with the following : 

http://golang.org/src/io/io.go?s=2834:3082#L64

[Wojciech S. Czarnecki]

Dnia 2015-03-17, o godz. 11:46:01
Dan Kortschak <dan.ko...@adelaide.edu.au> napisał(a):

> will return a io.EOF in some
> cases where the []byte parameter is zero length.

I personally always imagined EOF similarily to
zero-width something. The EOF is to be returned
if, and only if, we'd *crossed over* it. Till that
we're still on the stream.

s. - in stream byte
n. - not in stream byte

s.s.s.s.s.s.s.|n.n.n.n.n.n.n
^ zero width EOF

Thereafter:

> will return a io.EOF in some cases where the []byte
> parameter is zero length.

seems wrong to me. Returning zero bytes from the stream
of zero bytes still does not cross over the (zero width) EOF.
Doing it twice, might. (Imagined as: the first read leaves
stream index touching left side of the EOF, second read will
step over it to its right side.)

my $0.02


--
Wojciech S. Czarnecki
^oo^ OHIR-RIPE

[Dan Kortschak]

This is exactly what I am saying. The first read in my example *may* return an EOF after having read exactly the 17 bytes in the stream. Then, according to the docs (previous to minux's change) *any* subsequent read should return an EOF, irrespective of len(p).

On 18/03/2015, at 1:05 AM, "atd...@gmail.com" <atd...@gmail.com> wrote:

Yes, but for n > 0 number of bytes read.

If n == 0 and EOF was not returned on the previous read, I reckon it should return (0,nil).

Otherwise, you should indeed see (0, EOF) being returned.

On Tuesday, March 17, 2015 at 1:58:26 PM UTC, kortschak wrote:

But io.Reader docs say you can.

On 18/03/2015, at 2:10 AM, "atd...@gmail.com" <atd...@gmail.com> wrote:

Quick correction,

 

If n == 0 and EOF was not returned on the previous read, I reckon it should return (0,nil)  when len(p) == 0

in accordance with the following : 

http://golang.org/src/io/io.go?s=2834:3082#L64

I really don't understand why people keep quoting this section; it says what you should do when you have a nil err and len(p) is 0, is says absolutely nothing about what should be done when err is not nil.

The crucial text then is "should return 0, EOF on the next read regardless", given that the previous read has read all the data and may return an EOF.

[atd...@gmail.com]

On Tuesday, March 17, 2015 at 5:18:36 PM UTC, kortschak wrote:

This is exactly what I am saying. The first read in my example *may* return an EOF after having read exactly the 17 bytes in the stream. Then, according to the docs (previous to minux's change) *any* subsequent read should return an EOF, irrespective of len(p).

Haha, I see.

http://golang.org/src/io/io.go?s=2130:2607#L64 should probably be kept for len(p) > 0 (so any subsequent attempt at reading a non zero number of bytes should return (0,EOF) )

 

http://golang.org/src/io/io.go?s=2834:3082#L64

I really don't understand why people keep quoting this section; it says what you should do when you have a nil err and len(p) is 0, is says absolutely nothing about what should be done when err is not nil.

The crucial text then is "should return 0, EOF on the next read regardless", given that the previous read has read all the data and may return an EOF.

I read it as saying that len(p) == 0 is the only time when a Read should return (0,nil). Otherwise, a read will either return (0,Err!=nil) | (0,EOF) | (n>0, nil) | (n>0, Err!=nil), (n>0, EOF).

Because reading "nothing" from a byte buffer is always fine.

Now, the one thing I was confused about in your example  was that you had n == len(p). It does not return EOF or any error but obviously the next Read with len(p)>0 should return (0,EOF), since there is no more bytes to read. 

On the other hand, if len(p) == 0, it should return (0,nil).

[Uli Kunitz]

I argue that strings.Reader is not required to return EOF here, because the End of File has not been encountered. All requests can be satisfied without an error including the request to read zero bytes. So it is perfectly fine and its behaviour doesn't violate the description of io.Reader.

While the behaviour proposed would be ok as far as I can see, I would regard it as strange for the single reason that reading as much bytes as there are in the stream should not lead to an error. Reading nothing afterwards should also not lead to an error.

[Dan Kortschak]

The discussion is not moving forward AFAICT.

The issue here from what I can see is that people are bringing conceptual baggage to the table rather than arguing logically from the text of the documentation; e.g. there is no notion of an EOF being "encountered" in the documentation - io.EOF is defined as being returnable when there is "no more input is available" which can be known without reading past the last data byte.

[Uli Kunitz]

The io.Reader documentation contains following two sentences:

When Read encounters an error or end-of-file condition after successfully reading n > 0 bytes, it returns the number of bytes read. It may return the (non-nil) error from the same call or return the error (and n == 0) from a subsequent call.

The third word is encounters. 

The documentation talks then about an instance of the general case, where the next call must return EOF. But this is a special case and in the general case only one of the following calls must return EOF. The documentation only discourages returning 0 bytes without an error with the exception of a zero-length buffer, but doesn't outlaw it.

In my reading you could even state that returning (0, nil) for all calls with a zero length buffer even after encountering EOF and returning n less than len(buf) doesn't violate the specification. The type strings.Reader does exactly this. Note also that an EOF error doesn't need to be repeated.

I believe the Reader specification doesn't provide more specificity by intention. Readers that are concurrently filled while read might not be able to fill the buffer completely at some point in time so one cannot assume that reading less bytes than the buffer size indicates EOF.

I hope I could show that strings.Reader doesn't violate the io.Reader specification and while your proposed behaviour is satisfying the specification it doesn't forbid other behaviours including strings.Reader.

[Dan Kortschak]

On Tue, 2015-03-17 at 15:03 -0700, Uli Kunitz wrote:
> The io.Reader documentation contains following two sentences:
>
> When Read encounters an error or end-of-file condition after successfully
> reading n > 0 bytes, it returns the number of bytes read. It may return the
> (non-nil) error from the same call or return the error (and n == 0) from a
> subsequent call.
>
>
> The third word is encounters.

Yes, sorry. How do you define "encounter"? I would say it's encountering
a state, which is defined in io.EOF as when there is no further input
available.

> The documentation talks then about an instance of the general case, where
> the next call must return EOF. But this is a special case and in the
> general case only one of the following calls must return EOF. The
> documentation only discourages returning 0 bytes without an error with the
> exception of a zero-length buffer, but doesn't outlaw it.

I'm not asking for it to be outlawed.

> In my reading you could even state that returning (0, nil) for all calls
> with a zero length buffer even after encountering EOF and returning n less
> than len(buf) doesn't violate the specification. The type strings.Reader
> does exactly this. Note also that an EOF error doesn't need to be repeated.

After comments on the gerrit CL, I can see why people see this to be the
case, however, I stand by the claim that the wording is unfortunate
(removal of the word "regardless", would fix that). The emphatic nature
of that clause, to my mind, makes the sentence it is in take priority
over the last para of the io.Reader documentation.

> I believe the Reader specification doesn't provide more specificity by
> intention.

I'm not claiming it's not specific enough, but rather that it's too
specific.

> Readers that are concurrently filled while read might not be
> able to fill the buffer completely at some point in time so one cannot
> assume that reading less bytes than the buffer size indicates EOF.

I think this is a different issue.

> I hope I could show that strings.Reader doesn't violate the io.Reader
> specification and while your proposed behaviour is satisfying the
> specification it doesn't forbid other behaviours including strings.Reader.

I agree, when regardless is interpreted to refer only to the previous
clause. In my English, that is not unambiguously possible.

thanks