guarantees on net.Conn and net.UDPConn Read

[Dan Kortschak]

The Conn and UDPConn Read methods look like io.Reader Read methods, but
there is no explicit claim that Conn is an io.Reader. Are the Read
methods of these two types identical in behaviour to io.Reader?
Specifically, are the reads allowed to fill the buffer with arbitrary
numbers of bytes in 0 <= len(p) <= n?

Also, can UPDConn.Read fill the buffer with more than one packet if the
buffer would accommodate that? (I am guessing yes, and that if this is
important then the ReadFrom method should be used).

thanks
Dan

[Dan Kortschak]

bump?

[robert engels]

There is no claim because that is not how Go interfaces work - if you implement it, you are it - that being said, I wouldn’t use Read() with UDPConn - I would use one of the packet/msg reader methods instead.

Read() is a pass through to fd.Read() so it will be implementation dependent, but posix should be that only a single packet is returned. Since UDP is packet oriented, anything else would be unusable, since the packet size forms the boundaries. see http://developerweb.net/viewtopic.php?id=2952

The buffer size controls the size of the OS kernel buffer assigned to the socket - nothing to do with streaming reads.

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/1557706800.21310.59.camel%40kortschak.io.
> For more options, visit https://groups.google.com/d/optout.

[Dan Kortschak]

This is not quite true. The language itself doesn't make claims other
than types and method names. However, there are conventions around the
semantics of methods in an interface. For example, a Read method that
returns 0, nil is allowed for io.Reader, but frowned upon unless the
buffer is zero length, and a Read method that fills the buffer with n
bytes and returns n-1 (or n+1), nil is not a correct implementation of
io.Reader. Nor is an implementation correct if it retains the buffer.

Just as the BDNF is not a complete definition of the grammar of the
language, interface type decls are not complete definitions of
interfaces.

This is why I am asking.

[robert engels]

Which is a big problem with the Go “interface” specification…

But the de-facto standard is as I stated - if it implements the method it implements the interface. There is no requirement that the method/struct “be documented” to be an implementor - buyer beware!

We is also why you should never use method names that collide with “stdlib interfaces” unless you intend them to have the same semantics.

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/1557712717.21310.68.camel%40kortschak.io.

[Dan Kortschak]

Yes, I'm aware of all this. However, the net.UDPConn Read method states
that it "Read implements the Conn Read method", and does not the
io.Reader interface; net.Conn has the explicit method, not an embedding
of io.Reader.

I'd like to hear from someone who contributed to net.

[robert engels]

As I said, I reviewed the code - which you can do as well. It is below. At least for “unix-ish” systems - it goes to syscall.Read() which is why I linked you to the unix documentation on the difference between read() and recv().

Instead of waiting for someone else to confirm, you can think about it logically - sending packet based messages as a stream would not work as you have no boundaries, and no ordering guarantees - so streaming them would be useless - it would require every protocol on top of UDP to have embedded length and framing information in order to detect packet starts. Not to mention it would be impossible to read the OOB data (like sender address - since a UDP socket can receive from multiple senders - so you would have no ability to know where the packet came from (unless bound to a single sender).

udpsock.go -> net.go -> poll/fd_unix.go 

// Read implements io.Reader.

func (fd *FD) Read(p []byte) (int, error) {

if err := fd.readLock(); err != nil {

return 0, err

}

defer fd.readUnlock()

if len(p) == 0 {

// If the caller wanted a zero byte read, return immediately

// without trying (but after acquiring the readLock).

// Otherwise syscall.Read returns 0, nil which looks like

// io.EOF.

// TODO(bradfitz): make it wait for readability? (Issue 15735)

return 0, nil

}

if err := fd.pd.prepareRead(fd.isFile); err != nil {

return 0, err

}

if fd.IsStream && len(p) > maxRW {

p = p[:maxRW]

}

for {

n, err := syscall.Read(fd.Sysfd, p)

if err != nil {

n = 0

if err == syscall.EAGAIN && fd.pd.pollable() {

if err = fd.pd.waitRead(fd.isFile); err == nil {

continue

}

}

// On MacOS we can see EINTR here if the user

// pressed ^Z.  See issue #22838.

if runtime.GOOS == "darwin" && err == syscall.EINTR {

continue

}

}

err = fd.eofError(n, err)

return n, err

}

}

-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/1557715058.21310.72.camel%40kortschak.io.

[Kurtis Rader]

On Sun, May 12, 2019 at 7:38 PM Dan Kortschak <d...@kortschak.io> wrote:

Yes, I'm aware of all this. However, the net.UDPConn Read method states
that it "Read implements the Conn Read method", and does not the
io.Reader interface; net.Conn has the explicit method, not an embedding
of io.Reader.

And the Conn documentation says: "Conn is a generic stream-oriented network connection."  UDP sockets are not stream-oriented. As Robert has already pointed out the Read() method on such an object is a wrapper around the read() syscall. Which for a UDP socket is equivalent to recv() with flags set to zero. You can't use it to read multiple packets with a single call. Nor can you use it to read a packet incrementally. At least on UNIX like platforms the excess bytes will be discarded. 

--

Kurtis Rader

Caretaker of the exceptional canines Junior and Hank

[Dan Kortschak]

Thank you. I have reviewed the code. I was hoping for some friendly and
helpful input.

[Dan Kortschak]

Thank you, that has clarified what I was wanting to confirm.

Dan

[robert engels]

I am sorry if you think I was not helpful, nor friendly - I was trying to be both.

That is why I took the time to pass on the relevant documentation, and code.

-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/1557716698.29848.7.camel%40kortschak.io.

[Dan Kortschak]

No worries. Thanks for trying.

Dan

> > > > > > On May 12, 2019, at 8:58 PM, Dan Kortschak <dan@kortschak.i

[Dan Kortschak]

Further to this, you can see that having two diametrically opposite
claims (1. UDPConn.Read implements Conn.Read and Conn is a generic
stream-oriented network connection cf 2. UDP is not stream oriented)
might be somewhat confusing.

On Sun, 2019-05-12 at 20:00 -0700, Kurtis Rader wrote:

[jake...@gmail.com]

On Thursday, May 9, 2019 at 2:54:17 AM UTC-4, kortschak wrote:

The Conn and UDPConn Read methods look like io.Reader Read methods, but
there is no explicit claim that Conn is an io.Reader. Are the Read
methods of these two types identical in behaviour to io.Reader?

 

I can not say definitively, but on unix, UDPConn.Read() is actually net.conn.Read(), whch calls net.netFD.Read(), which in turn calls poll.FD.Read(). You can see there in the code:

// Read implements io.Reader.

The windows and plan9 versions also land in Read() functions that have the same comment. So it seems to me that UDPConn.Read() is an io.Reader. Of course, that only means that it conforms to the requirements of io.Reader.

Specifically, are the reads allowed to fill the buffer with arbitrary
numbers of bytes in 0 <= len(p) <= n?

 

Yes, according to io.Reader docs, an implementation may do that. Looking at the actual implementations of the poll.FD.Read functions, I would say that it seems like they will always return the full UDP packts, if it is <= len(p). A full search of the relevant platform docs would be required to say definitively.

Also, can UPDConn.Read fill the buffer with more than one packet if the
buffer would accommodate that? (I am guessing yes, and that if this is
important then the ReadFrom method should be used).

It looks to me like this will in fact read one packet at a time on amd64 unix. The function will return after a single successful syscall(SYS_READ) which, IIUC will just be one packet for UDP. Since there is no actual documentation guaranteeing this, you would have to look at the source code, and platform docs, for every platform you care about to be 100% sure. However, if it did not work this way, it would make UDP programming very hard, or even impossible. So I think it very likely.

 

thanks
Dan

 

[Dan Kortschak]

Thanks, Jake. This was very helpful.

On Mon, 2019-05-13 at 09:19 -0700, jake...@gmail.com wrote:
> On Thursday, May 9, 2019 at 2:54:17 AM UTC-4, kortschak wrote:
> >
> >
> > The Conn and UDPConn Read methods look like io.Reader Read methods,
> > but 
> > there is no explicit claim that Conn is an io.Reader. Are the Read 
> > methods of these two types identical in behaviour to io.Reader? 
> >
>  
> I can not say definitively, but on unix, UDPConn.Read() is actually 

> net.conn.Read() <https://golang.org/src/net/net.go?s=6157:6199#L163>,

> whch 
> calls net.netFD.Read(), which in turn calls poll.FD.Read() 

> <https://golang.org/src/internal/poll/fd_unix.go#L144>. You can see

> there 
> in the code:
>
> // Read implements io.Reader.
>
> The windows and plan9 versions also land in Read() functions that
> have the same comment. So it seems to me that UDPConn.Read() is an
> io.Reader. Of course, that only means that it conforms to the

> requirements of io.Reader <https://golang.org/pkg/io/#Reader>.

>
> Specifically, are the reads allowed to fill the buffer with
> arbitrary 
> >
> > numbers of bytes in 0 <= len(p) <= n? 
> >
>  

> Yes, according to io.Reader <https://golang.org/pkg/io/#Reader> docs,

[Ian Lance Taylor]

On Mon, May 13, 2019 at 2:45 PM Dan Kortschak <d...@kortschak.io> wrote:
>
> Thanks, Jake. This was very helpful.

It does seem that the docs in this area should be improved. Alas.

Ian

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/1557783886.17937.0.camel%40kortschak.io.

[Dan Kortschak]

Yes. There's also the unfortunate collision with io.ReaderFrom's method
name which is directionally opposite to net.PacketConn's ReadFrom. This
is unfixable because of Go1.

Dan