Normal data and ancillary messages on sockets
Albert Strasheim
I've been looking at UNIX sockets with the idea of fixing issue 1101
and supporting the sending of file descriptors using the SCM_RIGHTS
ancillary message.
As far as I can tell, things get a bit tricky when one has a mix of
normal data and ancillary messages arriving on the same socket.
Firstly, with a socket file descriptor, it seems one has a few options
for getting data:
- call read(), which doesn't allow any special flags
- call recv(), which takes the special MSG_* flags and is normally
used for connected sockets
- call recvfrom() which has the additional src_addr parameter
- call recvmsg(), which can deal with a mix of normal data and
ancillary messages, given a msghdr that is set up properly
When using recvmsg, normal data goes into buffers pointed to by
msghdr.msg_iov and ancillary messages go to msghdr.msg_control.
I need to test what happens when msg_iov is null and there is normal
data on the socket, or if msg_control is null and there is ancillary
messages on the socket. My guess is that you will get a msghdr back
with the MSG_TRUNC or MSG_CTRUNC flag set and the data will be
discarded.
If my understanding is correct, doing read() and recvmsg() on the same
socket to receive normal data and ancillary messages can cause you to
lose data. Also, the rio mutex in fd.go would indicate that this is
the wrong thing to do anyway.
Thus, if you are expecting a mix of normal data and ancillary messages
on a socket it seems one always needs to use recvmsg with a msghdr
with buffers for both kinds of data.
Perhaps the simplest way to support this would be with something like:
type Conn interface {
Recvmsg(b []byte, ancil []byte) (n int, nancil int, msgflags int,
err os.Error)
}
Further functions for parsing the ancillary messages could then be
added to the net package at a later stage.
The ip(7) manual page also has a long list of ancillary messages that
can arrive on IP sockets that might be useful, so this issue isn't
restricted to UNIX sockets.
Does anybody have any thoughts on a better way to deal with this?
Regards
Albert
Adam Langley
> I need to test what happens when msg_iov is null and there is normal
> data on the socket, or if msg_control is null and there is ancillary
> messages on the socket. My guess is that you will get a msghdr back
> with the MSG_TRUNC or MSG_CTRUNC flag set and the data will be
> discarded.
(This information may be Linux specific)
As I recall, control messages are associated with normal data. Thus,
you need to read a byte of normal data in order to get a control
message. You cannot set msg_iovlen to 0 and expect to get only control
messages.
The converse is not true. You can set controllen to 0 and any control
messages will be discarded (and MSG_CTRUNC will let you know that this
occurred.)
MSG_TRUNC will occur on a datagram based socket when a partial
datagram was read. The rest of the datagram is lost.
> If my understanding is correct, doing read() and recvmsg() on the same
> socket to receive normal data and ancillary messages can cause you to
> lose data.
This is correct. If you wish to mix read() and recvmsg() you must
somehow know when to expect control messages so that you can use
recvmsg to catch them. (This is how Chrome's renderer communication
works. Due to a detail of Chrome's design, read() is significantly
faster so it used by default and the IPC protocol is framed such that
recvmsg is only used when needed.)
> type Conn interface {
> Recvmsg(b []byte, ancil []byte) (n int, nancil int, msgflags int,
> err os.Error)
> }
That seems reasonable to me, but rsc often has good ideas that I don't think of.
AGL
Albert Strasheim
On Oct 7, 2:31 pm, Adam Langley <a...@golang.org> wrote:
> On Thu, Oct 7, 2010 at 7:55 AM, Albert Strasheim <full...@gmail.com> wrote:
> > I need to test what happens when msg_iov is null and there is normal
> > data on the socket, or if msg_control is null and there is ancillary
> > messages on the socket. My guess is that you will get a msghdr back
> > with the MSG_TRUNC or MSG_CTRUNC flag set and the data will be
> > discarded.
>
> (This information may be Linux specific)
>
> As I recall, control messages are associated with normal data. Thus,
> you need to read a byte of normal data in order to get a control
> message. You cannot set msg_iovlen to 0 and expect to get only control
> messages.
> The converse is not true. You can set controllen to 0 and any control
> messages will be discarded (and MSG_CTRUNC will let you know that this
> occurred.)
>
> MSG_TRUNC will occur on a datagram based socket when a partial
> datagram was read. The rest of the datagram is lost.
Thanks. I also found some notes here (for future reference):
http://www.lst.de/~okir/blackhats/node121.html
> > If my understanding is correct, doing read() and recvmsg() on the same
> > socket to receive normal data and ancillary messages can cause you to
> > lose data.
>
> This is correct. If you wish to mix read() and recvmsg() you must
> somehow know when to expect control messages so that you can use
> recvmsg to catch them. (This is how Chrome's renderer communication
> works. Due to a detail of Chrome's design, read() is significantly
> faster so it used by default and the IPC protocol is framed such that
> recvmsg is only used when needed.)
>
> > type Conn interface {
> > Recvmsg(b []byte, ancil []byte) (n int, nancil int, msgflags int,
> > err os.Error)
> > }
>
> That seems reasonable to me, but rsc often has good ideas that I don't think of.
I have started work on changing fd.go to get a feel of what has to
happen.
Patch for Recvmsg looks something like this: http://pastebin.com/btyfWZMq
Some of the code can probably move from there into syscall, so that
syscall doesn't have to export anyToSockaddr. I also need to add a
SetLen function for Iov and a SetControllen function for Msghdr to
deal with the different data types on 386/amd64.
If Russ could give us his thoughts, I can start by making the
necessary changes to this CL:
http://codereview.appspot.com/2331044/
to get syscall ready, and then do the remaining work and testing in
net.
Regards
Albert