Consider the following example, consisting of a TCP client (sending
zeros) and a lazy TCP server (which does not consume received data).
$ socat -ddd OPEN:/dev/zero TCP:localhost:8003,sndbuf=2000,rcvbuf=2000
2009/12/04 17:30:21 socat[2880] E write(4, 0x954a758, 8192): Broken pipe
$ socat -ddd TCP-LISTEN:8003,reuseaddr,sndbuf=2000,rcvbuf=2000
EXEC:'sleep 30'
2009/12/04 17:30:21 socat[2879] E write(3, 0x89c2008, 3008): Broken pipe
Thus, the values Recv-Q/Send-Q should match the values SO_RCVBUF /
SO_SNDBUF configured. Isn't it? Yet this is not the case... why?
$ netstat -tpn
Proto Recv-Q Send-Q Local Address Foreign Address
State PID/Program name
tcp 3008 0 127.0.0.1:8003 127.0.0.1:54629
ESTABLISHED 2879/socat
tcp 0 2176 127.0.0.1:54629 127.0.0.1:8003
ESTABLISHED 2880/socat
Note that the server says he received 8003 bytes, which is consistent
with Recv-Q.
But the client said he sent 8192 bytes, which does not correspond to the
Send-Q:2176.
Moreover, none of its values is consistent with the setting of
SO_RCVBUF/SO_SNDBUF.
Does someone could explain?
Best regards
SO_RCVBUF and SO_SNDBUF are, ostensibly, the limits to how much can be
queued to the socket. Recv-Q and Send-Q are how much are actually
there.
Recv-Q will be that data which has not yet been pulled from the socket
buffer by the application.
Send-Q will be that data which the sending application has given to
the transport, but has yet to be ACKnowledged by the receiving TCP.
> $ socat -ddd OPEN:/dev/zero TCP:localhost:8003,sndbuf=2000,rcvbuf=2000
> 2009/12/04 17:30:21 socat[2880] E write(4, 0x954a758, 8192): Broken pipe
If the sndbuf and rcvbuf settings correspond to setsockopt() calls for
SO_SNDBUF and SO_RCVBUF respectively, it is important to keep in mind
that Linux, unlike virtually every other *nix I've encountered, very
much considers the setsockopt() call a "suggestion" rather than a
"demand." It will set the actual socket buffer size to something
else, which one can see with a subsequent getsockopt() call (eg what
netperf does). It adds-in space for overhead (overheads of the
buffers that get queued to the socket buffers IIRC)
rick jones
--
firebug n, the idiot who tosses a lit cigarette out his car window
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...
> Does someone could explain?
You are making the naive assumption that the send queue and the
receive queue are set in units of application-level data bytes. They
are not.
DS
perhaps it is out of date, but the netstat manpage on my linux system
has this to say about the Q's:
Recv-Q
The count of bytes not copied by the user program connected
to this socket.
Send-Q
The count of bytes not acknowledged by the remote host.
Which certainly sounds like application-level bytes to me.
rick jones
--
a wide gulf separates "what if" from "if only"
> perhaps it is out of date, but the netstat manpage on my linux system
> has this to say about the Q's:
>
> Recv-Q
> The count of bytes not copied by the user program connected
> to this socket.
>
> Send-Q
> The count of bytes not acknowledged by the remote host.
>
> Which certainly sounds like application-level bytes to me.
I was talking about how the queue sizes are set, not how they are
measured. This description is TCP-specific, but the SO_SNDBUF/
SO_RCVBUF socket options are protocol-neutral.
Think about UDP. Where do you think the source IP and port are stored
if not in the receive queue?
DS
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/socket.h>
int main(void)
{
int j, i;
socklen_t l;
j=socket(AF_INET, SOCK_STREAM, 0);
if(j<0) return -1;
i=16384;
l=sizeof(i);
if(setsockopt(j, SOL_SOCKET, SO_RCVBUF, &i, l)!=0) return -2;
if(getsockopt(j, SOL_SOCKET, SO_RCVBUF, &i, &l)!=0) return -3;
printf("recvbuf=%d bytes\n", i);
close(j);
}
It does not fail, but the output is *NOT* 16,384.
DS
So let's play with ad-hoc lient / server
(see attached files)
$ ./lazyServerTCP 8003
SO_RCVBUF=4000
SO_SNDBUF=4000
$ ./clientTCP localhost 8003
SO_RCVBUF=4000
SO_SNDBUF=4000
80715
$ netstat -tpn
Proto Recv-Q Send-Q Adresse locale Adresse distante Etat
PID/Program name
tcp 65930 0 127.0.0.1:8003 127.0.0.1:47969
ESTABLISHED 16937/lazyServerTCP
tcp 0 14784 127.0.0.1:47969 127.0.0.1:8003
ESTABLISHED 16939/clientTCP
I see that Recv-Q plus Sed-Q equals the amount of user data sent.
However I can't figure out why Recv-Q is 16 times larger than the actual
socket buffer...?
Regards.
--
> However I can't figure out why Recv-Q is 16 times larger than the actual
> socket buffer...?
Why should one have anything to do with the other? You're measuring
two completely different things and wondering why the measurements are
different -- well why shouldn't they be? The receive buffer is
something you at socket level. The Recv-Q is something at TCP level.
The receive queue size may affect how fast the receive buffer can
grow, but it won't directly affect its ultimate maximum size.
DS
Let's play with wireshark. Do a "Follow TCP stream".
The amount of user data sent before the TCP window becomes zero is equal
to Recv-Q
So, I understand the definition of Recv-Q:
The count of bytes not copied by the user program connected to
this socket.
> The receive buffer is
> something you at socket level. The Recv-Q is something at TCP level.
Well, but netstat gives a Recv-Q per socket... (a TCP socket in my case)
> The receive queue size may affect how fast the receive buffer can
> grow, but it won't directly affect its ultimate maximum size.
... sorry, I don't understand what are socket receive/send buffers :-(
This is another level of buffer?
What is in it?
Is it after or before Recv-Q (along data flow within the socket)
Regards
Thank you for your explanations and your patience
Perhaps this is decades of BSD-based precedent interacting with
Linux's desire to be different?
rick jones
--
oxymoron n, Hummer H2 with California Save Our Coasts and Oceans plates
> > perhaps it is out of date, but the netstat manpage on my linux system
> > has this to say about the Q's:
> >
> > ? ?Recv-Q
> > ? ? ? ?The ?count ?of ?bytes ?not copied by the user program connected
> > ? ? ? ?to this socket.
> >
> > ? ?Send-Q
> > ? ? ? ?The count of bytes not acknowledged by the remote host.
> >
> > Which certainly sounds like application-level bytes to me.
> I was talking about how the queue sizes are set, not how they are
> measured. This description is TCP-specific, but the SO_SNDBUF/
> SO_RCVBUF socket options are protocol-neutral.
> Think about UDP. Where do you think the source IP and port are stored
> if not in the receive queue?
Off to the side, with the rest of the meta-data :) However, that may
not be the case under Linux, which relates to how it likes to
effectively double (up to a limit) what one requests in a setsockopt()
call.
rick jones
--
A: Because it fouls the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?
According to "A User's Guide to TCP Windows", there is a direct
relationship between TCP window and SO_RCVBUF
http://www.ncsa.illinois.edu/People/vwelch/net_perf/tcp_windows.html
so... I'm lost...
What are socket buffers?
Regards.
> What are socket buffers?
It's implementation-defined, and you will have to get into the
horribly gory details of the implementation to understand it.
Why do you care? Is this for curiosity? Because if you think you need
to know it to do something at application level, you're doing it
wrong. An application should never try to "teach TCP" its protocol.
DS
Yes, that's it.
I'm just looking for a high-level information
Regards
> > Why do you care? Is this for curiosity?
>
> Yes, that's it.
> I'm just looking for a high-level information
Unfortunately, the high-level information is that the underlying
network protocol provides you an ability to set something it calls a
"send buffer" and something it calls a "receive buffer" in units of
bytes. And that's it.
DS
ok
Thank you for your patience
Regards