TCP socket Buffer model

[ourtidbits]

I have a TCP Server written in node. It accept socket connections, gets requests and send response. I use standard socket events like 'data' (to read from socket), 'error', 'end', 'drain' and use socket.write() to write back the responses to the socket. I have a requirement to provide exact count of bytes actually written on the socket (i.e. sent out on the wire). For that I use the following at the socket 'end' event.

socket.bytesWritten - socket.bufferSize 

bufferSize is subtracted to account for any bytes that could not be sent out before the connection was broken for any reason.

Issue: In some scenarios, wireshark on the server box as well as on the client records lot less number of bytes written than what is being calculated using (socket.BytesWritten - socket.bufferSize) - in the order of few 100Ks. 

So question are 

1. Where all in the node stack can the data be buffered up before it gets successfully written out on the socket? 

2. Is there a max size of buffers at each of this layer and are there ways to modify them without having to recompile node?

3. Any pointers on where to look at to determine the discrepancy in the data sent value calculated in the node app to what is being shown in wireshark logs. 

Any insight is appreciated.

Thanks!

[Jimb Esser]

As for the discrepancy, Node only knows when it has been written to the socket handle (passed on to the OS), which usually has it's own buffering in the OS.

Wireshark, I'd assume, only sees packets that made it through the OS buffers and have been written to the wire.

The discrepancy is when data has made it to the OS but does not make it to the wire, this would happen if the connection was forcibly terminated on the other end.

Depending on how low-level Wireshark is, it may also be counting re-sends (when a TCP packet is lost, the packet will get re-sent, causing additional bytes on the wire that your Node app would have no possible way of knowing about (unless you use a module that implements TCP in Node, there might be one of those...).

I don't have any suggestions about how to resolve this discrepancy, sorry!

If I wanted to know exactly how many bytes were being sent, I'd probably use OS-specific functionality (e.g. shell out to ifconfig in Linux) to simply poll it, but that wouldn't get any per-connection stats.