Is it possible to determine total number of bytes read/written for an HTTP request? (including headers)

[aro...@gmail.com]

I'm interested in tracking total bandwidth on a per-request basis.  It's easy to intercept the ResponseWriter and track the number of bytes sent out for the response body, and it's easy to intercept the Request.Body to count the number of bytes read from the remote source, but both of these ignore the request framing and the headers.

I realize that I could also make my own net.Listener and wrap the net.Conn to count the number of bytes read/written there, but how do I associate that with a given request?  Especially with HTTP/2 long-lived connections, a net.Conn may be used for a number of interleaved requests.

In particular, I'm interested in doing bandwidth accounting for multiple hosted services, so I want accurate per-service usage numbers.

- Augusto

[aro...@gmail.com]

Huh.  Is this actually not possible? (That is, without modifying net/http directly?)

[simon place]

i think there should be several tools to do this tracking by using kernel functions, probably X-platform.

maybe in the design of the go package this was assumed to be the way its done.

[aro...@gmail.com]

Interesting.  I wouldn't expect kernel functions to give me per-request statistics, however, I'd expect per-process (or if I'm lucky per-thread) statistics.  Is that what you were imagining?

[simon place]

i was imagining the os/kernel (network stack) is controlling the routing of data, so knows the process handling the port, how many bytes go through to it and the ip it came from, plus the same for the response.

i had a play with 'iptraf-ng' ( the older 'iptraf' only worked for local addresses for me) and then was watching bytes going to/from a simple go server i knocked up, with local browsers, remote browsers, remote servers. plus all sorts of other background stuff going on, like requests/responses from a time server, dropbox, and lots of things i needed to google.

[Kevin Malachowski]

Given TCP connection reuse I don't think the os/kernel will be able to tell apart the TCP connections any better than intercepting the net.Conn with a custom listener. Especially if you're using anything encrypted, staying in Go is going to be necessary for you to get the sort of data you need.

You might be able to use your wrapped Listener strategy if you are willing to set "Connection: Close" on your incoming connections. I suspect that will degrade performance though. You might be able to hack together a Listener which is actually able to pool http client connections yet convince the http library to Close them when it is done with each request, but that sounds hard to do well and would likely to tightly coupled to the http package's implementation.

Do you need an exact count? You could probably get a pretty good estimation by just serializing the HTTP headers, counting how long that is, and adding in the size of the Body.

[aro...@gmail.com]

Thanks!

I'm not willing close the connection on every request -- that's too painful.

I don't need an _exact_ count, but I want to be confident that my counts are really close.  Serializing the headers and adding the size of the body is the closest that I've come up with to do accurate per-request statistics.

Another idea I had is to have a custom net.Listener that outputs wrapped net.Conn implementations that will count the number of bytes read/written to it.  It would also track the list of active net.Conns.  I think it might be possible to associate the net.Conn with the request via the local address using http.LocalAddrContextKey. Then I could count just the number of bytes per connection and require that each connection is associated with a given service.  I think that would give exactly accurate per-connection bandwidth, but it would not give per-request bandwidth at all.

- Augusto

[Михаил Стойков]

I'm doing similar thing so that a connection can be throttled including to change the throttle in the middle of the request. This can be reused to count outgoing bytes (it was actually a feature that I thought on when designing it, I can't seem to find an implementation of it at the time).

Unfortunately my solution works only for http1/* . H2 on the other hand will be much more complex to support due to multiplexing and the fact that some frames are not for any request but for the whole connection. Counting outgoing bytes is still possible per connection, as you said yourself.

The way I did it in nedomi is:

- implement net.Listener so that I can wrap the new connection in net.Conn implementation (this one puts deadlines on each write/read and also is able to throttle the writer trough a custom throttle writer) the implementation of ReadFrom is because we needed sendfile compatibility - you can take a look at this commit for more info

- add connstate listener to the server so that you know when a request is closed so that you don't mix reuses of same connection https://github.com/ironsmile/nedomi/blob/01acf83d0d615b4f8726e94edf4470985ee176de/app/app.go#L136
- have a place where to store opened connections (the race condition in Size is optional :P ) https://github.com/ironsmile/nedomi/blob/master/app/connections.go

- get the connection out of connection store in the main handler and put in the context (line 54) https://github.com/ironsmile/nedomi/blob/01acf83d0d615b4f8726e94edf4470985ee176de/app/net.go#L46
- Get the connection out of the context in a given later handler and set throttle https://github.com/ironsmile/nedomi/blob/01acf83d0d615b4f8726e94edf4470985ee176de/handler/throttle/throttle.go#L39

You can of course just get the bytes that were written/read when ConnState changes to Inactive and not have the ability to do from inside a handler.

Depending on the way your app works you can probably block in the first handler if there is already a request in progress on a given h2 connection in order to be able to differentiate between request (read the doc for ConnState - it won't go to inactive if the connection has an http request in progress so you will need to flush responses to be sure everything was sent before giving the next request a go ahead).
I have came up with the hacks above nearly two years ago and there are probably better ways now. I would argue to go look at the implementation in go and try to figure out a way given it - it is the way I have did :).

~MStoykov