http.Client.Do() returning EOF error rarely but predictably

[Daniel Bryan]

I'm running a small monitoring service in Go that runs a series of HTTP requests sequentially every minute, reporting on the status of various services and comparing HTTP responses to what's expected.

We're getting a number of false alarms at the moment, which seem to come down to client.Do() returning an EOF error roughly once every 2000-2500 requests.

This is the essence of my code:

var (
    tr = &http.Transport{
              TLSClientConfig: &tls.Config{
              InsecureSkipVerify: true,
              ServerName: HOST,
            },
        DisableKeepAlives: true,
    }
    result = newCheckResult()
    client = http.Client{Transport: tr}
    req *http.Request
    res *http.Response
)

if req, err = http.NewRequest(method, url, nil); err != nil {
    return nil, err
}

req.Host = HOST
req.Close = true

if res, err = client.Do(req); res != nil {
    defer res.Body.Close()
}

if err != nil {
    return nil, err
}

// Response handling logic here; does not read in the response body

EOF is being returned from that last block there. It's hard to match up precisely, but I'm fairly certain that the Apache servers I'm hitting have logged these requests as a response with code 200.

The roundabout way of creating the requests is to avoid problems with hostname verification, and to ensure that SNI works by setting the ServerName. The failing requests are all actually over plain HTTP, so I don't think the TLS config has any effect.

I've tried digging into the code but it's difficult to see where an EOF would come from in the RoundTripper. I found some old bugs and a StackOverflow thread that suggest bugs or quirks with the connection pooling, which is why I see Close to true and disable keep-alive.

Is there some way to get better visibility over why a request actually failed, rather than just EOF? It's a peculiar error to get as I don't even try to read in the response body.

[Caleb Doxsey]

The docs say:

An error is returned if caused by client policy (such as CheckRedirect), or if there was an HTTP protocol error. A non-2xx response doesn't cause an error.

So EOF shouldn't happen and would seem to imply a protocol error of some sort. Are the servers just plain Apache? Or are they some sort of custom code?

Often useful for debugging HTTP requests is fiddler (http://www.telerik.com/download/fiddler), or the much lower level wireshark (http://www.wireshark.org/).

Just taking a shot in the dark here: but maybe the Content-Length header of the response is wrong?

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[James Bardin]

On Thursday, February 13, 2014 10:05:37 AM UTC-5, Caleb Doxsey wrote:

Just taking a shot in the dark here: but maybe the Content-Length header of the response is wrong?

The content-length isn't wrong, since the go client wouldn't get to that until it tries to read the resp.Body.

You can get an EOF if there is no double newline at the end of the headers though.

[Andy Balholm]

EOF means that the socket was closed before that was expected. It often signals a dropped network connection. When I get it, it usually means that the http.Transport was reusing an old connection, and that connection got closed by the server without the Transport's connection-reuse logic noticing it.

[Daniel Bryan]

I added some instrumentation last night and verified that client.Do() is what's returning EOF.

On Friday, February 14, 2014 3:13:35 AM UTC+11, Andy Balholm wrote:

EOF means that the socket was closed before that was expected. It often signals a dropped network connection. When I get it, it usually means that the http.Transport was reusing an old connection, and that connection got closed by the server without the Transport's connection-reuse logic noticing it.

Interesting. As mentioned, I've done req.Close = True, as well as DisableKeepAlives: True, so it's odd that it'd be reusing a TCP connection. I might try a retry loop with a loop to see what affect that has - though it's an awful solution. 

[James Bardin]

On Thursday, February 13, 2014 5:04:49 PM UTC-5, Daniel Bryan wrote:

Interesting. As mentioned, I've done req.Close = True, as well as DisableKeepAlives: True, so it's odd that it'd be reusing a TCP connection. I might try a retry loop with a loop to see what affect that has - though it's an awful solution. 

If you can instrument it further, I'm willing to bet that you're not getting a complete header from some server, probably missing the blank line to terminate the headers. I know some libraries like curl/python don't report this as an error, but go wants that extra '\r\n' no matter what.

[Dave Cheney]

My money is apache closed a connection which the client has kept in a pool. 

This is a tricky problem to solve, most clients retry gets. 

On 14 Feb 2014, at 3:13, Andy Balholm <andyb...@gmail.com> wrote:

EOF means that the socket was closed before that was expected. It often signals a dropped network connection. When I get it, it usually means that the http.Transport was reusing an old connection, and that connection got closed by the server without the Transport's connection-reuse logic noticing it.

--

[Brad Fitzpatrick]

Except we don't consider it "extra".  ;-)

I haven't seen that in a long time, though. Most servers are pretty compliant.

I would be interested in seeing a packet capture, though.  Or at least an analysis of one, if it contains secrets.