http.Get() and "connection reset by peer" errors

[SauliusGurklys]

Hello,

the small program below could be run as

./ program -url <someURL>

or

./ program -skipbody -url <someURL>

On my machine, running it without -skipbody parameter, in doGet() function, "ioutl.ReadAll()" is performed, which seems
to cause to keep open the connection (seen with "lsof | grep program") to http server, so every second http.Get() returns error:

Get <someURL>: read tcp <ip>:80: connection reset by peer

In web server log no such error requests are logged, so it seems that  these requests tries to reuse previous connection, which is closed.

If I add -skipbody parameter then no error occurs.

I'm using:
 * web server - nginx
 * OS - ubuntu 12.04
 * go version devel +b279bb7f7c3b Tue Sep 16 19:54:26 2014 -0700 linux/amd64

Am I doing something wrong? Or is it a bug of some net/* package?
Any comments?

package main

import (
    "flag"
    "io/ioutil"
    "log"
    "net/http"
    "time"
)

var (
    skipBody bool
    url      string
)

func init() {
    flag.BoolVar(&skipBody, "skipbody", false, "skip body")
    flag.StringVar(&url, "url", "", "url")
}

func main() {

    flag.Parse()

    if url == "" {
        log.Fatal("Need URL!")
    }

    go func(url string, skipBody bool) {
        i := 0
        for {
            i++
            log.Printf("R #%d", i)
            err := doGet(url, skipBody)
            if err != nil {
                log.Println(err)
            } else {
                log.Println("OK")
            }
            time.Sleep(60 * time.Second)

        }
    }(url, skipBody)

    time.Sleep(10 * time.Minute)
}

func doGet(url string, skipBody bool) (err error) {

    resp, err := http.Get(url)
    if err != nil {
        return
    }
    defer resp.Body.Close()

    if skipBody {
        return
    }

    data, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        return
    }

    log.Printf("len(data) = %d ", len(data))
    time.Sleep(5 * time.Second) // wait a little bit

    return
}

Thanks,
--
Saulius

[James Bardin]

How large is the response body that you're discarding?

Can you log some more information about the repsonse, maybe all the headers? 

Also, if you want to reuse the connection, you need to do something like this in your skipBody block:

    io.Copy(ioutio.Discard, resp.Body)

But if the response is large, it may be more efficient to discard the body, and Dial again.

[James Bardin]

OK, looks like the current http.Transport will reuse a connection even if the body isn't read, and it knows it's buffered the entire response.

My guess is that nginx is accepting an HTTP/1.1 connection, and erroneously closing it. Maybe there's something in the response headers or your config could help shed some light on it.

[SauliusGurklys]

Yes, connection is HTTP/1.1 and Keep-Alive is on.

On my web server keep-alive connections timeout, I guess (as I never changed it), in about 60s.
Because of it every second request, performed within 60s, is reset and http.Get() returns "connection reset by peer" error,
as it seems that request is performed at the moment when server resets the connection.
Seconds earlier or later and http.Get() succeeds without error as either  same connection is reused or new connection
is created as previous timeouts and is closed.

So now I'm just wandering.

If I'm repeatedly use only http.Get() with HTTP/1.1 keep-alive and if time between requests
equals web server's keep-alive timeout then I always will get an error and I can do nothing about it.
And I think http.Get() works unreliably.

--
Saulius

[James Bardin]

On Thu, Sep 18, 2014 at 5:45 AM, SauliusGurklys <s4u...@gmail.com> wrote:

Yes, connection is HTTP/1.1 and Keep-Alive is on.

On my web server keep-alive connections timeout, I guess (as I never changed it), in about 60s.
Because of it every second request, performed within 60s, is reset and http.Get() returns "connection reset by peer" error,
as it seems that request is performed at the moment when server resets the connection.
Seconds earlier or later and http.Get() succeeds without error as either  same connection is reused or new connection
is created as previous timeouts and is closed.

So now I'm just wandering.

If I'm repeatedly use only http.Get() with HTTP/1.1 keep-alive and if time between requests
equals web server's keep-alive timeout then I always will get an error and I can do nothing about it.

if you're consistently racing the server when it's timing out connections, then adjust some of the timing on either end. I have a habit of setting delay values to a prime slightly off the duration I want, to avoid things like this from synchronizing. In large distributed systems, we often add some randomization to that delta too.

 

And I think http.Get() works unreliably.

There's not much the http client can do if the server is closing the connection within a couple milliseconds of starting the request. If you're making requests this infrequently, then don't use keepalive on the client and Dial a new connection each time.

[SauliusGurklys]

James,

Thank you for your comments. Yes, I agree, with current version of go direct Dial would be solution for occasional http.Get() errors.

But I still think, that such errors should be caught earlier and not should reach program's, using http.Get(), code.

--
Saulius

[James Bardin]

On Fri, Sep 19, 2014 at 3:41 AM, SauliusGurklys <s4u...@gmail.com> wrote:

But I still think, that such errors should be caught earlier and not should reach program's, using http.Get(), code.

The client however, has no way of knowing if this is an error caused by a server innocently timing out a connection at the same moment I want to make a request (which would be very rare in most circumstances) in which case it could retry, or more commonly if the connection was purposely dropped by that server or an intermediary.

If I make a request, and get an RST in response, that's a problem I want to know about, and make an informed decision.