TCP connection reuse when doing HTTP

Sun Frank

unread,

Apr 10, 2018, 9:17:24 AM4/10/18

to golang-nuts

Hi guys:

my question is how to reuse underlying TCP connection when doing a lot of frequent short HTTP requests using golang?

I tried things from stackoverflow and other blogs including:

https://stackoverflow.com/questions/17948827/reusing-http-connections-in-golang

https://awmanoj.github.io/tech/2016/12/16/keep-alive-http-requests-in-golang/

they say the key is to close resp.Body, here's my test code(as the answer from stackoverflow):

// init HTTPClient
func init() {
    httpClient = createHTTPClient()
}
const (
    MaxIdleConnections int = 10
    RequestTimeout     int = 5
)
// createHTTPClient for connection re-use
func createHTTPClient() *http.Client {
    client := &http.Client{
        Transport: &http.Transport{
            MaxIdleConnsPerHost: MaxIdleConnections,
        },
        Timeout: time.Duration(RequestTimeout) * time.Second,
    }
    return client
}
func httpClientCode() {
    var endPoint string = "http://localhost:8080/doSomething"
    for i := 0; i < 3; i++ {
        req, err := http.NewRequest("POST", endPoint, bytes.NewBuffer([]byte("Post this data")))
        if err != nil {
            log.Fatalf("Error Occured. %+v", err)
        }
        req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
        response, err := httpClient.Do(req)
        if err != nil && response == nil {
            log.Fatalf("Error sending request to API endpoint. %+v", err)
        } else {
            io.Copy(ioutil.Discard, response.Body)
            response.Body.Close()
        }
    }
}

then I start a http server:

python3 -m http.server 8080

and then i start package capture:

sudo tcpdump -i lo -s 0 -w /tmp/p1.pcap dst port 8080

then run: go run main.go, i still got 3 times of TCP handshake, even after i change MaxIdleConnections to 1

any thoughts? thx

Alex Efros

unread,

Apr 10, 2018, 10:58:04 AM4/10/18

to golang-nuts

Hi!

On Tue, Apr 10, 2018 at 06:14:46AM -0700, Sun Frank wrote:
> my question is how to reuse underlying TCP connection when doing a lot of
> frequent short HTTP requests using golang?

This happens by default: https://play.golang.org/p/XnzQoGqQlno

> they say the key is to close resp.Body

Yes, but any correct code should close all resources.
Not closing resp.Body is just one of ways how bad code may break things
(like reuse of TCP connections).

> then I start a http server:
> python3 -m http.server 8080
> and then i start package capture:
> sudo tcpdump -i lo -s 0 -w /tmp/p1.pcap dst port 8080
> then run: go run main.go, i still got 3 times of TCP handshake, even after
> i change MaxIdleConnections to 1

- MaxIdleConnections doesn't limit amount of connections client may open.
- New connections may open because server closes old ones, not because of
client setup.
- Server may close connections for many reasons, for ex. if it see header
"Connection: close" with protocol HTTP/1.1 or if it see protocol
HTTP/1.0 without "Connection: keep-alive" or a lot of other reasons.

--
WBR, Alex.

Sun Frank

unread,

Apr 10, 2018, 10:20:44 PM4/10/18

to golang-nuts

Hi Alex:

thanks for your answer, really helped me.

As you mentioned, the connection was indeed closed by the server side: in the packages captured, the server side sent the FIN first; seems like that python's HTTP module and nginx all close the conenction on server side

i wrote a simple http server, which do not close the connection, shows the TCP connection was reused:

thx again, Have a nice day!

B.R.

在 2018年4月10日星期二 UTC+8下午10:58:04，Alex Efros写道：

Reply all

Reply to author

Forward