How to decrease contention in re-using connections in http.Client

[Suraj Narkhede]

As it is already raised as issue here - https://code.google.com/p/go/issues/detail?id=6785, I am trying to get opinions from folks on how to reduce the contention till issue is fixed - 

When there is an attempt to reuse connections at higher rate, there is contention in GetIdleCon and PutIdleCon - 

316 @ 0x415989 0x415a0b 0x429eee 0x42a130 0x49ad56 0x45056e 0x450ab8 0x44ef26 0x43390d 0x4330db 0x43456f 0x433bbf 0x400e4f 0x415c20
#	0x42a130	sync.runtime_Semacquire+0x30			/usr/local/go/src/pkg/runtime/sema.goc:199
#	0x49ad56	sync.(*Mutex).Lock+0xd6				/usr/local/go/src/pkg/sync/mutex.go:66
#	0x45056e	net/http.(*Transport).getIdleConn+0x9e		/usr/local/go/src/pkg/net/http/transport.go:402
#	0x450ab8	net/http.(*Transport).getConn+0xa8		/usr/local/go/src/pkg/net/http/transport.go:452
#	0x44ef26	net/http.(*Transport).RoundTrip+0x416		/usr/local/go/src/pkg/net/http/transport.go:201
#	0x43390d	net/http.send+0x43d				/usr/local/go/src/pkg/net/http/client.go:195
#	0x4330db	net/http.(*Client).send+0x15b			/usr/local/go/src/pkg/net/http/client.go:118
#	0x43456f	net/http.(*Client).doFollowingRedirects+0x97f	/usr/local/go/src/pkg/net/http/client.go:343
#	0x433bbf	net/http.(*Client).Get+0xaf			/usr/local/go/src/pkg/net/http/client.go:275
#	0x400e4f	main.startClient+0x16f				

347 @ 0x415989 0x415a0b 0x429eee 0x42a130 0x49ad56 0x44fd5e 0x455b87 0x454c1b 0x454931 0x483291 0x480f6d 0x400f0b 0x415c20
#	0x42a130	sync.runtime_Semacquire+0x30		/usr/local/go/src/pkg/runtime/sema.goc:199
#	0x49ad56	sync.(*Mutex).Lock+0xd6			/usr/local/go/src/pkg/sync/mutex.go:66
#	0x44fd5e	net/http.(*Transport).putIdleConn+0xce	/usr/local/go/src/pkg/net/http/transport.go:342
#	0x455b87	net/http.func·023+0x97			/usr/local/go/src/pkg/net/http/transport.go:853
#	0x454c1b	net/http.(*bodyEOFSignal).condfn+0xab	/usr/local/go/src/pkg/net/http/transport.go:1161
#	0x454931	net/http.(*bodyEOFSignal).Read+0x321	/usr/local/go/src/pkg/net/http/transport.go:1133
#	0x483291	io/ioutil.devNull.ReadFrom+0xb1		/usr/local/go/src/pkg/io/ioutil/ioutil.go:151
#	0x480f6d	io.Copy+0x13d				/usr/local/go/src/pkg/io/io.go:349
#	0x400f0b	main.startClient+0x22b			

This results in -

1. New connections being made and teared down very fast and thus exceeding the ports available.

2. System resources are not used optimal. For a simple hello world server, for loop clients using http.Client gives 112k rps with 64% CPU. Network is not bottleneck here. Because even with 1k response and 100ms response time from server I still get 82k rps with this client at high concurrency. 

Code : http://play.golang.org/p/MA-a2ZAVNe . N go-routines are created for startClient in main, where n is the concurrency desired by user. I used client (*http.Client) object per go-routine instead of global, as I thought, contention will be across one client object. But is it not the case?

Can someone please suggest alternatives/options to avoid contention in putting and getting connection? 

[Dmitry Vyukov]

You also need to create Transport per goroutine, because by default
all Client's use the single DefaultTransport.

Have you tried tuning Transport.MaxIdleConnsPerHost? Try to use a
single transport but set MaxIdleConnsPerHost to number of goroutines.

[Suraj Narkhede]

Thanks Dmitry. Just changed the code to do it with transport per go-routine and throughput increased by 33% now. :). I am still working on it and aim to reach throughput of 300k which I get using other clients. Currently at 187k.

MaxIdleConnsPerHost i had set to 10000, but it did not help much due to contention.

Will keep it posted here.