Go application hanging and throwing Proxy error: Request Canceled or I/O timeout

[dkbala...@gmail.com]

I am currently facing an issue in my load environment during the load testing. The description of the issue is given below,

We have a Jersey REST application which is being guarded by a GO application. GO app validates the browser cookie and creates a single host reverse proxy and serve the request if its a valid one, Initially we got the below error 

http: proxy error: dial tcp HOSTNAME:PORT can't assign requested address

Even after upgrading it to 1.6, the same issue is happening but GO App log shows the below error
"http: proxy error: net/http: request canceled"

I've enabled the DisableKeepAlives and performed the load testing and the same issue is happening, Now Go app shows the below errors.

http: proxy error: dial tcp HOSTNAME:PORT: i/o timeout
http: proxy error: net/http: request canceled

Code snippet for creating proxy is given below

var proxy = httputil.NewSingleHostReverseProxy(proxy_url)

http.DefaultTransport.(*http.Transport).DisableKeepAlives = true

We use proxy.ServeHttp after we validate the request.  Normally the above proxy error is not occurring suddenly but it occurs after 2 or 3 hours after which load testing is started

After disabling keep alive connection, we see lots of TCP connecion in TIMED_WAIT state but its not getting closed but the application is hanging and see the proxy error in the log.

So we have to either call https://golang.org/pkg/net/http/#Transport.CloseIdleConnections or increase MaxIdleConnsPerHost to some higher value to fix this issue.

The same test cases were working fine in the past but its failing now suddenly.

Appreciate your help.

[Shawn Milochik]

I don't have any insight as to why it's suddenly happening when you change Go versions, but I've run into the same problem previously, and re-using HTTP clients and setting MaxIdleConnsPerHost completely solved it for me.

https://paperairoplane.net/?p=556

[Dave Cheney]

Your application has probably run out of file descriptors. 

On linux you can get a sense of the number of active file descriptors your program is using by looking in the /proc/$PID/fd directory. Where $PID is the process id of your program.

/proc/$PID/limits will tell you the current limits for that process.

[dkbala...@gmail.com]

Thanks Shawn. Increasing the maxidleconnectionperhost is my next option. Will there be any issue due to increasing this?

[Shawn Milochik]

On Tue, Jun 28, 2016 at 9:10 AM, <dkbala...@gmail.com> wrote:

Thanks Shawn. Increasing the maxidleconnectionperhost is my next option. Will there be any issue due to increasing this?

I haven't had any issues with running out of file handles or "can't assign requested address" messages after making that change. That was mid-February, so I'm going to say no issues.

[dkbala...@gmail.com]

Thanks everyone for your valuable suggestions. I tried all the solutions but nothing worked out. Then I reviewed the dependent system changes and got to know that their recent changes caused all these issues. So its nothing to do with the Go application.

Apologies for the confusion.