accept4: too many open files;

[l...@pinkfroot.com]

I have the following code which was working fine in test but in live I am getting 500 errors reported to the clients with multiple lines of 

2017/10/17 15:44:52 http: Accept error: accept tcp [::]:9090: accept4: too many open files; retrying in 1s

2017/10/17 15:44:53 http: Accept error: accept tcp [::]:9090: accept4: too many open files; retrying in 1s

2017/10/17 15:44:54 http: Accept error: accept tcp [::]:9090: accept4: too many open files; retrying in 1s

https://play.golang.org/p/Q5snIxXBpX

Now I know I can increase the limits and probably need to but am wondering if I am just prolonging the inevitable here and it will eventually fall down again.

The web server should be quite simple and I can't see how it is leaking file descriptors at all!  Can anyone else see how/why?

[Shawn Milochik]

What version of Go are you using? Have you tried HTTP/2?

The only thing that jumps out at me is that r.Body() is never closed.

[l...@pinkfroot.com]

This is 1.9.1 and I had not thought of switching to http2 although do need to maintain the 1.1 for older clients.

The closing of the body could explain it though which I thought was automatic on a return from the handlers? Do I need to explicitly call that in the handlers??

[Tamás Gulácsi]

Yes.

[l...@pinkfroot.com]

In the end the fix seemed to be implementing some timeouts.  I found a good guide here...

https://blog.cloudflare.com/the-complete-guide-to-golang-net-http-timeouts/

On Tuesday, October 17, 2017 at 10:01:17 PM UTC+1, Tamás Gulácsi wrote:

Yes.

[Eliezer Croitoru]

Hey,

It really depends on couple factors and I am not sure I am following exactly what body are you closing or supposed to close?
Also it depends on the client side of the connection and the keep-alive settings of the connection.
The other factor is the actual load on the system.
If it's a Linux based server you should take a look at the output of ss or netstat to understand if connections are on an ESTABLISHED state more then they are supposed to be or,
that connections are on time_wait or another state.
If connections are on ESTABLISHED mode then one solution might fit but if it's on time_wait or close_wait it's a whole other story.
The basic 1k or 4k FD limit is not a fit for a web server so if you are using one of these upper the limit to 16k first and then see if you are still affected by the issue.
..unless if you have too many connections in ESTABLISHED state and I would encourage you to turn on iptables and use iptstate or conntrack utils to see how many idle connections you are having.
I have used iptables connection tracking more then once to find out that indeed closing idle connections which are maintaining the "keep-alive" are the cause for similar issues.
Have you tried disabling "keep-alive"??

All The Bests,
Eliezer

----
http://ngtech.co.il/lmgtfy/
Linux System Administrator
Mobile: +972-5-28704261
Email: eli...@ngtech.co.il

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mailto:golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Karan Chaudhary]

CMIIW, I doubt if it is explicitly needed to close the request's body in handler,  as it seems,  close of request's body is handled by stdlib upon completion of handler.

Handler,  in net/http,  is called here: https://github.com/golang/go/blob/master/src/net/http/server.go#L1804

Finish Request: https://github.com/golang/go/blob/master/src/net/http/server.go#L1804

Request's body is closed here: https://github.com/golang/go/blob/master/src/net/http/server.go#L1525

Though,  HTTP response's body should be closed explicitly for connection reuse and stuff.