http: Accept error: accept tcp 0.0.0.0:8080: too many open files

[Everson Alves da Silva]

Hello,

I tried running a simple go server for static serving. It's currently running on nginx, serving 30gb of static files each day. After a few minutes I started getting:

http: Accept error: accept tcp 0.0.0.0:8080: too many open files.

server code is:

func main() {

err := http.ListenAndServe(":8080", http.FileServer("/static_root", "/"))

if err != nil {

log.Printf("error running docs webserver: %v", err)

}

}

Can I successfully use go http.ListenAndServe for serving a huge amount of static files? What should I do to make it work?

[Brian Ketelsen]

You should look at ulimit. Here's mine:

Chapterhouse:Projects bketelsen$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 256
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 709
virtual memory (kbytes, -v) unlimited

I'm on a mac - which defaults to 256 open files max. If I don't change that, I can't do any useful benchmarks. Search for how to change this open file limitation on your operating system. Nginx is probably caching/reusing it's file handles. I'd go read the source code, but it's in very confusing C and I'm allergic.

Brian

[Everson Alves da Silva]

The actual limit on the server is 1048575, a pretty big number. That number was reached after a few minutes. I guess just increasing it won't solve the problem. Is there any go server that takes care of that?

[Everson Alves da Silva]

@mtoader suggested it might be a leak.  I am serving 500k files per day. I should not hit 1M limit in about 20 minutes.

[Everson Alves da Silva]

So, I updated go and ran again. looks like the problem was fixed. Running now for a few hours without problem. 

[Brad Fitzpatrick]

Is there a reason you have nginx is front of Go?  Go does great on its own.

(but yeah, like other people said: increase your ulimit)

[Everson Alves da Silva]

Hello, I am not mixing GO with nginx. Current service is using nginx to serve static files. I am switching to use Go instead. I have no complaints with nginx, but I am want to see the power of Go in a real scenario. So far good, after the update.

[Everson Alves da Silva]

Hello,

Now the problem is a little different, I am using this go server the for 2 days, the memory usage starts very low ( under 20mb) and then keeping increasing, even when the site traffic goes low, it still keeps increasing slowly. Maybe there's some leak or http.ListenAndServe is keeping some cache that never expires.

I'll switch back to use nginx but will be happy to test again if someone fix something on the code or suggest a better approach to serve static files using go.

[André Moraes]

On Sat, Jul 16, 2011 at 5:20 PM, Brad Fitzpatrick <brad...@golang.org> wrote:
> Is there a reason you have nginx is front of Go?  Go does great on its own.
> (but yeah, like other people said: increase your ulimit)

I had Go webserver as the web-frontend and I am currently proxing my
backend via Go.

Nothing to complain yet.

--
André Moraes
http://andredevchannel.blogspot.com/