Too many open files / broken pipe issue.

[Juhee LR]

Hello , 

I am currently working for a company and one of our site is having little trouble. It is built in golang . I am facing this particular issue for over 3 months now. No matter what solution i've tried so far , it ends up failing. So the site is working fine , but somewhere due to the code there is some kind of leak which is causing open files not being closed and due to that server crashes and the site goes down until IT restarts. So i've so far tried to deal with "defer" and removed all the defer in code without return type. There is nothing we have not closed. Also increased server ulimit for soft and hard files . Still randomly site breaks down with "too many open file" error or "broken pipe" . 

Here is the go env :

GO111MODULE=""

GOARCH="amd64"

GOBIN=""

GOCACHE="/Users/juhi/Library/Caches/go-build"

GOENV="/Users/juhi/Library/Application Support/go/env"

GOEXE=""

GOFLAGS=""

GOHOSTARCH="amd64"

GOHOSTOS="darwin"

GONOPROXY=""

GONOSUMDB=""

GOOS="darwin"

GOPATH="/Users/juhi/go/"

GOPRIVATE=""

GOPROXY="https://proxy.golang.org,direct"

GOROOT="/usr/local/go"

GOSUMDB="sum.golang.org"

GOTMPDIR=""

GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"

GCCGO="gccgo"

AR="ar"

CC="clang"

CXX="clang++"

CGO_ENABLED="1"

GOMOD=""

CGO_CFLAGS="-g -O2"

CGO_CPPFLAGS=""

CGO_CXXFLAGS="-g -O2"

CGO_FFLAGS="-g -O2"

CGO_LDFLAGS="-g -O2"

PKG_CONFIG="pkg-config"

GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/z9/hr670btx0_gdq81bgc3q42w40000gn/T/go-build387581625=/tmp/go-build -gno-record-gcc-switches -fno-common"

P.S have also attached screenshot of the logs and the go version is : go version go1.13.4 darwin/amd64

and I have also tried putting "time-stamps" but that generated more error so we removed that. 

[Steven Hartland]

Few things which may help:
1. Use pprof to look at goroutines, see if you have a leak there.
2. What does linux utility lsof say?

Don't forget that each network connection uses a "file" so it may not be a real file hand instead a network socket, which given your screen shot could be due to network disconnects.

    Regards
    Steve

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/adf07d15-e8bd-4184-b717-2b73d04cf6b0%40googlegroups.com.

[Uli Kunitz]

Please read the section about timeouts in  https://blog.cloudflare.com/exposing-go-on-the-internet/ very carefully. This should explain your file descriptor issues. The broken pipe and connection resets are probably clients that closed connections because of timeouts on their side, which may a hint that you have performance issues on the server. You should load test your development server before rollout to production. Apache JMeter or Selenium that are popular with Java developers. Go has Vegeta, but I have never used it.