tools/techniques for tracking down "too many open files"

[Sonia Hamilton]

I'm getting errors due to "too many open files":

Couldn't get hostname: open /proc/sys/kernel/hostname: too many open files

What tools/techniques could I use to track this down? I've got deferred closes on the two locations in my program where I open files, so I'm thinking it's in one of the third party libraries I'm using.

Doing a count on the number of file descriptors shows a definite problem:

% while [ 1 ] ; do

ls /proc/9787/fd | wc -l ; sleep 1

done

...

138

157

472

1023

1023

276

180

...

Sonia Hamilton.

[David Symonds]

The files under /proc/PID/fd/ should all look like symlinks to the
real file that's opened. That might provide some clues.

[Yves Junqueira]

Try also:

$ lsof -p <PID>

(your problem is not related to Go so you may wanna try one of the Stack Exchange sites instead)

On Sep 14, 2012 3:40 AM, "David Symonds" <dsym...@golang.org> wrote:

The files under /proc/PID/fd/ should all look like symlinks to the
real file that's opened. That might provide some clues.

--

[Greg Ward]

On 13 September 2012, Sonia Hamilton said:
> I'm getting errors due to "too many open files":
>
> Couldn't get hostname: open /proc/sys/kernel/hostname: too many open files
>
>
> What tools/techniques could I use to track this down? I've
> got deferred closes on the two locations in my program where I open files,
> so I'm thinking it's in one of the third party libraries I'm using.

Code review?

Years of experience with other garbage-collected languages (mainly
Python and Java) has taught me to always, always, always explicitly
release non-memory resources. IOW, if you open a file, you close it.
If you are 100% sure that your own code religiously closes every file
that it opens without relying on GC, then you should probably audit
those third-party libraries.

Alternately, write tiny test programs that just focus on one
third-party library at a time until you have found the culprit... or
convinced yourself that those libraries are all fine. ;-)

Greg

[Dave Cheney]

It might be easier to work from the usual suspects. Sonia, are you
using net/http ? If so, are you always calling request.Body.Close(),
that is a frequent culprit.

> --
>
>

[Steve Phillips]

> It might be easier to work from the usual suspects. Sonia, are you
> using net/http ? If so, are you always calling request.Body.Close(),
> that is a frequent culprit.

If we don't use request.Body specifically, but _do_ use some other
part (say, request.FormFile), then request.Body doesn't need to be
Close()'d, does it? (I've had "too many open files" issues and don't
Close Body unless I read from it...)

[Dave Cheney]

You should always close the body.

http://code.google.com/p/go/source/browse/src/pkg/net/http/client.go?name=release#98

--

[Sonia Hamilton]

Thanks guys for all your answers. @DaveCheney you were close - it was a call to net :)

I (believe) I tracked the problem down to the snmp library I'm using, it was doing a net.DialTimeout without a deferred close; I've submitted a patch.

    conn, err := net.DialTimeout("udp", ...

    if err != nil {

        ...

    }   

    defer conn.Close() // added

Sonia.

[Rémy Oudompheng]

On 2012/9/14 Sonia Hamilton <sonia.s...@gmail.com> wrote:
> Thanks guys for all your answers. @DaveCheney you were close - it was a call
> to net :)

Open files are memory-allocated structures (os.File, for example).
When they are leaking you should be able to see them in a memory
profile and see what function allocated them;

Rémy.

[Sonia Hamilton]

Thanks Remy,

Yes I've been using the profiler for another problem - and I've noticed encode/json seems to have a memory leak (similar to issue 2539 [1] for encoding/gob).

It's the weekend here in Australia, I'm going to write a demo prog later and post it to the list, see if the problem is caused by my usage of encode/json or there is indeed a memory leak.

[1] http://code.google.com/p/go/issues/detail?id=2539&q=json&colspec=ID%20Status%20Stars%20Priority%20Owner%20Reporter%20Summary

Sonia.

On Friday, 14 September 2012 10:57:41 UTC+10, Sonia Hamilton wrote:

[Rémy Oudompheng]

In my experience I have not seem a memory leak in either encoding/json
nor encoding/gob. It is possible that they use up more memory than
sould be needed, but repeating the operation millions of times does
not introduce a memory usage drift.

The memory profile shows where the things were created, not where they 'live'.

So if you have your own leaks, but the things you are leaking come
from a JSON decoding, the memory profile will show where the leaking
objects were decoded, not the place where the leak happens.

Rémy.

2012/9/15, Sonia Hamilton <sonia.s...@gmail.com>:

> --
>
>
>

[Zippoxer]

That says to close response body, not request body.