defer f.Close() and "too many open files"

[c...@aditazz.com]

I ran a test loop opening up a small JSON file and closing it with defer f.Close().  At about 240, I started to get a "too many open files" error.  When I changed the code to an explicit f.Close(), I no longer had the problem.  Thoughts?

I had first thought that my code was problematic in another section where I am opening up directories and was not doing a defer dir.Close() (or an explicit dir.Close()), but when I ran a loop to test this section, I never got the "too many open files" error.  Is the error contingent on the size of the file/directory as opposed to just some number of open files/directories?  Does Go automatically/eventually close directories?  I'm confused...

[Matthew Kane]

Can you show code? I would guess that you are doing something like:

for /* stuff */ {

    f, err := os.Open()

    if err != nil { // die

    }

    defer f.Close()

}

Deferred function calls are not called until the function that called defer returns, so what you could here is:

for /* stuff */ {

    func() {

        f, err := os.Open(/*stuff*/)

        if err != nil { // die

        }

        defer f.Close()

    }() // <-- note parens making this a call

}

but the explicit Close is much clearer.

On Tue, Jul 2, 2013 at 2:00 PM, <c...@aditazz.com> wrote:

I ran a test loop opening up a small JSON file and closing it with defer f.Close().  At about 240, I started to get a "too many open files" error.  When I changed the code to an explicit f.Close(), I no longer had the problem.  Thoughts?

I had first thought that my code was problematic in another section where I am opening up directories and was not doing a defer dir.Close() (or an explicit dir.Close()), but when I ran a loop to test this section, I never got the "too many open files" error.  Is the error contingent on the size of the file/directory as opposed to just some number of open files/directories?  Does Go automatically/eventually close directories?  I'm confused...

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
matt kane
twitter: the_real_mkb / nynexrepublic
http://hydrogenproject.com

[Kamil Kisiel]

deferred calls are only executed when the function exits. If you are simply calling defer within a loop you're basically just adding more calls to the stack that will be executed when the function exits. You can solve this problem either like you did by calling f.Close() explicitly, or else moving the body of the loop to its own function.

[Chuck Han]

Yeah, right after posting, I thought a question like that would come up right after I made the post :-)

The loop is actually connecting to a ZMQ server, so the defer code is actually running in a separate process from the loop.  However, in the ZMQ code, I'm doing the following (note where the defer is commented out now), but that raises the question of whether or not f goes out of scope when the case goes out of scope.  It should, right?  Or is it then the function itself ends?  If that's the case, then I have the same problem since the ZMQ server is running an endless loop...

case "foo", "bar":

{

f, err := os.Open(fmt.Sprintf("%s/%s.json", path, strings.Replace(route, "results", "", 1)))

if err != nil {

reply = fmt.Sprintf("{ \"error\": \"%s\" }", err)

} else {

//defer f.Close()

r := bufio.NewReaderSize(f, 4*1024)

line, isPrefix, err := r.ReadLine()

reply = ""

for err == nil && !isPrefix {

reply = fmt.Sprintf("%s %s", reply, string(line))

line, isPrefix, err = r.ReadLine()

}

if isPrefix {

reply = fmt.Sprintf("{ \"error\": \"%s\" }", "buffer too small")

} else if err != io.EOF {

reply = fmt.Sprintf("{ \"error\": \"%s\" }", err)

}

                                                        f.Close()

[Jesse McNelis]

On Wed, Jul 3, 2013 at 4:15 AM, Chuck Han <c...@aditazz.com> wrote:

Yeah, right after posting, I thought a question like that would come up right after I made the post :-)

The loop is actually connecting to a ZMQ server, so the defer code is actually running in a separate process from the loop.  However, in the ZMQ code, I'm doing the following (note where the defer is commented out now), but that raises the question of whether or not f goes out of scope when the case goes out of scope.  It should, right?  Or is it then the function itself ends?  If that's the case, then I have the same problem since the ZMQ server is running an endless loop...

Defer doesn't care about scope, it only cares about functions.

It's not a 'destructor' or 'finalizer'. It only defers a function call until the function it's called in returns.

In your case you either have to close the file at the end of the loop or put the code in the loop in it's own function so that defer is called in a function that can return.

--
=====================
http://jessta.id.au

[Matthew Kane]

On Tue, Jul 2, 2013 at 2:15 PM, Chuck Han <c...@aditazz.com> wrote:

The loop is actually connecting to a ZMQ server, so the defer code is actually running in a separate process from the loop.  However, in the ZMQ code, I'm doing the following (note where the defer is commented out now), but that raises the question of whether or not f goes out of scope when the case goes out of scope.  It should, right?  Or is it then the function itself ends?  If that's the case, then I have the same problem since the ZMQ server is running an endless loop...

Indeed, the function itself must end. 

I would keep the explicit f.Close(). I don't think there is anything in there that can panic() unless your universe is no longer sane (out of memory, something unsafe scribbled all over memory and now f is nil).

[Chuck Han]

Okay, thanks.  Any insight on my second question?

[Jesse McNelis]

On Wed, Jul 3, 2013 at 12:39 PM, Chuck Han <c...@aditazz.com> wrote:

Okay, thanks.  Any insight on my second question?

The os package adds a finalizer to os.File. So eventually the file may be automatically closed.

Finalizers run at some point after an object goes out of scope, but may not run at all.

http://golang.org/pkg/runtime/#SetFinalizer

Don't rely on it.

--
=====================
http://jessta.id.au

[Matthew Kane]

f is no longer in scope after the end of the case clause. The braces are not strictly necessary as they would be in C. (At least according to the playground; the spec is actually a bit fuzzy on this)

e.g. http://play.golang.org/p/uU3P9IxwLu

Note that there is no guarantee that any finalizer will be called at any time before the end of the program.

--

You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Nick Craig-Wood]

On 02/07/13 19:00, c...@aditazz.com wrote:
> I ran a test loop opening up a small JSON file and closing it with defer
> f.Close(). At about 240, I started to get a "too many open files"
> error. When I changed the code to an explicit f.Close(), I no longer
> had the problem. Thoughts?

This is tangential to your problem, but don't use `defer f.Close()` -
you are ignoring errors on Close if you do that! In this
particular case you are only reading files so you'll probably get away
with it, but it is bad practice to ignore errors.

From the linux man pages

> Not checking the return value of close() is a common but
> nevertheless serious programming error. It is quite possible that
> errors on a previous write(2) operation are first reported at the
> final close(). Not checking the return value when closing the file
> may lead to silent loss of data. This can especially be observed
> with NFS and with disk quota.

--
Nick Craig-Wood <ni...@craig-wood.com> -- http://www.craig-wood.com/nick

[Chuck Han]

Okay, thanks.  I for sure won't rely on it--was just curious about the behavior.

Thanks all for your responses!  Chuck