File descriptors are leaking on web server

[Max]

I have website written in go and running on cloud server with 512mb for around 1 year.

10-30 people per day visit that site.

There is default limit -n on number of files (1024)

Every 2-4 months server (Go app) is going down. It can't open new file as it opened 1024 files already.

I think the problem is timeout

http://golang.org/src/pkg/net/http/server.go#L1488

I do not set it. So I guess it is 0 by default and go does not close certain percent of connections.

Please, confirm if it is correct.

When I test it in office connections usually are closed but in real life much more connections got hang

and it is default settings for http.listenAndServe(host, nil)

[Brad Fitzpatrick]

Yes, you should set the HTTP server's read and write timeouts.

There is currently no default value of these, so they'll last forever, even when you don't necessarily want them to.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Ibrahim M. Ghazal]

On Mon, Jul 1, 2013 at 6:15 PM, Brad Fitzpatrick <brad...@golang.org> wrote:
> Yes, you should set the HTTP server's read and write timeouts.
>
> There is currently no default value of these, so they'll last forever, even
> when you don't necessarily want them to.
>

Does that mean Go servers by default are vulnerable to DoS attacks
similar to Slowloris?

[Brad Fitzpatrick]

Not in the same way. The cost for Go (a goroutine reading occasional slow bytes) is much, much cheaper than a server with a thread or process per request.

[Max]

If we talk about default configuration then 1024 connections could be consumed in 1-3 minutes by another go app that DOSing Go web server. Load on server is 0 but server is not responding.

In my case nobody is DOSing. Certain percent of connections are opened by regular clients and kept opened on server side for (ever) months.

In 2-4 month server is down as 1024 (all) connections stay opened for months by clients who left site.

It repeated several times. I had that servers running since summer 2012.

[Rodrigo Kochenburger]

I'm curious, do you know what causes the server to keep the conn open? Is it because the client ask for keep-alive and does not close it or you're not closing the request body?

[Brad Fitzpatrick]

You're no longer talking about Slowloris.  (as in: spoon-feeding bytes slowly to defeat naive timeouts)

Now you're just talking about somebody closing their laptop in a coffee shop.

Without any HTTP or TCP timeouts, connections lives forever.

--

[Max]

Do handler should close request body?

http://golang.org/doc/articles/wiki/

"close" word is not mentioned in that article

I believed request is closed when handler ends.

I think the problem is "timeout == 0" If client on other side disappear and does not send disconnect packet then connection is left hang.

[Brad Fitzpatrick]

It's not necessary.  The HTTP server knows when the handler is done.

--

[Chao Liu]

Setting ReadTime kills non-idle connections. Is that expected?

[Chao Liu]

With ReadTimeout or WriteTimeout set, a non-idle connection will be closed too.
This means that go server cannot serve long-running tasks like downloading big files, streaming media. 

Is this by design?

On Tuesday, July 2, 2013 at 7:26:26 AM UTC-7, bradfitz wrote:

[Brad Fitzpatrick]

I don't understand.

For more options, visit https://groups.google.com/d/optout.

[James Bardin]

On Thursday, May 21, 2015 at 11:49:05 AM UTC-4, Chao Liu wrote:

With ReadTimeout or WriteTimeout set, a non-idle connection will be closed too.
This means that go server cannot serve long-running tasks like downloading big files, streaming media. 

Is this by design?

Yes.  ReadTimeout and WriteTimeout are hard deadlines, and close the connection after that amount of time.

If you want something that closes a connection only after a specific amount of idle time, you need to implement that with a custom net.Listener and net.Conn.

[Nick Craig-Wood]

Or you could use github.com/mreiferson/go-httpclient which does exactly
that. I've used it in several projects and it works well.

(See ReadWriteTimeout)

--
Nick Craig-Wood <ni...@craig-wood.com> -- http://www.craig-wood.com/nick

[roger peppe]

On 22 May 2015 at 13:56, Nick Craig-Wood <ni...@craig-wood.com> wrote:
> On 21/05/15 18:11, James Bardin wrote:
>> On Thursday, May 21, 2015 at 11:49:05 AM UTC-4, Chao Liu wrote:
>>
>> With ReadTimeout or WriteTimeout set, a non-idle connection will be
>> closed too.
>> This means that go server cannot serve long-running tasks like
>> downloading big files, streaming media.
>> Is this by design?
>>
>> Yes. ReadTimeout and WriteTimeout are hard deadlines, and close the
>> connection after that amount of time.
>>
>> If you want something that closes a connection only after a specific
>> amount of idle time, you need to implement that with a custom
>> net.Listener and net.Conn.
>
> Or you could use github.com/mreiferson/go-httpclient which does exactly
> that. I've used it in several projects and it works well.

What does that do that the stdlib doesn't? It looks like it
was written for go1.1, which I suspect didn't directly
support timeouts.

[James Bardin]

On Fri, May 22, 2015 at 9:03 AM, roger peppe <rogp...@gmail.com> wrote:

What does that do that the stdlib doesn't? It looks like it
was written for go1.1, which I suspect didn't directly
support timeouts.

Not much anymore. The only thing left in go-httpclient that ins't available by default is the ReadWriteTimeout, which is the "idle" timeout. It's also missing DialTLS and TLSHandshakeTimeout in the transport from newer releases. 

This is all moot though, since the question was about ReadTimeout and WriteTImeout in the http.Server. It can be implemented in the same manner as it is in that client, you just need to wrap it in a net.Listener to give to the server.