Only 282 connections on an HTTP server (long-polling)

[Bruno Michel]

Hi,

I'm trying to make an HTTP server on golang that accepts long-polling
connections. It accepts connections and when it receives an USR1 signal,
it responds to all clients.

The code is on https://gist.github.com/1039785

My problem is that it can't accept more than 282 clients simultaneously.
I think the cause is the VIRT memory: it takes 3G and I'm on a 32-bits
system.

Is there an obvious error on my code? An environment variable that I
should set?

++
Bruno

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I don't know what issues you're running into. I copied your code,
stubbed out the signal handler, and then ran it on Windows (8g). I then
wrote a trivial python script that connects to the port and issues a simple:

GET / HTTP/1.0
Host: localhost

And I'm able to get 500 connections without any problem. (I certainly
don't see the VIRT memory consumption that you're seeing.)

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk4BzoEACgkQJdeBCYSNAAMWkwCdEMDpxOvVdR0jnooTsZEejW2J
gnEAoJH5ZiGi9YOYp50R2w6y56k2MboP
=0YFV
-----END PGP SIGNATURE-----

[Bruno Michel]

Le 22/06/2011 16:10, gary...@gmail.com a écrit :
> After this change is made, you will probably need to change a system
> setting to increase the maximum number of file descriptors that the
> process can open.

Thanks for the tip. I can now keeps 10,000 connexions simultaneously
without problems.

++
Bruno

[André Moraes]

> Thanks for the tip. I can now keeps 10,000 connexions simultaneously without
> problems.

Confirmed!!! Go can solve the 10K problem!!!

:-D

--
André Moraes
http://andredevchannel.blogspot.com/

[Bruno Michel]

Hi,

I wanted to stress my server now that it can handle thousands of
connections, and I wrote a client in Go for that. But I have the same
problem: 3G of VIRT memory because of OS threads (one per connection).

The (simplified) code is on https://gist.github.com/1048668. Is there a
way to make an HTTP connection without using its own thread (or even a
TCP one)?

++
Bruno

[André Moraes]

After

> conn.Do(&req)
on line 25

O should close the response.

[andrey mirtchovski]

There are a couple of things going on in your code -- one is that
you're not closing the connection after you receive your response. The
bigger issue is that, even if you didn't run out of memory, the
runtime will exhibit a bug in this particular case which has to do
with scheduling goroutines -- it tries to open too many file
descriptors before it gives net.netFD a chance to start up the
pollServer caching mechanism. net/fd.go:startServer then gets a "pipe:
too many open files" when it is eventually run and does a nil
dereference, absent any error recovery mechanism. Adding a
runtime.Gosched call right after the "go Connect()" line will give the
runtime a chance to create a pollServer before the number of available
file descriptors is exhausted. You'll have to track down all the
missing error handling from your code next.

The go issue tracker gives me an error 500 at the moment so I can't
create an issue, but the attached code will trigger what I'm
describing against a local http server (copy/pasted from the
http.ListenAndServe documentation).

andrey

[Bruno Michel]

Oh, thanks! It works a lot better with the runtime.Gosched() :)

++
Bruno