[go-nuts] Server socket queue length

[silvia]

Hi.
I'm writing a server in Go and I would like to know how many
concurrent client requests it can serve. Is there a parameter in order
to set the queue length of the server socket? Or how can I find that
limit?

I've written a program that creates many clients and run each one in a
different thread. The client program attempts to connect to the server
and than sends a request message to it. The server uses the public
function ListenAndServe() of the http package to serve the clients.
Will the server send an error if the are too many incoming connection
requests from the clients, that is more than the queue length (I
imagine an error from the http.Serve() function)? Will it signal that
an incoming connection has been refused?
I have spawned 5000 client but I haven't had any error, either in the
server or in the client (that's good for Go, but not for my tests...).

Thank you for your help

[Russ Cox]

> I'm writing a server in Go and I would like to know how many
> concurrent client requests it can serve. Is there a parameter in order
> to set the queue length of the server socket? Or how can I find that
> limit?

There's no parameter. Go sets it to the value returned by the
listenBacklog function in src/pkg/net/ipsock.go, which defaults
to syscall.SOMAXCONN, the maximum value it can have
(modulo the TODO about reading /proc/sys/net/core/somaxconn).
I believe the standard max value is 128 on Linux.

Note that the limit is on the number of incoming connections
that have arrived but not yet been processed by a call to l.Accept.
It is not a limit on the total number of connections.

The work done by the http server per Accept is minimal - it just
kicks off a new goroutine - so I would imagine that 128 is more
than enough.

> I've written a program that creates many clients and run each one in a
> different thread. The client program attempts to connect to the server
> and than sends a request message to it. The server uses the public
> function ListenAndServe() of the http package to serve the clients.
> Will the server send an error if the are too many incoming connection
> requests from the clients, that is more than the queue length (I
> imagine an error from the http.Serve() function)? Will it signal that
> an incoming connection has been refused?

It will do whatever the accept system call does.
(I'm not sure what that is.)

Russ

[silvia]

Thank you for your clear reply.
I would like to ask you another question.
I've been monitoring my simple Go server and I've noticed that while
it is running it allocates new memory at each client connection and
never releases it. It seems really a memory leak problem.
I don't believe the problem is in my server program since the server
simply registers the http handler function and calls ListenAdnServe();
the handler function calls methods of the Arith type, the one of the
example in rpc package.
Are there any memory leak known problems and how to avoid them.

Thanks
Silvia

[Michael Hoisie]

If you find a memory leak in network-related code please craft a small
example and submit a bug report:
http://code.google.com/p/go/issues

AFAIK, there are no active network-related memory leaks, but there
have been some in the past, such as:
http://code.google.com/p/go/issues/detail?id=631

- Mike

[Russ Cox]

> I've been monitoring my simple Go server and I've noticed that while
> it is running it allocates new memory at each client connection and
> never releases it. It seems really a memory leak problem.
> I don't believe the problem is in my server program since the server
> simply registers the http handler function and calls ListenAdnServe();
> the handler function calls methods of the Arith type, the one of the
> example in rpc package.
> Are there any memory leak known problems and how to avoid them.

What kinds of growth are you seeing?
It's normal for the server to grow at first but
it should stabilize over time. If you're seeing
more than, say, a doubling of memory usage
after you think it should be in the steady state,
that's something we'd want to track down.

Russ

[silvia]

> What kinds of growth are you seeing?
> It's normal for the server to grow at first but
> it should stabilize over time.  If you're seeing
> more than, say, a doubling of memory usage
> after you think it should be in the steady state,
> that's something we'd want to track down.
>
> Russ

I'm running the test under Ububtu 9.10 in a virtual machine.
To see the memory usaage I use "ps au".
When I first run the server I have theese values:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME
COMMAND
silvia 27474 0.0 0.3 3552 1680 pts/1 Sl+ 01:04 0:00
httpserv

Then I execute 100 client connections with 1 request each and I have
got:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME
COMMAND
silvia 27474 0.2 2.1 15872 10816 pts/1 Sl+ 01:04 0:00
httpserv

The numbers grow initially, then the memory usage is steady till the
next client requests. I execute other 100 client connections and I
have:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME
COMMAND
silvia 27474 0.2 2.5 17928 12760 pts/1 Sl+ 01:04 0:00
httpserv

This time after other 1000 client connections:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME
COMMAND
silvia 27474 0.9 14.5 92636 73900 pts/1 Sl+ 01:04 0:04
httpserv

So, while the server is idle (steady state) there's no grow; I thought
that it should had released memory after serving the requests, but the
memory size never decreases.
Is this behavour "normal"?

Silvia

[Russ Cox]

If you have an http server running already, it is easy to

get a memory profile.  Add the line

    import _ "http/pprof"

to your package main, and then run your server and the

requests, so that the memory usage is bigger than you'd like.

Then run

    gopprof --web http://localhost:6060/debug/pprof/heap

which should open an SVG image of a memory profile in

your web browser.  It fetches the profile information from

that URL, which is served by adding the import.

(You may need to change the localhost:6060 to wherever

your servers runs.)

Russ

[silvia]

ok, thanks. I've done what you have said but now I don't know how to
interpret the SVG image.
I see that 95% of the memory is the "malg", while the remaining is
half bufio.NewReaderSize and bufio.NewWriterSize. For instance, 140 MB
malg and 3MB bufio.NewReaderSize and bufio.NewWriterSize each.
What is "malg"?
Can you tell me what should I do further?

Silvia

[Russ Cox]

Malg is the function the allocates a new goroutine.
Perhaps you have goroutines that are deadlocked,
never finish running, and pile up. A destructive way
to see the goroutines in your program is to run

killall -ABRT yourprogram

In the ensuing crash, you'll get a stack trace for
every goroutine in the program. I think you'll find
there are more than you expected.

Russ

[silvia]

> Malg is the function the allocates a new goroutine.
> Perhaps you have goroutines that are deadlocked,
> never finish running, and pile up.  A destructive way
> to see the goroutines in your program is to run
>
> killall -ABRT yourprogram
>
> In the ensuing crash, you'll get a stack trace for
> every goroutine in the program.  I think you'll find
> there are more than you expected.

I've made some tests but I've seen meaningless (for me) results. Some
examples:
a) run my server; run 10 clients. With your profiling instruction I
have 6MB heap (malg 5MB).
killall -> 2 gorountine
b) run my server; run 20 clients. With your profiling instruction I
have 11,5MB heap (malg 10,5MB).
killall -> 1 gorountine
c) run my server; run 200 clients. With your profiling instruction I
have 98MB heap (malg 95MB).
killall -> 1 gorountine
d) run my server; run 1000 clients. With your profiling instruction I
have 243MB heap (malg 234MB).
killall -> 3 gorountine

So, no many goroutines killed but much heap used.

The strange thing is that my server code doesn't execute any goroutine
directly, only by invoking ListenAndServe().
These tests are done with Java clients which use http connections: I
don't know if this could be a problem.

Silvia

[silvia]

> Malg is the function the allocates a new goroutine.
> Perhaps you have goroutines that are deadlocked,
> never finish running, and pile up.  A destructive way
> to see the goroutines in your program is to run
>
> killall -ABRT yourprogram
>
> In the ensuing crash, you'll get a stack trace for
> every goroutine in the program.  I think you'll find
> there are more than you expected.
>
> Russ

I've done more tests, with server and client written in Go. I've run
the same amount of client sessions sequentially and in parallel
executing the same code as a goroutine.
From the profiler output the heap of the server increases sensibly if
I run the clients as goroutines while it is steady if the clients are
run sequentially:
- Sequential client runs
initial server heap: 1.0MB
server heap after 200 client services: 1.5MB (of which malg 1.0MB)
- Parallel client runs
initial server heap: 1.0MB
server heap after 200 client services: 103.5MB (of which malg 100.5MB)

When I kill the server, in the stack trace there are only 2
goroutines:
SIGABRT: abort
Faulting address: 0x1816
PC=0x805367b

futex+0x1f /home/silvia/go/src/pkg/runtime/linux/386/sys.s:108
futex()
futexsleep+0x4f /home/silvia/go/src/pkg/runtime/linux/thread.c:50
futexsleep(0x80e72a8, 0x0, 0x3)
futexlock+0x81 /home/silvia/go/src/pkg/runtime/linux/thread.c:118
futexlock(0x80e72a8, 0x3)
notesleep+0x26 /home/silvia/go/src/pkg/runtime/linux/thread.c:203
notesleep(0x80e72a8, 0x1)
nextgandunlock+0x117 /home/silvia/go/src/pkg/runtime/proc.c:339
nextgandunlock()
scheduler+0xdd /home/silvia/go/src/pkg/runtime/proc.c:492
scheduler()
mstart+0x66 /home/silvia/go/src/pkg/runtime/proc.c:385
mstart()
_rt0_386+0x88 /home/silvia/go/src/pkg/runtime/386/asm.s:75
_rt0_386()

goroutine 29 [4]:
gosched+0x66 /home/silvia/go/src/pkg/runtime/proc.c:513
gosched()
runfinq+0x3e /home/silvia/go/src/pkg/runtime/mgc0.c:344
runfinq()
goexit /home/silvia/go/src/pkg/runtime/proc.c:145
goexit()

goroutine 2 [3]:
runtime.entersyscall+0x5d /home/silvia/go/src/pkg/runtime/proc.c:534
runtime.entersyscall()
syscall.Syscall6+0x5 /home/silvia/go/src/pkg/syscall/asm_linux_386.s:
40
syscall.Syscall6()
syscall.EpollWait+0x71 /home/silvia/go/src/pkg/syscall/
zsyscall_linux_386.go:132
syscall.EpollWait(0x100, 0x6, 0x8d970590, 0x1, 0xffffffff, ...)
net.*pollster·WaitFD+0x111 /home/silvia/go/src/pkg/net/fd_linux.go:116
net.*pollster·WaitFD(0x6, 0x8d970590, 0x1, 0x1, 0xffffffff, ...)
net.*pollServer·Run+0xc9 /home/silvia/go/src/pkg/net/fd.go:232
net.*pollServer·Run(0x110990, 0x0)
goexit /home/silvia/go/src/pkg/runtime/proc.c:145
goexit()

goroutine 1 [4]:
gosched+0x66 /home/silvia/go/src/pkg/runtime/proc.c:513
gosched()
chanrecv+0x15c /home/silvia/go/src/pkg/runtime/chan.c:362
chanrecv(0x115d70, 0x15ca80, 0x1, 0x807d11b)
runtime.chanrecv1+0x37 /home/silvia/go/src/pkg/runtime/chan.c:435
runtime.chanrecv1(0x115d40, 0x11bd50)
net.*pollServer·WaitRead+0x4d /home/silvia/go/src/pkg/net/fd.go:272
net.*pollServer·WaitRead(0x115d40, 0x118f00, 0x0)
net.*netFD·accept+0x2b3 /home/silvia/go/src/pkg/net/fd.go:508
net.*netFD·accept(0x15c900, 0x118f00, 0x0, 0x0, 0xb, ...)
net.*TCPListener·AcceptTCP+0x53 /home/silvia/go/src/pkg/net/tcpsock.go:
253
net.*TCPListener·AcceptTCP(0x118f00, 0x8084076, 0x8, 0x0)
net.*TCPListener·Accept+0x36 /home/silvia/go/src/pkg/net/tcpsock.go:
263
net.*TCPListener·Accept(0x1109e8, 0x11bee0, 0x0, 0x0,
0x2026e158, ...)
http.Serve+0x6b /home/silvia/go/src/pkg/http/server.go:551
http.Serve(0x1109e8, 0x0, 0x0, 0x0, 0x0, ...)
http.ListenAndServe+0x84 /home/silvia/go/src/pkg/http/server.go:595
http.ListenAndServe(0x15ca20, 0x1109e8, 0x15ca40, 0x110828,
0x15ca20, ...)
main.main+0x105 /home/silvia/Desktop/go/src/cs/http-test/server/
httpserv.go:273
main.main()
mainstart+0xf /home/silvia/go/src/pkg/runtime/386/asm.s:83
mainstart()
goexit /home/silvia/go/src/pkg/runtime/proc.c:145
goexit()
eax 0xfffffffc
ebx 0x80e72a8
ecx 0x0
edx 0x3
edi 0x0
esi 0x80d7d30
ebp 0x0
esp 0xbf879f78
eip 0x805367b
eflags 0x292
cs 0x73
fs 0x0
g536)

I think this is a memory leak problem.
Is there something else I can do to solve it?

Silvia