How GOMAXPROCS is used by current 8g implementation?

[ziutek]

I use http package for my production Web application (raw http
package, without web.go or other wrapper). I am very happy with my new
Go implementation of this application. My experience shows that Go (on
Linux) is definitely ready for production use, certainly not less than
CPython which I use a lot.

I use 8g for compilation and I run my application like this:

GOMAXPROCS=2 nohup komunikaty $PARAMS </dev/null >$LOG 2>&1 &

A few days later I run ps -eL and I get:

PID LWP TTY TIME CMD
10189 10189 ? 00:00:11 komunikaty
10189 10190 ? 00:00:00 komunikaty
10189 10191 ? 00:00:00 komunikaty
10189 13747 ? 00:00:03 komunikaty
10189 15398 ? 00:00:01 komunikaty
10189 15400 ? 00:00:00 komunikaty
10189 29227 ? 00:00:03 komunikaty
10189 29793 ? 00:00:00 komunikaty
10189 29794 ? 00:00:02 komunikaty
10189 2426 ? 00:00:03 komunikaty
10189 2434 ? 00:00:00 komunikaty
10189 2460 ? 00:00:01 komunikaty
10189 23877 ? 00:00:04 komunikaty
10189 23880 ? 00:00:00 komunikaty
10189 26207 ? 00:00:01 komunikaty
10189 26208 ? 00:00:03 komunikaty
10189 26209 ? 00:00:00 komunikaty
10189 31882 ? 00:00:00 komunikaty
10189 31883 ? 00:00:02 komunikaty
10189 8119 ? 00:00:00 komunikaty
10189 8124 ? 00:00:00 komunikaty
10189 8127 ? 00:00:00 komunikaty
10189 8605 ? 00:00:00 komunikaty
10189 8607 ? 00:00:04 komunikaty
10189 8625 ? 00:00:00 komunikaty
10189 9027 ? 00:00:01 komunikaty
10189 9028 ? 00:00:00 komunikaty
10189 9029 ? 00:00:00 komunikaty
10189 9030 ? 00:00:00 komunikaty
10189 9031 ? 00:00:00 komunikaty
10189 9040 ? 00:00:05 komunikaty
10189 14426 ? 00:00:04 komunikaty
10189 14427 ? 00:00:00 komunikaty
10189 14428 ? 00:00:02 komunikaty
10189 14429 ? 00:00:00 komunikaty
10189 14430 ? 00:00:00 komunikaty

There are 36 threads. But I specify GOMAXPROCS=2. I don't use
gorutines explicitly in my code but the http and net packages uses
them a lot.

Isn't GOMAXPROCS a thread limit but only limit of threads working
simultaneously?

I know that internal scheduler (in 6g/8g implementation) schedules
gorutines on limited number of system specific threads for performance
reasons and to conserve system resources. So, how many gorutines had
to work at the same time to create 36 system threads on Linux?

--
Michał Derkacz
https://github.com/ziutek

[Jessta]

On Wed, Feb 2, 2011 at 10:31 PM, ziutek <ziu...@lnet.pl> wrote:
> There are 36 threads. But I specify GOMAXPROCS=2. I don't use
> gorutines explicitly in my code but the http and net packages uses
> them a lot.
>
> Isn't GOMAXPROCS a thread limit but only limit of threads working
> simultaneously?

It's only a limit on the number of running threads. The Go runtime
will spawn additional threads if a thread is blocking on some external
call.
A call out to C code will block a thread and any blocking syscalls will too.
Without seeing your code I can't really be sure what is triggering the
creation of these additional threads.
A lot of the current binds for databases are just wrappers around a C
library, so this might be your issue.

> I know that internal scheduler (in 6g/8g implementation) schedules
> gorutines on limited number of system specific threads for performance
> reasons and to conserve system resources. So, how many gorutines had
> to work at the same time to create 36 system threads on Linux?
>

Not a lot if they all have to block.

- jessta

--
=====================
http://jessta.id.au

[ziutek]

> A lot of the current binds for databases are just wrappers around a C
> library, so this might be your issue.

I don't use any wrapper around a C library. I use my (pure Go) MySQL
binds: https://github.com/ziutek/mymysql
It uses sync.Mutex for share connection to the MySQL server. I use
only one connection in my application - so far the performance of
using only one connection is sufficient.

The only calls in the code which may block are:
sync.Mutex.Lock(),
http.ResponseWriter.Write(),
net.Conn.Write().

[ziutek]

> The only calls in the code which may block are:
>  sync.Mutex.Lock(),
>  http.ResponseWriter.Write(),
>  net.Conn.Write().

And net.Conn.Read() of course.

[Russ Cox]

If you are only running native Go and your only real blocking
is network I/O (which should multiplex onto a single OS thread
via select, poll, kqueue, whatever), then I cannot explain how
you ended up with 36 OS threads. I created
http://code.google.com/p/go/issues/detail?id=1477
as a reminder to add profiling for why threads are
being created.

The most likely reason is that other system calls that do
not block for very long (open, read, gettimeofday) are
triggering the creation of new OS threads just in case
they block. But once you had a couple extra to handle
that case it should stop. I can't explain 36.

Russ

[ziutek]

I wrote simple test using http package:

package main

import (
"log"
"http"
)

func handler(wr http.ResponseWriter, req *http.Request) {
wr.Write([]byte("Ok!\n"))
}

func main() {
err := http.ListenAndServe(":8080", http.HandlerFunc(handler))
if err != nil {
log.Fatalln("ListenAndServe:", err)
}
}

Then compile it and run:

$ 8g test.go && 8l -o test test.8
$ GOMAXPROC=2 ./test

There is output from ps -eL |grep test

3176 3176 pts/0 00:00:00 test
3176 3177 pts/0 00:00:00 test

Next I use curl to get something from this application:

$ curl 127.0.0.1:8080
Ok!
$ curl 127.0.0.1:8080
Ok!

And ps -eL:

3176 3176 pts/0 00:00:00 test
3176 3177 pts/0 00:00:00 test

Next I use siege with 10 concurrent simulated users:

$ siege 127.0.0.1:8080 -c10 -d0 -t10s
** SIEGE 2.70
** Preparing 10 concurrent users for battle.
The server is now under siege...
Lifting the server siege. done.
Transactions: 11791 hits
Availability: 100.00 %
Elapsed time: 9.07 secs
Data transferred: 0.04 MB
Response time: 0.01 secs
Transaction rate: 1300.00 trans/sec
Throughput: 0.00 MB/sec
Concurrency: 9.85
Successful transactions: 11792
Failed transactions: 0
Longest transaction: 0.03
Shortest transaction: 0.00

And ps -eL:

3176 3176 pts/0 00:00:00 test
3176 3177 pts/0 00:00:00 test
3176 3260 pts/0 00:00:01 test
3176 3261 pts/0 00:00:00 test
3176 3262 pts/0 00:00:01 test
3176 3263 pts/0 00:00:01 test
3176 3264 pts/0 00:00:01 test
3176 3265 pts/0 00:00:01 test

there is 8 persistent threads. This number of threads is constant even
if I use siege with 100 concurrent users.

Is this the expected behavior? Maybe it is related to my 36 threads
"problem" (it isn't any problem for me if the number of threads will
not increase in time).

[Russ Cox]

Thanks. Created issue 1480.

Russ