Re: [go-nuts] why 1000 goroutine generats 1000 os threads?

[Rémy Oudompheng]

On 2012/7/24 <xiez...@gmail.com> wrote:
> run package main
>
> import (
> "fmt";
> "runtime";
> "strconv"
> )
>
> func main() {
> runtime.GOMAXPROCS(2)
> ch := make(chan int)
> n := 1000
> for i := 0; i < n; i++ {
> task(strconv.Itoa(i), ch, 100)
> }
> fmt.Printf("begin\n")
> for i := 0; i < n; i++ {
> <-ch
> }
> }
>
> func task(name string, ch chan int, max int) {
> go func() {
> i:= 1
> for i <= max {
> fmt.Printf("%s %d\n", name, i)
> //print(name + " " + strconv.Itoa(i) + "\n")
> i++
> }
> ch <- 1
> }();
> }
>
>
>
> $ ./par | less
>
> then ls /proc/<pid>/tasks | wc -l ，will output 1002
>
> strace -f ./par | less will also see so many clone sys call.

This is because your goroutines are concurrently calling Printf, which
is a blocking syscall. These are managed in different threads and if
you have 1000 of them outstanding at the same time, you get 1000
threads.

Rémy.

[ivan xie]

Printf uses Open and Write syscall at low level. If these syscall are blocking, most of io functions are blocking, then goroutine will be useless.

[Zippoxer]

Maybe when the syscall returns the thread is closed. But still, if all the goroutine does is long syscalls (like networking)? I heard that you can spawn thousands of goroutines without worrying for footprint, but what if all of them are doing syscalls for most of their lifespan?

[Jesse McNelis]

On Tue, Jul 24, 2012 at 8:03 PM, Zippoxer <zipp...@gmail.com> wrote:
> Maybe when the syscall returns the thread is closed. But still, if all the
> goroutine does is long syscalls (like networking)? I heard that you can
> spawn thousands of goroutines without worrying for footprint, but what if
> all of them are doing syscalls for most of their lifespan?
>

That's just how the operating system works. You can't do things the
operating system doesn't let you. Some of the packages make good use
of the async syscalls to reduce the number of threads eg. net pkg.

In the case of doing lots of Printf() you can reduce the syscalls it
does by giving it a buffered writer so that not every call is a
syscall.

It's easy to make silly examples of programs that will spawn lots of
threads, but rare to get real ones that do.

--
=====================
http://jessta.id.au

[⚛]

On Tuesday, July 24, 2012 8:23:47 AM UTC+2, ivan xie wrote:

Printf uses Open and Write syscall at low level. If these syscall are blocking, most of io functions are blocking, then goroutine will be useless.

A file descriptor in Linux and other operating systems is an abstraction.

fmt.Printf writes to os.Stdout. The file descriptor os.Stdout is opaque to the Go run-time. Go's run-time doesn't exactly know what will happen when it writes data to os.Stdout. The write can finish without blocking, or it can block for an indefinite amount of time. Because Go's run-time does not know why the block is happening, it has to assume the most general case.

In the presence of many concurrent writes the run-time has to start a new OS thread for each concurrent write because it is assuming that a write W can (somehow, indirectly) unblock write B. From the viewpoint of Go's run-time, the sentence "not returning from write B immediately" entails the sentence "write B is blocked because it is (indirectly) waiting for data from W".

It is true that the run-time could keep track of whether multiple concurrent writes are writing to the exact same file descriptor. However, this is not implemented.

[André Moraes]

If you have 1000 goroutines blocking, you have 1000 blocked threads
and 1 running.

In order to have a smaller number of threads, but still be able to
handle multiple requests, start N goroutines that listen on a channel.

If your channel is buffered, than you can handle N*(buffer size) tasks
without blocking the dispatcher.

With N = 10 and BUFFER = 100 you can dispatch 1000 requests.


--
André Moraes
http://amoraes.info